Josh Duggar has his wife Anna Duggar’s support amid his ongoing child pornography case.

“She’s standing by him,” a source tells PEOPLE exclusively. “She thinks Josh is innocent.”

“[Josh] has seen [his] children with Anna there. She has to be there,” the insider adds of Anna, who is currently pregnant with the pair’s seventh child.

Late last month, the U.S. Marshals Service confirmed to PEOPLE that Homeland Security arrested Josh, 33, and transported him to an Arkansas jail where he was held without bond.

While in police custody, the 19 Kids and Counting alum appeared in court virtually via Zoom and pleaded not guilty to charges of receiving and possessing child pornography.

At the time, U.S Chief Magistrate Judge Erin L. Wiedemann also declared during the hearing that Josh would be prohibited from returning to his Arkansas home with Anna, 32, and their six children if released. Instead, he would have to be “in a residence where there’s no minor in the home.”

Josh was released from jail one week after his arrest. As he awaits trial, he is required to stay with third party custodians Lacount and Maria Reber. He is being confined to their home with GPS tracking.