Do we need to go to court if we have a divorce agreement?

No.  Parties who reach an agreement do not need to appear in court in order to have their Divorce Decree issued.