What to do if 5 months ago we moved out of state with permission from my child’s father but now he wants partial physical and legal custody with visitation?

You have several sub-parts to your question-- so I'm going to try to address them in order:

Your first question is how is visitation determined.  Texas has visitation schedules that all judges use as a guideline for setting up a visitation routine.    When parent's live closer together, the other parent usually gets visitation the first, third, and fifth weekend each month.  If the parents live farther apart, that is cut back significantly on a monthly basis, but the parent is given extra visits in the summer to help make up for the lost time.  Most judges adhere to this schedule, but they can deviate from the routine if the parties request a deviation or it it's in the best interest of the child to deviate.  If the child hasn't seen or developed a relationship with the dad in four years, the judge can limit visitation on the basis that suddenly requiring overnight visits with a "stranger" would be too hard on the child.  Some judges will gradually increase the visitations to help the child get to know the parent before granting regular, standardized visits.

Your next question would be how would it work with us living in another state.  If he is granted visitation, then most likely, once a month, dad will have the right to see the child.  Who absorbs the cost will be up to the court and the parties.  The court can order the expense of the travel to be split, order one party to absorb the costs, and can order transportation in a certain way (plane, car, etc.)  The parties can also reach agreements regarding the same.

Your next question is can he make you move back?  The answer is no.  Even though you did not need his permission to move out of state, he specifically granted permission.  He can't agree to something and then complain about it later.  The second reason is for child stabilty--- the child has already endured the disruption of one move... most courts are not going to add to the disruption by requiring the child to pack their bags and move back.

Your next section was more of a statement rather than a question--"he has never lived with the child nor cared...."-- but it does raise another question.  If he has gone over a year with no (or minimal contact with the child) and he has provided no support for the child in over a year, then you may have a basis for terminating his parental rights.  Even if you do not go the termination route, it is certainly helpful information to give the court when requesting the court to deviate from the standard visitation schedule and limit his access to the child.