If a father agrees to a custody plan but then doesn’t use his time, can the mother go back to court and have custody lessened for him?

Here is the law in Tennessee (36-6-101): "If the issue before the court is a modification of the court's prior decree pertaining to a residential parenting schedule, then the petitioner must prove by a preponderance of the evidence a material change of circumstance affecting the child's best interest. A material change of circumstance does not require a showing of a substantial risk of harm to the child. A material change of circumstance for purposes of modification of a residential parenting schedule may include, but is not limited to, significant changes in the needs of the child over time, which may include changes relating to age; significant changes in the parent's living or working condition that significantly affect parenting; failure to adhere to the parenting plan; or other circumstances making a change in the residential parenting time in the best interest of the child."  The burden of proof is on you to show the court something more than just a lack of spending time with the child.  Possibly that his failure to show up for scheduled visits harms the child emotionally and ties your hands in some way to schedule activities in the best interest of the child.  You are going to have to really think and be creative here.  Good luck.