Must a customer pay for a vehicle repair that did not correct the problem?

He and the shop each have claims against each other. You write that they completed the repair but not correctly: that means that your father had to pay them for doing the work they contracted to. If the work was not done to a competent or professional level, however, he may have had a counterclaim against them for negligence--that is, for carelessness in how they did the work--and could potentially recover the amount he paid for the rebuild. As a practical matter, if the cost of the rebuild is equal to or greater than the contracted price charged by the repair shop, your father would not owe anything or would even be owed money (the difference in cost) if the rebuild was much more costly. Conversely, if the cost of the rebuild is less than the amount your father  owes, he would--offsetting the claims--owe the shop the difference in cost in their favor.

However, the problem is, a customer cannot unilaterally decide when to pay and when not, or when the work is good enough or not--or rather, the customer can tell the shop he was dissatisfied and that the doesn't feel he should pay, but if the shop disagrees, they have the right to file a legal claim for breach of contract against the customer to try to collect their money. The customer can then file a counterclaim or countersuit, seeking compensation for the additional costs he incurred. So as this answer started, the shop and your father each have a claim against the other, but your father cannot simply refuse to pay without running the risk of a lawsuit being filed against him, since he does not have the right to make that determination--only a court can decide whom owes what to whom.

At this point, the way to respond is to reply to the law firm that the work was done negligently and that as a result of the sloppy, unprofessional work, your father incurred $X in costs that the shop is liable for. Attach copies of the bills for the engine rebuild and any other supporting documentation, especially anything from the place that did the rebuild blaming the poor previous repair work. If the cost of the build is less than what the shop is seeking from your father, offer to pay the difference; if the cost of the rebuild is greater than what they are seeking, you can either seek the difference or offer to forgive it and call matters even.

But bear in mind that if the repair shop rejects the above, they can go ahead and file a lawsuit, and your father will have to defend. He could also, as stated, interject his own counterclaim in the lawuit.