You are rightly concerned about the long-term effect this might have on your son's self-esteem. From the information you provided, this seems to be more than a one-off occurrence.
But children are resilient to a degree, especially if at least one parent is emotionally mature, able to show empathy, and is consistent in his/her own parenting.
Key to divorce parenting is to ... remain focused on what is best for the child
I am glad to hear that you don't want to go back to court. I often remind parents that if you can't change the other parent during the marriage, it is even less likely you will able to change him/her afterwards.
It might seem unfair that your child has to go through this uncalled-for criticism, but life is hardly fair. You son is learning this lesson much sooner than others, and you could turn it into a life lesson for him.
Things don't have to turn nasty. You will not be able to change your ex, but a few things might help as damage control for your son. Change your own frame of reference, and start to call your ex "my son's father" instead. This change is symbolic of your relationship to that person. Changing how you refer to that person is the first step towards changing your attitude to him.
If you feel comfortable discussing it, ask your son if he wants to share his feelings. Sometimes children just want to get something off their chest.
To protect your child, you may need to point out that his father used inappropriate language. Your son will benefit if you can give him some tips on how to cope with his father's criticism and anger.