You need to teach 2 things.. a 100 percent solid down/stay and "leave it". If you have to use a collar and leash to keep him beside you and down, do it.
Do not continue trying to get interaction. The more they totally ignore him, the less stressed he'll be.
Once you can get him to stay at least 15 mins, get a male friend to come over. Put him in down stay and give the leave it command if he barks. Once he is QUIET, the person leaves. Keep repeating as often as you can get men to come over. Once he is quiet from the start, lengthen the time by a few minutes. ALWAYS tell him good boy and give a treat for being quiet.
It would help a lot if you can try this out in a park too. If he's less reactive to men there, they can walk past and toss a treat (not look at or talk to). Once he seems to like seeing them coming because he associates with treats, they can say "GOOD BOY" as they pass. Then they can stop and toss the treat. Eventually they can stop, say good boy, toss the treat, speak to you and move on. Eventually he will link men to good. Throw in some women too, but concentrate on men.