With training, the aggressive growl will most likely disappear. I would first get him very used to all members of the family encouraging him to accept giving up food etc. With your experience I expect you'll know how to do this. I really don't want to 'teach my grandmother to suck eggs' so if you need further advice on this please let me know.
I've reread your post to ensure that I'm sticking to your question and see that you are asking for a story but yes this is from personal experience. He has obviously allowed to get away with it so it'll take a long patient time.
Hugs to you both and best wishes for success.