I wouldn't be worried about hair loss, I'd be worried about the diseases associated with low testosterone because they have the potential to lower your quality of life and hair loss is just cosmetics.
I don't understand this obsession with your hair, some of us lose hair as we age, there's nothing we men can do about it, so there's no sense in worrying about what we have no control over.
Balled men have no problems with the ladies, the ladies are into bald and grey hair men. I had more grey hairs on TRT, stopped for awhile and the grey hair disappeared, but soon I'm going to restart TRT and am not worried about more grey hair.