Ya another good point. High E2 definitely has an effect on the testicles, proven in multiple studies where estrogen was administered to males, and it definitely without a doubt can decrease testicular size, so I don't think its crazy to assume once E2 is lowered that testicular function can only improve.
What is your latest E2 and SHBG levels?