Nah don't worry about you never getting all that back. I've read multiple times on forums about 60-70 year old guys getting erections, libido and orgasm quality back after not having any of that for 20-30 years. I'd say theres like a 99% chance that once you find your sweet spot of free T and sens E2, you'll get all that back. Just got to keep fine tuning and checking labs.
You definitely want E2 a little too high than a little too low though. E2 is extremely important for a number of things in males, sexual function absolutely being one of them.
Have you had your free testosterone level checked?