I agree with you on many points having the learned the hard way over the last several years that TRT is not the panacea promised by the low T marketing machine.
I frequently question why I do TRT considering the time and money involved. The biggest disappointment for has been zero improvement in libido. But on balance, I feel better on TRT than without as an over 50 yr old dude.
You sound like you're trying to convince yourself to quit TRT. If you want to quit, then quit. But don't be so quick to dismiss the notion that you may need to tweak your protocol to minimize the side effects including ED. It could be as simple as lowering your dose.
Your natural numbers are not terrible (many of us were in low 200s before starting TRT) so you will be just fine. If you jumped on TRT merely for a bump in muscles, then yeah its probably not worth it.