Gah, I need a stronger phrase

I can see why you're frustrated. The phrasing feels a little awkward, though it's grammatically correct.

You could always go with the simple "wished you'd never met him/her" (don't know the gender of this "partner"). In a way, it ups the stakes. Going from wishing you weren't partners to wishing you'd never met.

Or, you could try reversing the order.

"I don't think so. I'm sure there were times where you wished you had a different partner. He pulled lots of stunts, didn't much care for authority, often defied you, heck, I wouldn't blame you if you hated him a little sometimes."

Something like that. Reversing sentence order can be helpful. A good tactic is to print it out, cut out your sentences, then move them around on a flat surface. Also works with paragraphs, if you're not sure where to put them. You could also give it a day, clear your head, and come back to it. That always works for me when I'm editing.