The only gym leader I had trouble with was Drayden, mainly because I didn't have a mon on my team that could tackle dragon types. Quetzal (my Serperior) managed to hold his own against them, but when you have no elixirs, it gets tiring having to get out of there and go to the Pokecenter.
I ALSO HAD ISSUES WITH THE FIRST GYM BUT I WAS BEING A RETARD WITH BIDE WHILE HIS LILLIPUP POWERED UP LOL.
But otherwise, the game itself has been really hard compared to past gens. There's a 20 level jump after the E4, too. @_@;