Yeah definitely growing pains. They didn't quite have the camera angles figured out and a lot of game designers were trying to make their 2D philosophy fit in 3D which didn't always work out. The biggest difference is when they figured out dual analog controls, which was practically impossible on N64. Playing Goldeneye after games like Halo dropped was nearly impossible because of the difference in controls.

That said, some games hold up really well like Mario 64 and both Zelda games, though they're not without their camera issues.