Left joystick is loose

I've got a problem with my left joystick, it is not fixed in place.

It's hard to describe, but I can pull it up and down. In other words, if I shake the controller, the left joystick rattles in the case.

It works for the most part, but if I'm not holding the controller vertically, it can cause some unintended movement of the potentiometer.

My question is, is the problem likely to be the joystick knob (little silver thing that looks like a nipple) or the potentiometer? One of them is fairly easy to replace and doesn't require soldering, and the other is way above my soldering skill.