This is a common issue with the joysticks on the controllers. Over time and from use there is a little clip in there that wears down and you start to get stick drift. I have it on 2 different ones, only both are in the left stick for me. Always upwards though.
There are some supposed ways to try to fix this posted online, but none of them fixed mine (anything from pressing hard and moving in full circles to alcohol & cotton swabs to remove oils & dirt - none of which solve the underlying problem).
You can try to take the controller apart with a T-8 bit and spudger and see if anything is broken, but it may or may not be something you want ot get into.