sounds like there's extra play in the adjustment... if its noticeable and bothering you take it to the dealership and have them look at it. if they cant adjust it maybe they'll replace it for you.
or if that doesnt work... i really need a drivers side seat base from an 04+... if you want to trade yours for mine... mine has no height adjustment and sits on the high side (its a jdm base). but it bolts down solid
. dont know if you need that height adjustment or not but if not then maybe this is already in a good position for you as is.