Study both, you might as well double major in math and physics, you need the math as pre-reqs to upper level physics classes, and by the time your done with your math requirements for a physics degree,...