How would I set an angle instead of adding an angle?

I am trying to set the Y rotation of a part on its own axis, but when I do CFrame = CFrame * CFrame.Angles(), It adds the Angles to the current Angle. Is there any way I can just set the Angle while still keeping the object rotating on it's own Y axis?

yes there is. This is what you would have to do. Create a new CFrame containing only the position of the previous CFrame. And then add your angle to the new CFrame. Example on how this would be done:

-- Let's imagine this is our CFrame
local MainCFrame = CFrame.new(10,3,1) * CFrame.Angles(0,math.pi,0)


-- We create a new cframe that has the position of our old cframe
local position = CFrame.new(MainCFrame.p)

-- We create a cframe that has new an angle.
local orientation = CFrame.Angles(0,math.rad(90),0)

-- we merge them together into one cframe 
local RotatedCFrame = position * orientation