How can I rectify the camera jittering issue in this movement script?

Alright so, I made the code below that rotates the players camera depending on if they are moving to the left/right. However, whenever I move left with a, move forwards with w, and then move my camera to the left, the camera shifts back into its original position until I stop moving my camera. This isn't supposed to occur. (It also happens if you move backwards and right, basically the opposite as the example) Any idea why it might be doing that? (MUST BE IN FIRST PERSON MODE)

local camera = workspace:WaitForChild("Camera")
local movementDirection = camera.CFrame:vectorToObjectSpace(humanoid.MoveDirection).Unit
    if math.abs(movementDirection.X) > math.abs(movementDirection.Z) then
        print(movementDirection.X)
        if movementDirection.X < 0 then
            camera.CFrame *= CFrame.Angles(0,0,0.1)
        elseif movementDirection.X > 0 then
            camera.CFrame *= CFrame.Angles(0,0,-0.1)
        else
            print("not moving")
        end
    elseif math.abs(movementDirection.X) < math.abs(movementDirection.Z) then
        -- if z is negative then character is moving forward, else they are moving backward (in relation to camera)

        print("forward or backward")
    end
end)

It's been driving me up the wall lately and I just can't seem to figure out why it might be doing this.

ISSUE IN ACTION: https://gyazo.com/987c104f50c191d969ee3aecbb00fffa