Humanoid.MoveToFinished not working right?

So I have a line of code that makes the character walk from Point A to Point B, and then when its run again the character is meant to walk from Point B to Point A, But for some reason when the code is run it moves perfectly from Point A to Point B, but then for some reason when its run again it goes to Point B and then stops.

After doing some testing using prints I found out that when it reaches Point B the code acts like its reached both points at the same time, after reading the code over I don't think both of the points are being set to the same location but I'm not really sure what's happening.

Here's my code:

if Toggle == false then
    Character.Humanoid:MoveTo(game.Workspace.Walkpoint1.Position)
    print("Point A")
else
    Character.Humanoid:MoveTo(game.Workspace.Walkpoint2.Position)
    print("Point B")
end 
Character.Humanoid.MoveToFinished:Connect(function()
    print("Finished")
    if Move == 1 then
        if Toggle == false then
            Character.Humanoid:MoveTo(game.Workspace.Walkpoint2.Position)
            print("Point B")
            Toggle = true
        else
            Character.Humanoid:MoveTo(game.Workspace.Walkpoint1.Position)
            print("Point A")
            Toggle = false
        end
        Move = 2
    else
        if Move == 2 then
            Move = 1
            Playing = false
            StopEvent:FireClient(Player)
            game.Workspace.Portal.CanCollide = true
        end
    end 
end)

(also if this helps anything the code is on a server script and the StopEvent runs to a local script)