Why doesn't this magnitude detection script work?

The issue is that you never update Mag, the calculation is always only done once at the very top of the script. Instead, define Mag inside the event to prevent this.

local player = game.Players.LocalPlayer
local mouse = player:GetMouse()

mouse.KeyDown:Connect(function(key)
    if key == "e" then
    local Mag = (game.Workspace.Part.Position - player.Character:WaitForChild("HumanoidRootPart").Position).magnitude
        if Mag <= script.Parent.Range.Value then
    game.StarterGui.Gui.MainFrame.Visible = not game.StarterGui.Gui.MainFrame.Visible
    end
    end
end)

By doing this, the magnitude calculation runs every time you press the E key instead of just once at the very top of the script.

Additionally though this isn't needed, I'd recommend using UserInputService for any future work after this, it has several useful features that mouse.KeyDown doesn't have.