How do I fix my release mouse on key press script?

Try checking if player pressed R first before checking if IsReleasing is 0 or 1 instead of checking them both at the same time. I recommend using BoolValues instead cuz if the number was changed to 2 or any other number for some reason, the script wouldn't work unless putting it back to 0 or 1. And I also recommend to use attributes instead of using BoolValues. Attributes are basically custom properties you can add to instances in replacement of using BoolValues, IntValues, etc.

Also you don't need to check if a value is true or not you can just do Value = not Value.

local Player = script.Parent.Parent or game:GetService("Players").LocalPlayer
local UIS = game:GetService("UserInputService")
local PlayerGUI = Player:WaitForChild("PlayerGui")

script:SetAttribute("IsReleasing", false)
UIS.InputBegan:Connect(function(Input)
    local IsReleasing = script:GetAttribute("IsReleasing")
    if Input.KeyCode == Enum.KeyCode.R then
        script:SetAttribute("IsReleasing", not IsReleasing)

        local GUI = PlayerGUI:WaitForChild("LockMouseGUI", 3) -- might delay

        if not GUI then
            local newGUI = Instance.new("ScreenGui", PlayerGUI)
            newGUI.Name = "LockMouseGUI"
            GUI = newGUI
        end

        local Lock = GUI:WaitForChild("Lock", 3) -- might delay

        if not Lock then
            local newLock = Instance.new("TextButton", GUI)
            newLock.Name = "Lock"
            newLock.BackgroundTransparency = 1
            newLock.Text = ""
            Lock = newLock
        end

        Lock.Modal = script:GetAttribute("IsReleasing")
    end
end)

Your IsReleasing variable is becoming invariable as you're only setting it once. The variable is being set from when the script loads. Since you're not setting it after it has loaded, it will remain the same value from that point.

Move the variable inside the InputBegan event, so it updates with each key press.

EDIT in regards to your response:

It isn't updating the IntValue because you're not changing the IntValue's value. All you're doing is just hard storing the intvalue's value to the variable and then changing the variable instead of the intvalue's value.

local variable = IntValue.Value -- Setting the variable to the value of the IntValue

variable = 1 -- is just changing the variable to "1", not changing the value of the intvalue.

In the code above, on the first line, what's happening is that the intvalue's value is being hard stored to the variable. Whenever you change the variable it's just going to change the variable instead of changing the IntValue's value.

To have it change the IntValue's value you need to store the actual object to the variable and call .Value on it.

local variable = IntValue -- Is now referring to the IntValue object

variable.Value = 1 -- is now changing the value of the IntValue to 1

Hmm... this doesn't seem to be working. I've located the main problem though: the IntValue isn't even updating.