Loop not Changing the Position of my Part?

So I've been stumped for a while, I can't find anything about this issue online, and anything I've tried hasn't helped.

In my game I want a gem that can be picked up and will respawn in a new location a minute after it's been picked up. And if a minute has gone by and no-one has picked it up, then it will move.

Here's the code, I'm getting no errors.

(Server script inside the gem that's inside the workspace)

wait(3)
local TweenService = game:GetService("TweenService")

local RanNum = math.random(-658.15,520.34)
local RanNum2 = math.random(-1348.535,135.925)

local Gem = script.Parent
Gem.Position = Vector3.new(RanNum, -27.199, RanNum2)

local goal = {
    CFrame = Gem.CFrame * CFrame.fromAxisAngle(Vector3.new(0, 1, 0), math.rad(-180))
}

local tweenInfo = TweenInfo.new(2.0, Enum.EasingStyle.Linear, Enum.EasingDirection.InOut, -1, false, 0)

local tween = TweenService:Create(Gem, tweenInfo, goal)

tween:Play()

local Debounce = false
script.Parent.Touched:Connect(function(hit)
    if Debounce == false and Gem.Transparency == 0 then
        Debounce = true

        RanNum = math.random(-658.15,520.34)
        RanNum2 = math.random(-1348.535,135.925)

        Gem.Position = Vector3.new(RanNum, -27.199, RanNum2)

        local Reward = math.random(5,25)

        local Char = hit:FindFirstAncestorOfClass("Model")

        local player = game:GetService("Players"):WaitForChild(Char.Name)

        local Gems = player:WaitForChild("Gems")
        local TotalGems = player:WaitForChild("TotalGems")
        local Rebirths = player:WaitForChild("leaderstats"):WaitForChild("Rebirths")

        tween:Cancel()

        Gems.Value = Gems.Value + Reward
        TotalGems.Value = TotalGems.Value + Reward
        Gem.Transparency = 1
        game.ReplicatedStorage.Collect:FireClient(player)

        wait(60)

        tween:Play()
        Gem.Transparency = 0
        Debounce = false
    end
end)

while wait(5) do -- The problem is in this loop. (I only have it set to every 5 seconds for testing, I already tested it with a minute)
    if Debounce == false then
        print("Loop")

        RanNum = math.random(-658.15,520.34)
        RanNum2 = math.random(-1348.535,135.925)

        print("Coords")

        Gem.Position = Vector3.new(RanNum, -27.199, RanNum2)

        print("Moved")
    end
end

The picking up and respawning a minute later works fine, it's the loop that's the problem. The prints in the loop all fire and I tested to make sure the numbers for new co-ords were actually being changed. And they were. I also tried turning off the tween and starting it again, but that didn't work. I'm really stumped because picking it up works fine and the loop has the same code to change the position that the function does. If you can help I would appreciate it.

Edit: So, I found out the tween has to do with it, and apparently the function didn't actually change the position. I still don't know why, but I do know, if the tween never starts, it can change the position. But like I already said, I tried cancelling the tween and then changing the position but it didn't work. I tried this again, as well as pausing the tween, but it still doesn't work.