How do you Move/Drag models locally?

I have a model on the workspace that has multiple parts and I want to drag that model through the client side (so local script) but on the server side I don't want the other players to see where I am dragging it until I release it.

My attempt on solving this was having a local script in starter pack that can move the model with :MoveTo() so the player can drag it locally and when the player releases the model it fires a remote function that also uses :MoveTo()

The problem with this is that it is very buggy; when I drag the model it has a ping pong effect where i may drag it but it goes back to the former position time to time until I release and I don't know what causes this issue.

Please help, I expected it to be working with this concept and this "feature" is messing it up.

Edit:

Code snippet from the local script:

-- mouse_target is the model that is selected
-- temp_target/real_target is where they want the model to move to
function moveTarget()
    mouse.TargetFilter = mouse_target
    temp_target = mouse.Target
    if (is_down == true and mouse_target ~= nil) then
        if (temp_target ~= nil) then
            real_target = temp_target
            mouse_target:MoveTo(mouse.Hit.p)
        end
    else
        temp_target = nil
    end

end


-- model_module basically returns the Remote Function
function releaseTarget()
    if(real_target ~= nil and mouse_target ~= nil and is_down) then
        model_module.PlaceFunction(mouse_target, real_target)
    end
    is_down = false
    mouse_target = nil
    mouse.TargetFilter = nil
end

Code snippet from Remote Function

local function PlaceFunction(player, model, target)
    model:MoveTo(target.Position)
end