How do you set custom characters without the player reloading?

I have been making a game where the person will morph into a custom character, depending on their roll. So far I have this.

function Ghosting(player)   

    local ghostTemp = ghost:Clone()
-------function that sets the position--------------------
    ghostTemp:MoveTo(game.Workspace.PlayArea.GetSpot:Invoke())
    player.Character:Destroy()
    player.Character = ghostTemp
    ghostTemp.Parent = game.Workspace

    ------This is part of the model ------------------
    ghostTemp.RespawnMe.OnServerEvent:connect(function(player)
        player.Character:Destroy()
        player:LoadCharacter()

-------------Script that sets the camera to the character -------------------
        cameraset:Clone().Parent = player.Backpack
    end)
-----------gui--------------------
    ghostgui:Clone().Parent = player.PlayerGui

end

It works just fine in solo mode, but, when in server mode, some of the models don't work. The person is set to the character, but the character won't react; and the player's original character respawns as if it died.

I so far have two models. The one that it won't work on is basically a cloned version of myself with some scripts added. Strangest thing is that I have another model that is literally a brick with some local scripts. That one is just fine, both in local and server mode. The person can move it and there is no respawning. Is there a reason for this and any more info that you need?

P.S. the function above works. It's for the "brick".