How do I make this script just teleport the player, not all players?

Well, you're using a script for what seems to be a job cut out for a LocalScript. .MouseButton1Down is meant as something registered on the Client. However, if you use a LocalScript, the teleport will only be registered on the client. We need to use a RemoteEvent to register it on the server. You can read more about RemoteEvents and Server stuff here. First, you need to create a RemoteEvent in the ReplicatedStorage of your game. Name it something.

This is what your local script should look like.

script.Parent.MouseButton1Click:connect(function()
    game.ReplicatedStorage.TeleportEvent:FireServer()
end)

This is what your server script should look like, put it in the ServerScriptStorage.

game.ReplicatedStorage.TeleportEvent.OnServerEvent:connect(function(plr)
    local BuggyLoc = CFrame.new(-106.428, 121.84, -194.854)
    plr.Character.SetPrimaryPartCFrame(BuggyLoc)
end)

Hope that helped out, consider accepting this answer. If not, comment below and I'll be sure to help out further.

Assuming this is a server script and not a local script

local BuggyLoc = CFrame.new(-106.428, 121.84, -194.854)

script.Parent.MouseButton1Click:connect(function(Tele)
    for i, v in pairs(game.Players:GetPlayers()) do
        if v.Character then --Checking if the player has a Character
            if v.Character == script.Parent.Parent then --Checking if this player's character is the player's character
                    v.Character:SetPrimaryPartCFrame(BuggyLoc)
            end
        end
    end
end)