Why is my team teleport script teleporting all players?

So, in my game, I am writing a script that will teleport all the players on a team to a point. I have it so that the player joins the "lobby" team, and when they touch a brick, every one on the team teleport to a location, and those in the lobby will not teleport. Now, the script works when you touch it, and it teleports the player every 5 seconds, but upon testing in a server, it changes the team of everyone in the server. I can't find a reliable source on the internet on why this isn't working. Does anyone know why it isn't working? (P.S. the script is in server script service)

local gotobrick = workspace:WaitForChild("TestBrick2EletricBoogaloo")
local GameBrick = workspace:WaitForChild("TestBrick")

game.Players.PlayerAdded:Connect(function(Player)
	Player.CharacterAdded:Connect(function(character)		
		Player.Team = game.Teams.Lobby
		
		GameBrick.Touched:Connect(function(hitgame)
			if hitgame.Parent:FindFirstChild("Humanoid") then
				Player.Team = game.Teams.Playing
			end
		end)
	
		character:WaitForChild("Humanoid").Died:Connect(function()
			Player.Team = game.Teams.Lobby
		end)
		
		while true do
			if Player.Team == game.Teams.Playing then
				Player.Character.HumanoidRootPart.CFrame = gotobrick.CFrame
			end
			wait(5)
		end
	
	end)		
end)