How to make a team Auto Balance script?

The teams service should automatically (try to) balance teams when players join but if you wanted to rebalance teams you could write your own function to handle that. Take a look at this wiki page that shows how to rebalance teams -I don't think the actual method works though but it wouldn't hurt to try.

However, to answer your question fully,

function getPlayersOnTeam(teamColor)
    local players = {}
    for _, p in pairs(game.Players:GetPlayers()) do
        if p.TeamColor == BrickColor.new(teamColor) then
            table.insert(players, p)
        end
    end
    return players
end

function getSmallestTeam(teams)
    local numPlayers, smallestTeam = math.huge
    for team, players in pairs(teams) do
        if #players < numPlayers then
            numPlayers = #players
            smallestTeam = team
        end
    end
    return smallestTeam
end

game.Players.PlayerAdded:connect(function(player)
    local reds = getPlayersOnTeam("Bright red")
    local blues = getPlayersOnTeam("Bright blue")
    local greens = getPlayersOnTeam("Bright green")
    local smallestTeam = getSmallestTeam({
        ["Bright red"] = reds,
        ["Bright blue"] = blues,
        ["Bright green"] = greens
    })
    player.TeamColor = BrickColor.new(smallestTeam)
end)

Whenever a player joins, the code above will put them on the smallest team out of the Bright red, Bright blue and Bright green teams.