Code

Essentially you want to run an if statement checking if the player count is lower than 5. You can do this by checking every time a player joins.

local val = 5 -- player limit

function check()
    if #game.Players:GetChildren() < val then -- # is essentially converting a value in to number.
        return true
    else
        return false
    end
end

game.Players.PlayerAdded:Connect(function()
    local val1 = check() -- getting returned value
    if val1 == true then -- if there are less than 5 players
        -- execute function
        print("Less than 5 players are in")
    else
        print("More than 5 players are in!")
    end
end)

Yay, it's solved. Here's the solution. I made a value in the 'Players' tab, it adds 1 whenever 1 player joins.

local Players = game.Players
local PlayerNumber = game.Players.PlayerNumber
Players.PlayerAdded:Connect(function()
    PlayerNumber.Value = PlayerNumber.Value + 1
end)

This one is the if and then statement, I tried it without the '==' this time.

local map1 = game.workspace.Map1
local map2 = game.workspace.Map2
local map3 = game.workspace.Map3
local Players = game.Players
local PlayerNumber = game.Players.PlayerNumber
local playercountdeny = 'not enough players'
if PlayerNumber.Value < 8 then
    print (playercountdeny)

end