Is it a good practice to infinitely nest a function?

I was thinking of how I was going to structure the flow of my game and I began to think how one state could move to another.

Of course, you could run a function, have it fully execute and complete, and then carry on, but what if the function can't efficiently reach it's end?

Say I had 3 functions:

function Menu()
--Allows user to engage in game. Calls Game() to start
end

function Game()
--User plays a game until the game until GameOver()
end

function GameOver()
--User ends game and goes back to Menu()
end

If you notice, each function calls each other. If this behavior were put in action, it would be an infinitely long function

Menu()
	Game()
		GameOver()
			Menu()
				Game()
					GameOver()
						Menu()
							....

So if you call Menu, you won't ever get past that function call

Menu() --function call, starts game

print("HI MOM") --this code is never reached

Is this a good practice to run such a system, or is there an alternative way to structure game flow?

I guess it's alright in my book if the game doesn't have any other functions like that. Just call menu in a coroutine:

coroutine.wrap(Menu)()

It is acceptable to do this only if you use proper tail-calls -- see https://www.lua.org/pil/6.3.html for examples. If you fail to use proper tail-calls, you will eventually overflow the stack. (As mightydifferent says, you can get past your initial call to Menu using coroutines.)

The link provided mentions that doing this is much like using goto -- this statement is not in many langauges for a good reason: it can easily lead to bad code. Having gotos everywhere, if they aren't carefully organized, can make it incredibly difficult to follow the logic of a bit of code. If you have 10s of functions that all call each other for different reasons, you may have a fairly difficult time figuring out what's going on if something goes wrong.

Alternatively, much of Roblox is structured around using events. For instance, leaderboard code doesn't do much until a player enters the game, then it acts (gives the player information and sets up more event listeners so that it can act when the player dies, for instance).