How do I create a subclass in Lua?

I recommend checking out the inheritance section of my answer here: https://scriptinghelpers.org/questions/59107/what-is-object-oriented-programming-and-how-is-it-used#57533

In this way of implementing class-based OOP with single-inheritance, essentially, if there's no key in the created object, the __index event of the objec'ts metatable looks it up in the class, and if there's no key in the class, the __index event of the class's metatable looks it up in the superclass, and so on.

As for protected methods / member, one possible idea would be to set your __index metamethod as a function rather than a table shortcut, using a proxy table, and to call getfenv() to determine whether or not the calling function is the class itself or a subclass, and if it is, return a table of protected members which has the __index metamethod set to the class table, or if the calling function is not the class itself or a subclass, return a lookup in the normal class table.

Alternatively, when extending a class, you could set the __index metamethod to be a table of protected members of the superclass, and set the __index metamethod of the protected member table to be the public superclass table.