How does OOP work when using it in game? What are some in-game examples.

There is ton of examples, the thing is that OOP is not neccessary it just makes your script more simple and readable, you can always find a way without using OOP.

I am using OOP for pretty much anything, i don't use it in cases where it's not needed of course but if you don't understand where it fits, take a look at roblox Instances! They are great example of OOP, you have one main class, Instance, and then you have subclasses like Decals, Parts etc.

So here is example of Roblox OOP with Instances, take a look at this tree

Instance

Decal

Part

Script

ServerScript

LocalScript

ModuleScript

You have one main class, Instance, in that, they specify default properties for ALL subclasses of the Instance table, in Roblox that's Name, Parent and some other stuff, that way you don't have to rewrite properties for each instance but you have them in main class. Now there are subclasses, Decal for example, it has properties like Texture or Visible, so all the properties are stored in table of that subclass. Script is also a subclass which has property Source, it has also 3 another subsubclasses which are ServerScript, LocalScript and ModuleScript, they all are connected to the Script subclass which means they will also have the property. Since they are also connected to Instance, it means that they will also have property Name and Parent, that simple.

Now there are also functions with OOP, like Destroy, that's a function of Instance class which means all Instances have access to it, let's say it looks like this

function Instance:Destroy()
	self.Parent = nil
	self:_lockParent()

	for _, Connection in ipairs(self._connections) do
		Connection:Disconnect()
	end
end

Now methods and variables that start with _ which are _connections and _lockParent in this case are hidden to public and only used by Roblox so that's why they have the underscore to be harder to type, these 2 probably do not exist but each Instance has hidden property which stands for all connections connected to it, and Roblox also somehow must lock the parent after destroying, i did not implement the method but i hope that's understandable.

Now the OOP comes here with function Instance.new, remember Car.new? That's the same thing!.. Creating new instance accepts Class argument which is a string, that way you specify the subclass of the Instance and it creates metatable connected to the subclass. All subclasses are connected with metatable to their parent, so that's why they have access to method and properties of Instance since they all are connected with it in some way.

Let's say Instance.new looks something like this

local Instance = {
	Properties = {
		Name = '',
	}
}
Instance.__index = Instance

local RobloxInstanceClasses = {
	Decal = {
		Properties = {
			Texture = '',
		}
	},
}

--/This will set metatable of all instance subclasses to
--\Instance to let them have access to functions like
--\Destroy  and properties like Name
--/NOTE that this is a dirty way of doing things, i am just
--\trying to make it simple to understand, roblox's system
--\is a lot better than this poor loop
for Name, Class in pairs(RobloxInstanceClasses) do
	Class.__index = Class

	setmetatable(Class, Instance)
end

function Instance.new(Class, Parent)
	Class = RobloxInstanceClasses[Class]

	--/This self will be the Instance that will be returned
	local self = {}

	--/This will copy all properties of that class into this
	--\Instance's table, again roblox has better way of doing
	--\this but just for sake of example..
	for Name, Value in pairs(Class.Properties)
		self[Name] = Value
	end
	
	--/This will copy Instance's properties into the Decal
	--/That is Name only in this case
	for Name, Value in pairs(Instance.Properties)
		self[Name] = Value
	end

	--/Here it sets metatable of newly created Instance to
	--\the class it belongs to, this let's you access methods
	--\of that class, since Decal subclass is also connected
	--\to the Instance class (connected in that loop at the start),
	--\you can access methods of Instance too (Destroy for example).
	return setmetatable(self, Class)
end



-- TEST --
local Decal = Instance.new('Decal')
print(Decal.Texture) -- prints ''
Decal.Texture = 'rbxassetid://123'
print(Decal.Texture) -- prints 'rbxassetid://123'

--/For roblox it would print decal but since our Instance system is very bad
--\it will just print blank space since Instance.Properties.Name is blank too.
print(Decal.Name)

And this is the whole point of OOP, classes and subclasses, of course Roblox's instance system is a lot better than this one, this one is dirty but i hope it explains at least something, there are a lot of simpler examples of OOP but i just could not think of example other than this because others for me are too simple.

This example also did not include subsubclasses, maybe they don't exist in roblox, i don't know but i hope you get the main idea behind it. Now why you use __index? That's i guess you know but the point is that let's say Instance:Destroy() is a function of Instance and you create new metatable like this local a = setmetatable({}, Instance), this won't give the metatable access to functions of Instance but if you do Instance.__index = Instance, it gives the metatable access to it's metamethods, that means when you do a:Destroy() it will search for that function inside of the table first but it won't find it so it will look for it's __index metamethod and look in it, in this case it's Instance so it will look for Instance method called Destroy which exists! Okay it got too long sorry oh no!