How to prevent errors coming to the output when trying to fire method from a metatable?

I'll try to make this as simple as possible because metatables can be a confusing concept for a number of reasons.

The __index metamethod is meant to be somewhat of a second check for the table it is assigned to. In your example, when you call metatable:testFunction() The script will go through the table that is provided, which would be the empty table directly after the setmetatable function

local metatable = setmetatable({} -- < This is the original table that you are "attaching" metamethods to

Once the script realizes oh no, there isn't anything named testFunction in the table!, it will turn to the metatable to double check. If the index in the original table cannot be found, it will do one of two things:

If __index is set to a table, the script will then search for the function in the table

If __index is set to a function, the function will be called with two arguments; the original table and then what was trying to be accessed.

In the case of your script, the index would be testFunction, because testFunction could not be found in the original table.

EDIT: To clarify, the reason you are getting an error is that even though you are invoking the index metamethod you aren't returning anything for the script to use as a function

So how do I make it not error?

You could return a function to be called in the __index function. An example of this is below

local metatable = setmetatable({}, {
	__index = function(oldTbl, index)
		print(index)        
		print("Just invoked __index metamethod")
		return function(self, ...) --Returning a function that can be called!
			print(...)
		end
	end
})

metatable:testFunction("Hello world!") --Prints "Hello world!" 

Note that since you are doing metatable:testFunction("Hello world!") you are saying "okay, whatever is returned by metatable.testFunction is going to be a function. If you aren't sure whether this is going to be a function or not, I would recommend setting up a check like this

local metatable = setmetatable({}, {
	__index = function(oldTbl, index)
		print(index)        
		print("Just invoked __index metamethod")
		return
	end
})

local func = metatable.testFunction

if func ~= nil then --Checking to make sure the function is there and not nil
	metatable:testFunction("Hello world!")
end

This second method will still invoke the __index metamethod, but it won't error like your original script was because it checks if the function actually exists or not

I hope this helped! If it didn't feel free to comment and I'll try to explain it better