Does it create something in the game? If so, would I have to detect if something has been added somewhere and then kick that player if the created thing is called a certain thing?
Sorry, not possible. Lua by itself can't detect any DLLs, but its C API can, but since Roblox has C disabled, you're toast. On the upside, however, the only programming language I know that can "detect" DLLs is Visual Basic. Two example code lines:
Set objFSO = CreateObject("Scripting.FileSystemObject") Wscript.Echo objFSO.GetFileVersion("c:\windows\system32\scrrun.dll")