How do I set functions "off limits" in a customized environment?

You'd want to set the function's environment to an empty table, with a metatable using __index to control what you can, and can't do. Which, you got close to, but not quite.

First, let's establish what we don't want anyone to access:

local Blocked = {
	"getmetatable",
	"setmetatable",
}

Now, let's make a function to check if an index is on our list:

function IsBlocked(tab, key)

	-- Iterate the "tab" variable (should be a list)
	for i,v in pairs(tab)do

		-- if v is equal to the key we returned, then ...
		if v==key then

			-- return the key
			return v
		end

	end

	-- No match was found, return nil.
	return nil
end

Set the new environment:

setfenv(1,setmetatable({},{
    __index = function(t, key)

	-- Get the blocked key
        local keyBlocked = IsBlocked(Blocked, key)

	-- if blocked key exists, then ...
        if keyBlocked then
            print("Error: "..keyBlocked.." is locked.")

		-- return nil
            return nil

        else -- If not blocked ...

		-- return old environment index
            return getfenv()[key]
        end
    end
}))

Final product:

local Blocked = {
	"getmetatable",
	"setmetatable",
}

function IsBlocked(tab, key)
	for i,v in pairs(tab)do
		if v==key then
			return v
		end
	end
	return nil
end

setfenv(1,setmetatable({},{
    __index = function(t, key)
        local keyBlocked = IsBlocked(Blocked, key)
        if keyBlocked then
            print("Error: "..keyBlocked.." is locked.")
            return nil
        else
            return getfenv()[key] 
        end
    end,
	__metatable = "no metatable for u",
}))

print(getmetatable) -- > Error: getmetatable is locked.
print(setmetatable) -- > Error: setmetatable is locked.

You can test it here as well: http://repl.it/BCua/1