Make a Hold E or Press E system exploit proof?

Hello, I'm making a input service type of system similar to Jailbreak. In my game, the player will hold E for a certain amount of seconds and it will change the brickcolor of a part.

I have scripted all of this in a localscript for now as I'm unable to figure out a way to make this exploit proof. I already know how to use RemoteEvents and common practices I can use to see if the remote event being fired is valid. I have a fair knowledge in remote events, The only issue is there is no way to check on the server if the 'E' key has really been pressed for that long, meaning exploiters can just fire the event and bypass holding e

heres some code if it helps

local functions = {
	["Turn Green"] = {
		func = function(part) part.BrickColor = BrickColor.new("Lime green") end
	};
	["Turn Red"] = {
		func = function(part) part.BrickColor = BrickColor.new("Really red") end
	};
}

if numValue.Value >= 360 then --if held for however many seconds
	holding.Value = false
	numValue.Value = 0
	local name
	local func 
	for i,v in pairs(functions) do -- All the functions, normally this would be in a server script
		if desc.Text == i then
			name = i
			func = v.func 
			break
		end
	end
	if name and func then
		if name == "Turn Green" or name == "Turn Red" then
			func(obj.Value) -- this whole piece should have been in a server script, only issue is i cant get around that remote event exploit issue
		end
	end
end

Any input would be greatly appreciated!