Make a Hold E or Press E system exploit proof?

Hello, I'm making a input service type of system similar to Jailbreak. In my game, the player will hold E for a certain amount of seconds and it will change the brickcolor of a part.

I have scripted all of this in a localscript for now as I'm unable to figure out a way to make this exploit proof. I already know how to use RemoteEvents and common practices I can use to see if the remote event being fired is valid. I have a fair knowledge in remote events, The only issue is there is no way to check on the server if the 'E' key has really been pressed for that long, meaning exploiters can just fire the event and bypass holding e

heres some code if it helps

local functions = {
    ["Turn Green"] = {
        func = function(part) part.BrickColor = BrickColor.new("Lime green") end
    };
    ["Turn Red"] = {
        func = function(part) part.BrickColor = BrickColor.new("Really red") end
    };
}

if numValue.Value >= 360 then --if held for however many seconds
    holding.Value = false
    numValue.Value = 0
    local name
    local func 
    for i,v in pairs(functions) do -- All the functions, normally this would be in a server script
        if desc.Text == i then
            name = i
            func = v.func 
            break
        end
    end
    if name and func then
        if name == "Turn Green" or name == "Turn Red" then
            func(obj.Value) -- this whole piece should have been in a server script, only issue is i cant get around that remote event exploit issue
        end
    end
end

Any input would be greatly appreciated!