How do i stop the brick from changing color?

Thats script doesnt include any bools, try this instead, it might help:

First make a new local script, put it INSIDE the PLAYERGUI/STARTERPACK:

local part = game.Workspace.Part -- I like setting variables just to make things looks more neat
local Player = game.Player.LocalPlayer -- The Player you are referring to
local Mouse = Player:GetMouse() -- get the mouse

local CanChange = false -- This is the bool if the brick will change color or not

Mouse.KeyDown:connect(function(key) -- Function what happen when you press "e"
    if key:lower() == "e" then
        if CanChange == true then -- If the CanChange color is true then you can change color, if not, then nothing happens
            part.BrickColor = BrickColor.new("Bright red")
        end
    end
end)

part.Touched:connect(function()
    CanChange = true -- If you are touching, then you can change part color
end)

part.TouchEnded:connect(function()
    CanChange = false -- If you're not touching, then u cant change color
end)

You need to disconnect the KeyDown event when the player steps off the part. It would also be beneficial to check to see if it was the player who touched the part.

Good wiki articles to read:

RBXScriptConnection

TouchEnded

local part = game.Workspace.Parta
local player = game.Players.LocalPlayer
local mouse = player:GetMouse()

local KeyDown -- Will store the RBXScriptConnection from KeyDown

part.Touched:connect(function(hit)
    if player.Character and hit.Parent:IsDecendantOf(player.Character) then
        KeyDown = mouse.KeyDown:connect(function(key)
            if key:lower() == "e" then
                part.BrickColor = BrickColor.new("Bright red")
            end
        end)
    end
end)

part.TouchEnded:connect(function(hit)
    if player.Character and hit.Parent:IsDecendantOf(player.Character) and KeyDown then
        KeyDown:disconnect()
        KeyDown = nil
    end
end