How can I disable the numerical ID's for keys like Shift, Alt and Ctrl?

I've made a custom hotbar UI, and it works well, however upon pressing Shift, LeftAlt or Ctrl the keys will select a few hotkeys on my hotbar.

I've deduced that the reason for this is that keys like "LeftShift", "LeftAlt" or "LeftCtrl" all have a number assigned to them for easy coding.

I'd like to disable that "feature" so players won't accidentally switch slots in their hotbars upon hitting a different button.

Here are examples of the behaviour:

Normal Behaviour while hitting keys 1-0 on the keyboard:

GIF

Abnormal Behaviour while hitting LeftShift, LeftAlt and LeftCtrl:

GIF

Here's the script responsible for the Navigation of the Hotbar:

local plr = game.Players.LocalPlayer
mouse = plr:GetMouse()
local selected 

mouse.KeyDown:Connect(function(key)
	if key == "1" or key == "2" or key == "3" or key == "4" or key == "5" or key == "6" or key == "7" or key == "8" or key == "9" or key == "0" then
		for i,v in pairs(script.Parent.Combat:GetChildren()) do
			if v:IsA("Frame") then
				v.BorderSizePixel = 0
				v.Number.TextColor3 = Color3.fromRGB(255,255,255)
			end
		end
		for i,v in pairs(script.Parent.Utility:GetChildren()) do
			if v:IsA("Frame") then
				v.BorderSizePixel = 0
				v.Number.TextColor3 = Color3.fromRGB(255,255,255)
			end
		end
		
		local keys = {"1","2","3","4","5","6","7","8","9","0"}
		
		for i,v in pairs(keys) do
			if v == key then
				selected = key
				if key == "1" or key == "2" or key == "3" or key == "4" or key == "5" then
					script.Parent.Combat[v].BorderSizePixel = 3
					script.Parent.Combat[v].Number.TextColor3 = Color3.fromRGB(255, 255, 0)
					print("Slot "..key.." selected!")
				elseif key == "6" or key == "7" or key == "8" or key == "9" or key == "0" then
					script.Parent.Utility[v].BorderSizePixel = 3
					script.Parent.Utility[v].Number.TextColor3 = Color3.fromRGB(255, 255, 0)
					print("Slot "..key.." selected!")
					print("Slots deselected!")
				end
			end
		end	
	end
end)

TL;DR I'd like to disable the numerical variable of keys like LeftShift, LeftCtrl and LeftAlt.