How can I disable the numerical ID's for keys like Shift, Alt and Ctrl?

I've made a custom hotbar UI, and it works well, however upon pressing Shift, LeftAlt or Ctrl the keys will select a few hotkeys on my hotbar.

I've deduced that the reason for this is that keys like "LeftShift", "LeftAlt" or "LeftCtrl" all have a number assigned to them for easy coding.

I'd like to disable that "feature" so players won't accidentally switch slots in their hotbars upon hitting a different button.

Here are examples of the behaviour:

Normal Behaviour while hitting keys 1-0 on the keyboard:

GIF

Abnormal Behaviour while hitting LeftShift, LeftAlt and LeftCtrl:

GIF

Here's the script responsible for the Navigation of the Hotbar:

local plr = game.Players.LocalPlayer
mouse = plr:GetMouse()
local selected 

mouse.KeyDown:Connect(function(key)
    if key == "1" or key == "2" or key == "3" or key == "4" or key == "5" or key == "6" or key == "7" or key == "8" or key == "9" or key == "0" then
        for i,v in pairs(script.Parent.Combat:GetChildren()) do
            if v:IsA("Frame") then
                v.BorderSizePixel = 0
                v.Number.TextColor3 = Color3.fromRGB(255,255,255)
            end
        end
        for i,v in pairs(script.Parent.Utility:GetChildren()) do
            if v:IsA("Frame") then
                v.BorderSizePixel = 0
                v.Number.TextColor3 = Color3.fromRGB(255,255,255)
            end
        end

        local keys = {"1","2","3","4","5","6","7","8","9","0"}

        for i,v in pairs(keys) do
            if v == key then
                selected = key
                if key == "1" or key == "2" or key == "3" or key == "4" or key == "5" then
                    script.Parent.Combat[v].BorderSizePixel = 3
                    script.Parent.Combat[v].Number.TextColor3 = Color3.fromRGB(255, 255, 0)
                    print("Slot "..key.." selected!")
                elseif key == "6" or key == "7" or key == "8" or key == "9" or key == "0" then
                    script.Parent.Utility[v].BorderSizePixel = 3
                    script.Parent.Utility[v].Number.TextColor3 = Color3.fromRGB(255, 255, 0)
                    print("Slot "..key.." selected!")
                    print("Slots deselected!")
                end
            end
        end 
    end
end)

TL;DR I'd like to disable the numerical variable of keys like LeftShift, LeftCtrl and LeftAlt.