How can I fix my spectating system?

Well, there were a few flaws in your code (which was possibly breaking your script). Let's fix your issues step-by-step:

First, Forward button. In your code, you have these lines for your Forward button. The fix is mentioned below as well as where was the error :

SpectateFrame.Forward.Activated:Connect(function()
    if SpectatorIteration < #Players:GetPlayers() then -- You have used here 'if SpectatorIteration <= #Players:GetPlayers() then', highlighting the '<='. It means the first part is getting executed if SpectateIteration is equal to or less that all players. That 'equal to sign' is the reason, for blocking the second part of your code, 'elseif SpectatorIteration == #Players:GetPlayers() then'.
        SpectatorIteration += 1
        SpectatePlayer(SpectatorIteration,"Forward")
    elseif SpectatorIteration == #Players:GetPlayers() then
        SpectatorIteration = 1
        SpectatePlayer(SpectatorIteration,"Forward")
    end
end)

Second, Back button. Well, after testing the Back button multiple times, I figured out that the Spectateiteration value was going down to 0. So, I edited your custom function and here it is what it should look like :

local function SpectatePlayer(Iteration,Action)
    local SelectedPlayer = Players:GetPlayers()[Iteration]
    if SelectedPlayer then
        if SelectedPlayer ~= Player then
            if SelectedPlayer.Character and SelectedPlayer.Character:FindFirstChild("Humanoid") then
                SpectateFrame.PlayerName.Text = SelectedPlayer.Name
                PlayerCamera.CameraSubject = SelectedPlayer.Character.Humanoid
            end
        else
            if Action == "Back" then
            -- The problem starts from here
                if SpectatorIteration ~= 1 then -- It is a double checker whether the value isn't 1
                    SpectatorIteration -= 1 -- If the value was 1, it turned it to 0, which was creating the problem
                    SelectedPlayer = Players:GetPlayers()[SpectatorIteration]
                    if SelectedPlayer then -- Double checking if the player exists
                        if SelectedPlayer.Character and SelectedPlayer.Character:FindFirstChild("Humanoid") then
                            SpectateFrame.PlayerName.Text = SelectedPlayer.Name
                            PlayerCamera.CameraSubject = SelectedPlayer.Character.Humanoid
                        end
                    end 
                else
                    SpectatorIteration = #Players:GetPlayers() -- Setting the value back to the max. player joined

                -- Rest is your code
                    SelectedPlayer = Players:GetPlayers()[SpectatorIteration]
                    if SelectedPlayer then -- Double checking if the player exists
                        if SelectedPlayer.Character and SelectedPlayer.Character:FindFirstChild("Humanoid") then
                            SpectateFrame.PlayerName.Text = SelectedPlayer.Name
                            PlayerCamera.CameraSubject = SelectedPlayer.Character.Humanoid
                        end
                    end 
                end 
            elseif Action == "Forward" then
                SpectatorIteration += 1
                SelectedPlayer = Players:GetPlayers()[SpectatorIteration]
                if SelectedPlayer then
                    if SelectedPlayer.Character and SelectedPlayer.Character:FindFirstChild("Humanoid") then
                        SpectateFrame.PlayerName.Text = SelectedPlayer.Name
                        PlayerCamera.CameraSubject = SelectedPlayer.Character.Humanoid
                    end
                end 
            end
        end
    end
end

Now, you have also mentioned that sometimes you have double-click to change the spectator. The problem was created by the Back button only, when the value changes to 0. So, this issue will be fixed automatically after applying Back button fix.

If you want the code, here it is. Just ignore some of the variables, because I had to re-create in Studio what you were trying to do.

local SpectatorIteration = 1
local Players = game:GetService('Players')
local Player = Players.LocalPlayer
local PlayerCamera = workspace.CurrentCamera
local SpectateFrame = script.Parent.Frame

local function SpectatePlayer(Iteration,Action)
    local SelectedPlayer = Players:GetPlayers()[Iteration]
    if SelectedPlayer then
        if SelectedPlayer ~= Player then
            if SelectedPlayer.Character and SelectedPlayer.Character:FindFirstChild("Humanoid") then
                SpectateFrame.PlayerName.Text = SelectedPlayer.Name
                PlayerCamera.CameraSubject = SelectedPlayer.Character.Humanoid
            end
        else
            if Action == "Back" then
                if SpectatorIteration ~= 1 then
                    SpectatorIteration -= 1
                    SelectedPlayer = Players:GetPlayers()[SpectatorIteration]
                    if SelectedPlayer then
                        if SelectedPlayer.Character and SelectedPlayer.Character:FindFirstChild("Humanoid") then
                            SpectateFrame.PlayerName.Text = SelectedPlayer.Name
                            PlayerCamera.CameraSubject = SelectedPlayer.Character.Humanoid
                        end
                    end 
                else
                    SpectatorIteration = #Players:GetPlayers()
                    SelectedPlayer = Players:GetPlayers()[SpectatorIteration]
                    if SelectedPlayer then
                        if SelectedPlayer.Character and SelectedPlayer.Character:FindFirstChild("Humanoid") then
                            SpectateFrame.PlayerName.Text = SelectedPlayer.Name
                            PlayerCamera.CameraSubject = SelectedPlayer.Character.Humanoid
                        end
                    end 
                end 
            elseif Action == "Forward" then
                SpectatorIteration += 1
                SelectedPlayer = Players:GetPlayers()[SpectatorIteration]
                if SelectedPlayer then
                    if SelectedPlayer.Character and SelectedPlayer.Character:FindFirstChild("Humanoid") then
                        SpectateFrame.PlayerName.Text = SelectedPlayer.Name
                        PlayerCamera.CameraSubject = SelectedPlayer.Character.Humanoid
                    end
                end 
            end
        end
    end
end

SpectateFrame.Back.Activated:Connect(function()
    if SpectatorIteration >= 2 then
        SpectatorIteration -= 1
        SpectatePlayer(SpectatorIteration,"Back")
    elseif SpectatorIteration == 1 then
        SpectatorIteration = #Players:GetPlayers()
        SpectatePlayer(SpectatorIteration,"Back")
    end
end)

SpectateFrame.Forward.Activated:Connect(function()
    if SpectatorIteration < #Players:GetPlayers() then
        SpectatorIteration += 1
        SpectatePlayer(SpectatorIteration,"Forward")
    elseif SpectatorIteration == #Players:GetPlayers() then
        SpectatorIteration = 1
        SpectatePlayer(SpectatorIteration,"Forward")
    end
end)

Lemme know if it helps!

Well you are making some uneccessary actions in your script and making it too long, exactly for your case there is modulus operator. So your Issue 1 is that you want to skip from last player to first and from first to last if i understood you right, so you would need to use modulus operator instead (Which also should fix Issue 2). If you don't know how does the modulus work, here is quick explanation:

5 % 5 = 0
12 % 5 = 2
7 % 5 = 2
6 % 5 = 1

Modulus is how much left when divided, so 5/5 is 1, nothing is left. Since you can't divide 12/5 (To get normal number) modulus divides in this case 10/5 and returns 2 because this is how much is left. (Sorry this explanation is trash).

But in your case, here is how you could easily use Forward and Back buttons: So first you need as you said to remove local player so you can't spectate yourself, this can easily be done with this function:

local Players = game:FindService("Players")

local LocalPlayer = Players.LocalPlayer

local function GetPlayers()
    local Players = Players:GetPlayers()

    table.remove(Players, table.find(Players, LocalPlayer.Name))

    return Players
end

First the function gets all players on the server but then it uses table.find to find index of value in the table, so all Players is table like

{
    "WideSteal",
    "kirda",
    "jesus69",
}

And local player is jesus69 so it will search for this value and if it finds it then it will return index of the value in the table (Just classic pairs loop). Since table.remove needs 2 arguments (Table and Index) it should remove the local player by name from the table. Now that you've got players table, use the modulus operator for SpectatorIteration:

local SpectatorIteration = 1

local function Back()
    SpectatorIteration = (SpectatorIteration - 1) % (#GetPlayers())

    if SpectatorIteration == 0 then
        SpectatorIteration = #GetPlayers()
    end

    SpectatePlayer(SpectatorIteration)
end

local function Forward()
    SpectatorIteration = (SpectatorIteration % #GetPlayers()) + 1

    SpectatePlayer(SpectatorIteration)
end

So what Back function does is that it subs 1 from the Iteration and then uses modulus to 'module' it, it then checks if the Iteration is 0 and if it is then it sets the Iteration to the last player. Forward does the same but it adds 1 instead, here you don't need to check if Iteration is 5 because i can't explain, sorry (But is easy to understand). And here you see i used SpectatePlayer function which looks like this:

local function SpectatePlayer(Iteration)
    local SelectedPlayer = GetPlayers()[Iteration]

    if SelectedPlayer then
        -- do your stuff
    end
end

You see here you don't have to check what action is what and, it uses the GetPlayers function which again prevents from spectating local player and should work good i guess. But maybe it does not work, i dunno you can test it.