Are functions used with table.sort allowed to yield?

Yielding

To yield means to pause the current thread for a certain amount of time. In RBX.Lua, the most common form of yielding is with the wait function. In your case, the method WaitForChild yields the current thread as well. Read the documentation on the method for more information.

As it turns out, when you call a function that yields, the lowest amount of time you can yield for is a single thread update. For example wait(0) still ends up yielding, just for the amount of time before the thread scheduler updates. The same thing with using WaitForChild on an object that already has a child that you are looking for applies. In your Compare function, you are definitely yielding.

Why does it matter

Callbacks in Lua are functions that are passed as arguments into functions. In programming, there are two types of callbacks: synchronous, which executes the callback immediately, and asynchronous, which executes the callback after a certain amount of yield time. For the sake of good design and to avoid conflicting with the language, you should always construct synchronous callbacks over asynchronous callbacks.

The type of callback requires depends on the case.

In this case, table.sort is expecting a synchronous callback. This is because table.sort is not a yield function; it will not pause your code until the table is sorted! If table.sort allowed for you to yield during sorting, then that would mean that there would be no guarantee that the rest of your code would be correct, assuming that the remainder of your code depends on the table to be sorted from table.sort.

Think of it like asking someone to do a task for you, and then immediately working on something else that entirely depends on the result of the task from before. It's impossible; you can't possibly know the results before they're calculated! The same thing applies here.

What you can do

In some cases, this can be pretty annoying. However, since table.sort is synchronous, we can assume that logic directly before it should be correct. We can move the scope of the variables that depend on WaitForChild outside of the callback.

Here is your original code...

function Compare(F, S)
    local Settings = script.Parent.Parent.Parent:WaitForChild("Settings")
    local SortBy = Settings:WaitForChild("SortBy")
    if SortBy.Value == "rarity" then
        local Info1 = Market:GetProductInfo(F.AssetId.Value)
        local Info2 = Market:GetProductInfo(S.AssetId.Value)
        local S1 = Info1.Sales
        local S2 = Info2.Sales
        if S1 < S2 then
            return true
        elseif S2 < S1 then
            return false
        else
            if F.RAP.Value >= S.RAP.Value then
                return true
            else
                return false
            end         
        end
    else
        error("Invalid sort type")
    end
end
-- ... some code here
table.sort(t, Compare) -- assuming some table t

And here is your code with some rearranging...

local Settings = script.Parent.Parent.Parent:WaitForChild("Settings")
local SortBy = Settings:WaitForChild("SortBy")

table.sort(t, function(F, S)
    if SortBy.Value == "rarity" then
        local Info1 = Market:GetProductInfo(F.AssetId.Value)
        local Info2 = Market:GetProductInfo(S.AssetId.Value)
        local S1 = Info1.Sales
        local S2 = Info2.Sales
        if S1 < S2 then
            return true
        elseif S2 < S1 then
            return false
        else
            if F.RAP.Value >= S.RAP.Value then
                return true
            else
                return false
            end         
        end
    else
        error("Invalid sort type")
    end
end)

Notice how all we did was move the yielding outside of the callback to a location where the callback could still access it. There's no reason to continuously check if an object exists during a synchronous callback because you can depend on Lua to execute the entire callback during a single update! If the object exists during the first part of the code, then you are pretty much guaranteed that it will exist for the remainder of the update.