Manipulating the rate that physics are calculated?

"I've already found out that it can slow down or speed up the physics on the server"

I believe you can only do this on your local client.

The only alternative I can think of is manually scripting the physics yourself and otherwise keeping all bricks anchored, but besides requiring advanced mathematical/physics knowledge, it would be laggier compared to normal (since lua runs slower than C/C++/whatever Roblox is coded in). If you had a limited number of objects that you needed to simulate physics with, I'm sure the processing requirements would be reasonable. If that's the case, use Roblox's RunService to run your simulation every physics frame.

One thing that might work depending on what you're doing is to let the physics get simulated at its normal pace, but use bricks that are invisible in an area that no player is allowed to enter. Each physics frame, record the CFrames of the bricks and the current DistributedGameTime (again, use RunService to have your recording function get called every frame). Now you can "play back" the physics anywhere in the workspace at any pace you like (though you might need to interpolate between the recorded CFrames to get the specific one you need -- ex if you record at time 0 and 0.03, but you want it to play back the physics at half speed, you'd need the CFrame at time 0.015, which doesn't exist -- you'd have to calculate an estimation of it based on the CFrames for that brick from time 0 and 0.03).

I tried slowing down the physics "forcefully" using the following script, but the output showed that this method doesn't work consistently, nor does it actually slow down the physics, it just lowers the frequency that the physics is updated (and consequently lowers the graphical FPS for the server)

--Requires a NumberValue called "DelayFactor" in the workspace. A value of 0 means no stalling, other values attempt to stall for 1/60th*DelayFactor of a second per physics frame. Strangely, a values of '1' through '1.9' appeared to do nothing in my tests.
coroutine.resume(coroutine.create(function()
    while true do
        wait(0.1)
        local p = Instance.new("Part")
        p.Position = Vector3.new(0, 20, 0)
        p.Parent = workspace
        game:GetService("Debris"):AddItem(p, 2)
        parts = parts + 1
    end
end))
parts = 0
lastTime = tick()
function SpinLock(length)
    local t = tick()
    while tick() - t < length do end
end
df = workspace.DelayFactor
game:GetService("RunService"):BindToRenderStep("slowdown", 1, function()
    if df.Value <= 0.0001 then return end
    SpinLock((1/60) * df.Value)
end)

while true do
    wait(1)
    print(workspace:GetRealPhysicsFPS(), workspace:GetPhysicsThrottling(), parts / (tick() - lastTime))
    lastTime = tick()
    parts = 0   
end