How to simulate Roblox Catalog 3D rotation?

Call the camera's position "focus"

Call the zoom distance "zoom"

Call the X rotation "pitch"

Call the Y rotation "bearing"

Track the change in the mouse's position, subtract the x delta from the bearing, add the y delta to the pitch. Divide the delta by some number like 100 first so it doesn't move too quickly.

Constrain the pitch. To avoid gimbal lock, the absolute value of the pitch should be less than or equal to math.pi/2-.14154.

camera.CoordinateFrame = CFrame.new(focus) * CFrame.Angles(0,bearing,0) * CFrame.Angles(pitch,0,0) * CFrame.new(0,0,-zoom)

Here's a demo:

set CharacterAutoLoads to false

use a LocalScript in StarterPlayerScripts

local focus=Vector3.new(0,10,0)
local pitch,bearing=0,0
local maxPitch=math.pi/2-.14154
local camera=workspace.CurrentCamera
camera.CameraType=Enum.CameraType.Scriptable
local down=false
local posx,posy=0,0
local zoom=10
game:GetService('UserInputService').InputBegan:connect(function(input)
    if input.UserInputType.Name=="MouseButton1"then
        down=true
        local pos=input.Position
        posx,posy=pos.x,pos.y
    end
end)
game:GetService('UserInputService').InputEnded:connect(function(input)
    if input.UserInputType.Name=="MouseButton1"then
        down=false
    end
end)
game:GetService('UserInputService').InputChanged:connect(function(input)
    if input.UserInputType.Name=="MouseWheel"then
        zoom=math.min(math.max(zoom-input.Position.z,1),100)
        camera.CoordinateFrame=CFrame.new((CFrame.new(focus)*CFrame.Angles(0,bearing,0)*CFrame.Angles(pitch,0,0)*CFrame.new(0,0,-zoom)).p,focus)
    end
    if down then
        local pos=input.Position
        local x,y=pos.x-posx,pos.y-posy
        posx,posy=pos.x,pos.y
        pitch=math.min(math.max(pitch+y/100,-maxPitch),maxPitch)
        bearing=bearing-x/100
        camera.CoordinateFrame=CFrame.new((CFrame.new(focus)*CFrame.Angles(0,bearing,0)*CFrame.Angles(pitch,0,0)*CFrame.new(0,0,-zoom)).p,focus)
    end
end)