How should I predict when a destructible brick is about to be hit?

So, I'm trying to create some destructible bricks.

The problem is that the :Touched event happens too late. By the time it gets triggered, the colliding part has already lost all its velocity. I want to predict when a part is about to hit this destructible brick and react accordingly.

There's a problem though.

There will be hundreds, if not thousands of destructible bricks in my game, and I want them to be collidable with anything.

That means, for example, if a car made of these destructible bricks crashes into anything, or if anything crashes into it, I want it to break into pieces.

The problem with this wish is that I can't figure out how in the world that would even be possible.

In my destructible ModuleScript, there is this:

local tracked = {}

local function disconnect(part)
    tracked[part] = nil
end

local function goingToTouch(part, otherPart)
    if #part:GetConnectedParts() < 2 then return end

    local damage = (otherPart.Velocity - part.Velocity).Magnitude

    if damage > 5 then
        part:BreakJoints()
        part.Velocity = otherPart.Velocity
        part.CanCollide = true
    end
end

local function connect(part)
    if part:IsA('BasePart') then
        tracked[part] = true
    end
end

local function getIVertices(cframe, size)
    local vertices = {}

    vertices[1] = cframe * CFrame.new(size.X *  0.5, size.Y *  0.5, size.Z *  0.5)
    vertices[2] = cframe * CFrame.new(size.X * -0.5, size.Y *  0.5, size.Z *  0.5)
    vertices[3] = cframe * CFrame.new(size.X *  0.5, size.Y * -0.5, size.Z *  0.5)
    vertices[4] = cframe * CFrame.new(size.X *  0.5, size.Y *  0.5, size.Z * -0.5)
    vertices[5] = cframe * CFrame.new(size.X * -0.5, size.Y * -0.5, size.Z *  0.5)
    vertices[6] = cframe * CFrame.new(size.X *  0.5, size.Y * -0.5, size.Z * -0.5)
    vertices[7] = cframe * CFrame.new(size.X * -0.5, size.Y *  0.5, size.Z * -0.5)
    vertices[8] = cframe * CFrame.new(size.X * -0.5, size.Y * -0.5, size.Z * -0.5)

    return vertices
end

local function pointInsideIPart(cframe, size, point)
    -- Not my code
    -- https://scriptinghelpers.org/questions/30656

    local rel = cframe:PointToObjectSpace(point)
    size = size / 2

    if rel.X < -size.X or rel.X > size.X then
        return false
    elseif rel.Y < -size.Y or rel.Y > size.Y then
        return false
    elseif rel.Z < -size.Z or rel.Z > size.Z then
        return false
    end

    return true 
end

local function doIPartsCollide(cf1, size1, cf2, size2)
    local vertices1 = getIVertices(cf1, size1)

    for _, vertex in ipairs(vertices1) do
        if pointInsideIPart(cf2, size2, vertex.Position) then
            return true
        end
    end

    local vertices2 = getIVertices(cf2, size2)

    for _, vertex in ipairs(vertices2) do
        if pointInsideIPart(cf1, size1, vertex.Position) then
            return true
        end
    end

    return false
end

local parts = {}

for _, desc in ipairs(workspace:GetDescendants()) do
    if desc:IsA('BasePart') then
        parts[desc] = true
    end
end

workspace.DescendantAdded:Connect(function(desc)
    if desc:IsA('BasePart') then
        parts[desc] = true
    end
end)

workspace.DescendantRemoving:Connect(function(desc)
    if desc:IsA('BasePart') then
        parts[desc] = nil
    end
end)

local connection
connection = game:GetService('RunService').Stepped:Connect(function(t)
    for part, _ in pairs(parts) do
        if part.CanCollide == true then
            for destructible, _ in pairs(tracked) do
                if (part.Velocity - destructible.Velocity).Magnitude > (part.CFrame.Position - destructible.CFrame.Position).Magnitude - (part.Size.Magnitude + destructible.Size.Magnitude) / 2 and doIPartsCollide(part.CFrame + part.Velocity * t, part.Size, destructible.CFrame + destructible.Velocity * t, destructible.Size) then
                    goingToTouch(destructible, part)
                end
            end
        end
    end

    print('did the thing')
end)

return {connect = connect, disconnect = disconnect}

There is a MESS on line 103 that crushes ALL my performance. Commenting out 103-105 gives all the performance back, so I know it's the problem.

I don't even know if this is possible in ROBLOX using Lua. Should I just use Glue joints or something? What should I do?

P.S. The script that requires the ModuleScript uses CollectionService to keep track of the destructible bricks. I don't otherwise include it because it's pretty simple, but if you need to test it just put this into a ModuleScript in ReplicatedStorage called DestructibleCollection, and put the code I posted above in a child of the script (another ModuleScript) called DestructionModule:

local collectionService = game:GetService('CollectionService')
local destruction = require(script.DestructionModule)

for _, part in ipairs(collectionService:GetTagged('destructible')) do
    destruction.connect(part)
end

collectionService:GetInstanceAddedSignal('destructible'):Connect(destruction.connect)
collectionService:GetInstanceRemovedSignal('destructible'):Connect(destruction.disconnect)

return nil

And in a script somewhere else, both on client and server, I use this code:

local replicatedStorage = game:GetService('ReplicatedStorage')

require(replicatedStorage.DestructibleCollection)

Sorry if I left any info out, I couldn't revise this post much as I'm short on time. I'll review it later.