How do I make a hole in floors without solid modelling?

The Hole Instance is just a decorative feature, a small black circle. They don't actually cut into objects.

In order to make complex cuts into parts, your best option is solid modeling (though this cannot be done during game execution -- only in Studio).

Otherwise, you have to manage the geometry yourself.

For Axis Aligned Boxes (AAB), this is relatively simple, since the result is more AABs. With more knowledge, like that the surface is actually flat (like a floor) this is even simpler.

A voxelized approach, however, may be sufficient -- split the surface into a grid of dots and eliminate those that are unnecessary. To avoid excessive lag, you could keep this voxel memory only in Lua, but display merged parts wherever there is contiguous flooring.