How to make a Part that only collides with one other Part?

One possibility, if the total number of parts that need these complex interactions is low, is to run an invisible "simulation" elsewhere in your place and update the "real", visible parts in real-time.

Let's take your example. In one simulation, you would have Part1 and Part2, allowing them to interact normally. In a second simulation you'd have Part3 and a second Part2. (All parts would have CanCollide = true and be invisible). These simulations would have to be contained so that they don't interact with each other. Perhaps they could be several thousand studs apart. Now, Part1 through Part3 are supposed to exist and be visible to the players, so you'd have a clone of each part that is Visible and Anchored; you'd have to update the CFrame of the visible parts based on the CFrames of the invisible ones (of course you'd have to add an offset to get from the centre of the simulation to the centre of the "play area", where the players actually see Part1 through Part3).

I am unsure what you could do if you wanted players to be able to physically collide with some of these parts (ignoring Part2 since it doesn't move anyway); perhaps you could have a clone of the player in the simulation -- it would probably look a bit weird since it might take, say, 0.1 seconds before a Part responds to a player pushing it, and if the Part was already moving, the part might not go in the direction the player would expect.

Further, if you wanted to add more complex interactions (parts that collide with each other but with exceptions), you may have to run numerous simulations that may need the results of each other. ex, say we added Part4 which collides with Part1, Part2, and Part3, you'd have to run a simulation with Part1, Part2, and Part4 and another simulation with Part2, Part3, and Part4, and the Part4s of the two simulations would have to be "tied" together somehow, so that a change of one Part is transferred to the other. I suppose that if you could figure out how to "tie" these together, you'd solve the problem of letting players interact with the bricks in the "play area", since you could just have the simulated parts "tied" to the visible ones.