How would I detect GUI collision WITH the rotation property?

Two rectangles, A and B overlap if either

• A is entirely inside B
• • All corners of A are in B
• B is entirely inside A
• • All corners of B are in A
• At least one side of A intersects with at least one side of B

(These conditions are actually sufficient for any convex shapes, but only easy to calculate for convex polygons, and particularly so for rectangles)

Thus, if we can turn the GUI objects into rectangles, then we can solve this problem.

Defining a rectangle by four corners, a b c and d (as Vector2s) we can write a simple function for determining if another point x is in the interior of the rectangle.

function inInterior(a,b,c,d, x)
    local BA = b - a
    local DA = d - a
    local alongBA  = BA.unit:dot(x - a)
    local alongDA  = DA.unit:dot(x - a)
    return
        alongBA >= 0 and alongDA >= 0
        and alongBA <= BA.magnitude
        and alongDA <= DA.magnitude
end

The idea of this function is that using dot products, we define x in terms of the cartesian space defined by the rectangle's orientation, and then check that the coordinate is within the axis-aligned rectangle defined by the rectangle's width and height

Now, to determine if two segments are intersecting, we can check that each endpoint is on the opposite side of the other segment.

-- Two segments AB and CD
-- Are C and D on opposite sides of AB?
function opposite(a,b, c,d) 
    local N = Vector2.new( b.y - a.y, a.x - b.x )
    local DA = d - a
    local CA = c - a
    DA = DA:dot(N)
    CA = CA:dot(N)
    -- If DA and CA are of opposite sign...
    return DA + CA < math.abs(DA) + math.abs(CA)
end

So to check if two segments are intersecting, just apply this:

function intersecting(a,b,c,d)
    return 
        opposite(a,b, c,d) and opposite(c,d, a,b)
end

To determine if two rectangles are intersecting, then, we just need to check this pairwise on all of the sides:

function rectangleIntersecting(a,b,c,d, e,f,g,h)
    -- rectangles ABCD and EFGH
    -- Do any of their sides overlap?
    return
        intersecting(a,b, e,f)

end

To determine if one rectangle is in the other,

function rectangleInside(a, b, c, d, e, f, g, h)
    return
        inInterior(a,b,c,d, e)
        and inInterior(a,b,c,d, f)
        and inInterior(a,b,c,d, g)
        and inInterior(a,b,c,d, h)
end

Putting it all together, to determine if two rectangles are colliding,

function rectanglesColliding(a,b,c,d, e,f,g,h)
    if
        rectangleInside(a,b,c,d, e,f,g,h)
        or rectangleInside(e,f,g,h, a,b,c,d)
    then
        return true
    end
    return rectangleIntersecting(a,b,c,d, e,f,g,h)
end

The only remaining step is to get the corners from a GUI. This is relatively easy if the rectangle is given in terms of offset only.

(Use something like x +- w/2 * cos(rotation) and y +- h/2 * sin(rotation))

For now, though, I will leave that as an exercise to the reader.