how do i use atan2 so that it works in a 3d space?

To get the dot product of two vectors, take the sum of the products of corresponding coordinates.

(x_1, y_1, z_1) · (x_2, y_2, z_2) = x_1 * x_2 + y_1 * y_2 + z_1 * z_2

To find the magnitude (|u|), we can just use .magnitude

Using these two operations, we can apply the equation...

cos x = (u · v) / (|u| |v|)

...and then rearrange...

x = acos ((u · v) / (|u| |v|))

...to get our answer.

function dotProduct(u, v)
    return u.x * v.x + u.y * v.y + u.z * v.z
end

function angle(u, v)
    local top = dotProduct(u, v)
    local bottom = u.magnitude() * v.magnitude()
    return (top / bottom)
end