[math] How can I make vectors that satisfy these dot products?

The first thing to notice is that there are two angles involved here. They are easy to derive due to the dot product equality below, where a is a vector, b is a vector, and theta is the shortest angle between the two vectors in radians:

dot(a, b) = cos(theta)*mag(a)*mag(b)

Rearranged as:

theta = acos(dot(a, b)/(mag(a)*mag(b)))

The two angles are the angle between v and x (alpha) and the angle between v and y (beta). However, since v, x, and y are unit vectors and thus all have a magnitude of 1, we can simplify the equality above to the following:

theta = acos(dot(a, b))

Therefore...

alpha = acos(dot(x, v)) and beta = acos(dot(y, v)).

The dot product of x and v and the dot product of y and v are already defined in the problem as scalars a and b, respectively. Therefore...

alpha = acos(a) and beta = acos(b).

This problem can now be easily visualized as trying to find some x and some y such that they are both angled alpha and beta away from v, respectively, while still maintaining the fact that they are orthogonal (notice that the dot product of x and y is 0, so the shortest angle between them is pi/2).

Before getting too lost, it is important to note that one of the most powerful tools in mathematics is solving a simpler problem first.

This is a problem that is easy to solve in 2D. Given some 2D vector, we can apply a simple linear rotation transformation using the transformation matrix T(theta) below, where theta is the rotation in radians measured counterclockwise...

[ cos(theta) -sin(theta) ]
[ sin(theta) cos(theta) ]

Given v in 2D, there are two solutions.

1. x is T(-alpha)*v and y is T(beta)*v
2. x is T(alpha)*v and y is T(-beta)*v

Since we are only looking for a single solution, I will use the first one for simplicity.

Now that we know that it can be easily solved in 2D, we just need to be able to cast from 3D to 2D and back in order to solve this problem for 3D.

This can be done by creating a basis to cast in between the two dimensions using the Gram-Schmidt process on two orthogonal vectors. We can achieve this by letting v be the x axis and picking some new vector c that is orthogonal to v to be the other axis of the basis. Picking c is as easy as choosing a new vector that is not parallel to v (otherwise you would get 0!) and computing the cross product.

Then you're done! :)

Here is some old (but relevant) Lua code for projecting from 3D to 2D and back.

function scalarProjection(v, a)
	-- the scalar projection of vector `v` onto vector `a` is defined as the 
	-- dot product of v and a divided by the dot product of a and a
	-- (v*a)/(a*a)
	local result
	if not pcall(function() -- if Vector3
		result = v:Dot(a)/a:Dot(a)
	end) then -- else Vector2
		result = (v.x*a.x + v.y*a.y)/(a.x*a.x + a.y*a.y)
	end
	return result
end

function projection(v, a)
	-- the projection of a vector `v` onto a vector `a` is defined as the
	-- scalar projection of vector `v` onto a vector `a` multiplied by the
	-- vector `a`
	return scalarProjection(v, a)*a
end

function projectVectorOnPlane(v, a, b)
	-- project vector `v` onto the plane defined by the cartesian product of 
	-- `a` and `b`
	-- this is the same problem as projecting a vector `v` onto a vector space, 
	-- just in 3D
	-- to project a vector `v` onto a vector space `s`, calculate the sum of
	-- the projections of `v` onto each vector of a basis of `s` that spans `s`
	-- the basis of `s` in this case is effectively {`a`, `b`}
	-- therefore, the projection of `v` onto `s` is 
	-- (v*a)/(a*a)*a + (v*b)/(b*b)*b
	-- or simplified as 
	-- projection of v on a + projection of v on b
	return projection(v, a) + projection(v, b)
end

function newBasis(origin, crossAxisPoint, axisPoint)
	-- create a new 2D orthogonal basis using axisPoint - origin as one vector 
	-- and then deriving the cross axis from another point lying on the plane

	-- points relative to basis origin
	local axisPointFromOrigin = axisPoint - origin
	local crossAxisPointFromOrigin = crossAxisPoint - origin

	-- picking main axis
	local axis = axisPointFromOrigin.unit

	-- deriving cross axis 
	-- the cross axis is defined as the vector minus its projection onto the
	-- main axis
	local crossAxisProjection = projection(crossAxisPointFromOrigin, 
									axisPointFromOrigin)
	local crossAxis = (crossAxisPointFromOrigin - crossAxisProjection).unit

	return {origin = origin, axis = axis, crossAxis = crossAxis}
end

function projectWorldToBasis(point, basis)
	-- project a world space point onto a 2D orthogonal basis
	-- calculate the projection of a point onto our 2D basis, which is a plane
	-- then, convert to our 2D basis by projecting the projection onto the axis
	-- of the plane, which we calculated through `newBasis`
	
	-- point relative to basis origin
	local pointFromOrigin = point - basis.origin

	-- project the point onto the 2D basis
	local pointOnPlane = projectVectorOnPlane(pointFromOrigin, basis.axis, 
							basis.crossAxis)

	-- calculate scalar projection values
	local x = scalarProjection(pointOnPlane, basis.axis)
	local y = scalarProjection(pointOnPlane, basis.crossAxis)

	-- construct new 2D vector with calculated values
	return Vector2.new(x, y)
end

function projectBasisToWorld(point, basis)
	-- essentially inversing all of the steps done in projectWorldToBasis
	return basis.origin + point.x*basis.axis + point.y*basis.crossAxis
end