How do I fix issues of moving a model using :SetPrimaryPartCFrame()?

Hi all,

I'm having a go at making my own little placement system but I've instantly run into a problem. When I move the model the PrimaryPart (an invisible bounding box) teleports to the correct place but the rest of the model is literally 1000 studs above where it should be. What is causing this to happen?

local mouse = game:GetService("Players").LocalPlayer:GetMouse()

local item = game:GetService("ReplicatedStorage").wardrobe:Clone()
	local primary = item.GridPart
local plot = game:GetService("Workspace").Baseplate

local function move()
	local target, side = mouse.Target, mouse.TargetSurface
	local position = CFrame.new(mouse.Hit.X, primary.Size.Y/2, mouse.Hit.Z)
	
	if target == plot then
		print(target.Name)
		item.Parent = game.Workspace
		item:SetPrimaryPartCFrame(position)
		wait()
	else print("nothing")
		item.Parent = game:GetService("ReplicatedStorage")
	end
end

while true do wait(1)
	move()
end