How do I make the block stop growing and then fade away until it is invisible?

Hello, it seems you are new to scripting. I recommend reading the tutorials on the dev wiki for Roblox. https://developer.roblox.com/en-us/

Anyway, I'm going to answer your question now.

You're using a while loop. Because you made the parameter to make the loop run to wait(), this loop will run forever. You should use a for loop. This allows you to run a certain snippet of code a set amount of times.

https://developer.roblox.com/en-us/articles/Loops

Now, to fix your code.

nuke = script.Parent
pos = nuke.CFrame

nuke.Size = nuke.Size + Vector3.new(3,3,3)
nuke.CFrame = pos
wait(5)
-- i think you were trying to make the above code run, then to fade it away.
-- naked code is a bad practice, look into functions and wrap one around the above code!

for i = 0, 1, 0.1 do -- a for loop counts up from a specified number to a specified number, by a specified amount.
	nuke.Size = nuke.size + Vector3.new(3,3,3) -- not sure why you put "cancel" here but i removed that.
	nuke.Transparency = i
	wait(0.1) -- im not sure how fast you want this loop to run, so change this number to anything you want. this is based on seconds.
end 

I hope this helps you!