How to properly use and understand Raycasting?

I greatly suggest re-reading the Wiki, and developing a basic understanding of vectors if you do not understand. If you don't understand something on the Wiki, try asking a very specific question, not just saying it was "no help at all". Here's some helpful links:

https://devforum.roblox.com/t/how-to-think-about-vectors/11866

https://www.mathsisfun.com/algebra/vectors.html

https://www.khanacademy.org/math/precalculus/vectors-precalc

http://wiki.roblox.com/index.php?title=Making_a_ray-casting_laser_gun

http://wiki.roblox.com/index.php?title=Ray

http://wiki.roblox.com/index.php/Raycasting

http://wiki.roblox.com/index.php?title=API:Class/Workspace/FindPartOnRayWithIgnoreList

The key point that may be confusing is that though Ray.new takes a "direction vector" as a second argument, its magnitude is actually relevant. Workspace:FindPartOnRay() and other methods like it will only search for intersections as far as the magnitude of the ray's "direction vector" (up to a maximum of 5,000 studs). So, for example, you'd get the unit vector of the direction to raycast in, and scale it by the maximum number of studs you want the gun to be able to do damage in.

It is imperative to becoming a good scripter that you develop a basic geometric understanding of vectors.

EDIT: In response to comments on the answer

When you don't know what arguments a method takes, it's always a good idea to open the documentation and check. For example, we could use the Wiki search or a search engine of your choice with the terms "roblox wiki ____" or "roblox api ____". So let's look at the arguments of Workspace:FindPartOnRay():

http://wiki.roblox.com/index.php?title=API:Class/Workspace/FindPartOnRay

This method is used to find the first point of intersection on the ray from the start in a specified direction. The length in studs it will test for is based on the length of the direction vector passed as a second argument to Ray.new(). The maximum number of studs from the start in a direction a single ray can test for intersections on is 5,000.

The first argument it takes is the ray to search for intersections on. This is self-explanatory, you just pass it the ray object returned by the Ray.new() constructor. This is the only necessary argument, all the others are optional, as we can see on the Wiki documentation page they have default arguments if you pass nothing in.

The second argument is the instance to ignore descendants of when searching for intersections. This means that the instance and all of its children, and it's childrens' children and so on, will be ignored when trying to find the first point of intersection.

The third argument determines whether or not terrain cells should be treated like cubes regardless of their shape and orientation as the Wiki states in the usage section. In the function call you described, they would be treated as cubes.

There is a final fourth argument that determines whether or not the ray intersection test should ignore smooth terrain water.

So to summarize, this call local part, position = workspace:FindPartOnRay(ray, player.Character, false, true) will attempt to find insersections on the ray ray, ignoring and descendants of player.Character, not treating terrain cells as cubes, and ignoring smooth terrain water.

It's worth noting that there also exists a method Workspace:FindPartOnRayWithIgnoreList() which is sometimes useful for ignoring the local player's character among other things. This is different from Workspace:FindPartOnRay() because we can pass an array in, rather than only one instance.

A simple constructor for a ray:

-- LocalScript inside tool

local tool = script.Parent
local plr = game:GetService("Players").LocalPlayer

tool.Equipped:Connect(function(mouse)
    mouse.Button1Down:Connect(function()
        ray = Ray.new(tool.Handle.CFrame.p, (mouse.Hit.p - tool.Handle.CFrame.p).Unit * 300)
    end)
end)