how can i reflect rays for ray tracing?

Reflecting a ray is fairly simple.

To get the reflection angle (Denoted as R) we'll need a surface normal (Denoted as N), which you already have set up it seems, and the incident direction (Denoted as I). Incident direction is basically the angle at which the ray hits the surface, once we have it we can basically just do R = I. We can use the direction of the ray (Denoted as D) along with cosign to get it, like so:

I = cos(D) * N

This is of coarse only 2D, so we need to do it twice for each axis.

Ix = cos(Dx) * N

Iy = cos(Dy) * N

Since D = I then you basically have your ray's direction.

Now, time to convert to lua.

local R, Hit, Position, Normal = CastRay(nil, nil, nil)

local RR.Direction = cos(R.Direction) * Normal

Or something along the lines of that, modify it for your own use

Source

P.S Very exited to see whatever your working on if it means cool shaders like this, good luck. Also sorry about the response time.

https://ibb.co/VVJQRwb this is the image the magenta lines is a line created between first psotion and reflection position

and this is the new Code

local Cache = script.Parent:WaitForChild("Cache")

local Player = game.Players.LocalPlayer
local Character = Player.Character

local PixelSize = 20

local ViewPortSize = Cache.AbsoluteSize
local VSX = ViewPortSize.X
local VSY = ViewPortSize.Y

local Camera = workspace.CurrentCamera

function CastRay(Origin, Direction, Ray_Length)
    local R = Ray.new(Origin, Direction * Ray_Length)
    local Hit, Position, Normal = workspace:FindPartOnRay(R)

    return R, Hit, Position, Normal
end

function Cos(NumberX)
    return math.cos(NumberX)
end

function Render()
    for X = 0, VSX / PixelSize do
        for Y = 0, VSY / PixelSize do
            local F = Instance.new("Frame", Cache)
            F.Size = UDim2.new(0, PixelSize, 0, PixelSize)
            F.Position = UDim2.new(0, X * PixelSize, 0, Y * PixelSize)
            F.BorderSizePixel = 0

            local R = Camera:ScreenPointToRay(X * PixelSize, Y * PixelSize)

            local R, Hit, Position, Normal = CastRay(R.Origin, R.Direction, 500)

            print(Hit)

            if Hit then
                local RR, HitR, PositionR, NormalR = CastRay(Position, Vector3.new(Cos(R.Direction.X), Cos(R.Direction.Y), Cos(R.Direction.Z)) * Normal, 500)

                local Chance = 10

                if Chance == 10 then
                    local Laser = game.ReplicatedStorage.Part:Clone()

                    local M = (Position - PositionR).Magnitude

                    Laser.Parent = workspace

                    Laser.Size = Vector3.new(.1,.1,M)
                    Laser.CFrame = CFrame.new(Position, PositionR) * CFrame.new(0,0,-M / 2)
                end 

                print(HitR, " ReflectNORMAL")
            end

            wait()
        end

        wait()
    end
end

wait(20)

Render()