Punch script only allows me to punch at one angle?

I'm working on a fighting game that involves a lot of punching - recently I began to dabble in RayCasting as I figured it would make things run a lot smoother, so I whipped this up real quick based on some research I did:

-- locals, again

local ReplicatedStorage = game:GetService("ReplicatedStorage")
local punchdamageEvent = ReplicatedStorage.Events:WaitForChild("PunchDamageEvent")
local punchAnimScript = game:GetService("StarterGui"):WaitForChild("punch")

local HitReg = false
dmg = 20
CanDmg = true
minimumhealth = 20

-- punchy lad
local function onPunchDamageFired(Player)
	local rightArm = Player.Character:FindFirstChild("Right Arm")
	local rayOrigin = rightArm.Position
	local rayDirection = rightArm.Position - Vector3.new(0,1,0)
	local raycastParams = RaycastParams.new()
	raycastParams.FilterDescendantsInstances = rightArm
	raycastParams.FilterType = Enum.RaycastFilterType.Blacklist
	local raycastResults = workspace:Raycast(rayOrigin, rayDirection,raycastParams)
	
	if raycastResults then
		local hitPart = raycastResults.Instance
		
		if hitPart.Parent:FindFirstChild("Humanoid") and CanDmg == true then
			hitPart.Parent.Humanoid:TakeDamage(dmg)
			CanDmg = false
			wait(1.5)
			CanDmg = true
			print 'punch succeeded'
		end
	end
end
punchdamageEvent.OnServerEvent:Connect(onPunchDamageFired)

The script itself works just fine - I am able to hit a test dummy and damage it as intended. The only problem is, I can only attack the dummy at a certain angle. Changing the "rayDirection" does not effect anything, I still have to be at the same side of the Dummy, in the right spot. I assume I forgot something within the Ray portion of the script - but I couldn't find a solution online.