Why do only some parts of a player character change with CollisonGroup?

So basically, my goal here is to make it so that once a player joins the game, the game will automatically set the player character's collisongroup to the custom one I made "PlayerGroup", which has the ID 1. However, whenever I playtest the game on studio, only the player's head and RootPart has a collisiongroupID of 1. The unions, accessories, and everything else has a groupID of 0, yet when I print the part's ID in the script it prints out a 1. I'm just not sure what's going on here, or how to fix it.

local Players = game:GetService("Players")
local PhysicsService = game:GetService("PhysicsService")

local PlayerGroup = "Player"
PhysicsService:CreateCollisionGroup(PlayerGroup)


local function setCollisionGroupRecursive(object)
    if object:IsA("BasePart")then
        PhysicsService:SetPartCollisionGroup(object, PhysicsService:GetCollisionGroupName(1))
        print(object.CollisionGroupId)
    end
    for i, child in ipairs(object:GetChildren()) do
        setCollisionGroupRecursive(child)
    end
end


local function onCharacterAdded(character)
    setCollisionGroupRecursive(character)
end

local function onPlayerAdded(player)
    player.CharacterAdded:Connect(onCharacterAdded)
end

Players.PlayerAdded:Connect(onPlayerAdded)