Physics Glitches with Model Containing Humanoid?

So, I am currently making a truck for a game that automatically moves to certain locations at a certain speed (like the truck moves to a part). However, I've been going through many physics issues that makes it highly glitchy and unreliable. For my current method I am trying, I know that disabling collisions would be an easy fix, but I want them left on.

The issue I'e been going through is that it works just fine. However, the player can simply move against it/push it, causing the truck to move EXTREMELY fast unnaturally.

Image: https://gyazo.com/405251bce3708ae7b1b48b9723315728

The truck has a humanoid inside of it and all of the parts are welded.

How would I fix this issue, or are there any alternatives for this? I know I can use :SetPrimaryPartCFrame on the model and lerp it, but it doesn't look visually pleasing to me.

https://gyazo.com/7db508324cc5791dcf9a653223e67525