I keep getting "attempt to call a nil value" error?

The problem is very simple. You don't return the function itself. You returned the table instead.

local module = {}

local function OnRenderStepped(deltatime)
    print("RenderStepped")
end
return OnRenderStepped

-- from localscript 
local PlacementModule = require(script.PlacementModule)
game:GetService("RunService").RenderStepped:Connect(PlacementModule)

Though of course you didn't want that. You wanted to return the table with its functions and such.

local module = {}

function module.OnRenderStepped(deltatime) -- function is now part of the table
    print("RenderStepped")
end

return module

-- from localscript
local PlacementModule = require(script.PlacementModule)
game:GetService("RunService").RenderStepped:Connect(PlacementModule.OnRenderStepped)

On a side note, :connect() is deprecated, use :Connect() instead.