Trouble with 3D sound?

So I am working on a horror game project and I am trying to get the 3D sound right, expect I can't seem to. Every single brick that has a sound playing from it works very odd. When I am facing away, the sound plays very loudly. But when I am looking directly at it, you can't hear anything at all no matter how close or how far away you are. It really kinda ruins the environment.

If Soundscape has anything to do with it this is what I have it set to:

AmbientReverb: Hallway DistanceFactor: 1 DopplerScale: 1.5 RolloffScale: 1