Best way for mob player detection?

The other way could be Raycasting, possibly more efficient and will not lock onto players if they're hidden behind objects.

There are some optimisations you can add if you need better performance, but the rule of thumb is only optimise if you need to. If you haven't encountered a performance bottleneck, then you shouldn't worry about small things like this.

If you have lots of mobs updating their targeting simultaneously, you'll want to avoid performing the same computation over and over again for each of them. Instead, you'll want to cache some data about where players are which will be useful for all of the mobs and make it easier for them, individually, to make a decision.

One way to do this is to divide space into a grid of cells (either 3D or 2D, depending on your game). Each cell should be about as big as the detection range (assuming this is the same for every mob). Every time you update which mobs are targeting which players, fill the grid in such that each entry for a cell is a list of which players are in it.

Each mob then only checks whether it should target players either in the cell that it is in or an adjacent cell - if a player is far away from a mob, they won't be checked by the mob because they won't be in an adjacent cell. The mob will only query the distance (or raycast for line-of-sight) to players in nearby cells. Which players are in which cells will have already been computed at this point.

If you don't have lots of mobs, though, (at least a few hundred!) then this won't make much difference.

Any such optimisations really depend on the nature of your circumstances. Just how many mobs are we talking about, 2D or 3D, how often do they update, their assumptions about the world, etc...

For a similar, even more sophisticated technique, check out quadtrees and octrees.