It’s impossible to give an answer without seeing the results you are getting.
Please post the different results from mode 2 so I can see if the variation you are seening is normal. Mode 2 should work consistently, provided the background is all the way cutout. If there are small pieces of the background not cutout then it will find those and think it is part of the player. In that case you will need to check the “Use Select Subject For Player Edges” checkbox in the SPA settings. That will then use Select Subject to determine the edges instead of transparency so it will ignore if any small areas of the background are not cutout.