I recommend using the Auto edge, not the fixed pixel amount. The issue is because there are very small areas of green pixels. It could be just 1 o2 2 pixels which wouldn’t be noticed until it is contracted and made larger. For example, if there were 2 green pixels that got erased or partially erased, those 2 pixels becomes 6 pixels wide hole if the edge is contacted by 2 pixels on each side. The auto mode works differently and will often close the hole instead of making it bigger.
So either use auto… or figure out why you have areas with single green pixels to begin with. You may have to pixel peep in the original green screen image. It could be chromatic abberration or some type of light reflection too. If you fix that during the raw conversion then the issue won’t happen during extraction.
Anyway, watch this entire video. I discuss these settings and that effect you see in great detail already in this video.