Hi,
It's possible to remove a user from the 'All Users Group' Please refer to the Collab scripting and integration guide (specifically KMAPI in relation to groups). Using this approach may cause you some issues. The problem you'll have is that new users are added to the all user group automatically.
It would be better to create a new group and a new access policy to handle this scenario