If the total recipients is not changing when you add groups then they are already selected.
If you wish to remove all contacts first and then re-select a group or groups then go via the select all contacts manually route and then remove all contacts:
You can then re-select the group(s) you need.