After you've created a smart group, you have the option to exclude specific users from the group. This is useful for users that you're not able to filter out with the provided filters.
To exclude specific users, use the "x" next to the users email.
After you've excluded the users necessary, you will see them reflected under the "exclusions" tab of the smart group. 