Generally my policy is unless there is a REALLY good reason, all data in CRM should be shared. However, on rare occasion, there is a REALLY good reason.
When you have one special group who want to hide their, say, Activities from everyone else, a well-placed Business Unit will do the job (either have one Business Unit above the other, or assign two different roles to the Users who are separated by two Business Units. However, when you have two special groups who want to hide their records but see the generally available records, things get trickier. A hierarchy of Business Units will not work because you can only have one parent Business Unit, not two. Also, Security Roles do not quite get us there because the Users can either see all records in the system or only theirs.
One option is to use a Team. The great thing about Teams is they belong to a Business Unit but Users from anywhere can be added and suddenly gain the benefits as if they were in the Team’s Business Unit. So, in our example, we could use three Business Units, all children of the primary Business Unit of the system; two for the special groups and one for the general population. We have one base Security Role for all Users which allows access to the Activities (or whatever entity is being restricted) in their Business Unit only. We then have a Team on the general population Business Unit with a role with the same privileges as the User role (but not the same role as per Tip 677).
As long as the special group Users get added to the team on creation (either via an automated process or as part of the process for adding new users), they get to see their records and the general populations but no one else’s.