In order to remove any user from granted permission group, we need to follow below 3 steps. 

1. Remove the user from corresponding granted permission group  
2. Remove the user from Global Permissions group
3. Remove the user from relevant workflow group (if the user belonging to the payroll team)