I only want them to be able to target users and I read that its !%user but that doesn't work.
!%user means NOT user. ! = not.
That's why it's not working.
Roastchicken is correct, !%<group above user that is inherited> would work.
To specify a group only, use #
#user
Pretty sure that's explained in our FAQ (but too lazy to go review right now)