Along with running the ulx setgroupcantarget command, you can also change the can_target setting within XGUI-- Go to the groups tab, Select "Manage Groups..." in the drop-down box at the top, then click on each group, change the "can_target" entry, then click "Update". This makes it easier to look at the can_target fields of each of your groups.
is there a way to block them from using the commands on all higher groups BUT still be able to use it own members in their group?
This is a bit of a long explanation, but hopefully I can help you understand how can_target works
Say you had the following groups: superadmin, admin, operator, users
If you want each group to target their own group and any groups under them, you'd set their can_target to be this:
superadmin: * (or leave blank)
admin: !%superadmin
operator: !%admin
user: !%operator
To explain, the ! means not, % means group followed by the group name. So you can really just say it out loud for it to make sense:
'ulx setgroupcantarget operator "!%admin"' means "operator can target NOT group admin" (This also means they can't target superadmins, because the superadmin group inherits from admin)
So, before, when you had "ulx setgroupcantarget mod !%mod,^", that meant "mod can target NOT group mod, can target self" (The comma basically just adds another separate parameter)
Anyways, hope that made some sort of sense!