How are you adding users to this custom group. If you are adding them using the settings/users.txt, and have ULib installed, then you may encounter problems trying to use custom user groups. Instead try adding your users to /data/ULib/users.txt. The format is different' but shouldn't be too difficult to understand if you already have some other users in there (admins and superadmins from /settings/users.txt are added there automatically).
If you are using the list in /data/ULib/users.txt, then please check for any errors on startup/connect (both server and clientside), and post your /data/ULib/users.txt & /data/ULib/groups.txt files.