Both operator and guest inherit from user, so you do have branching. I recommend specifying what users can target, instead of what they can't. For example, for "VIP" you might set can_target to "#A,#B,#C,#D". Using "#" instead of "%" ignores inheritance, so you're specifying groups explicitly.
That's a fair amount of information condensed into a few sentences (very few people complain that I'm too wordy). Let us know if you need us to expand on anything.