Way you have it is the basic idea, but, you need to use lower case when comparing groups. "user" not "User"
A better way to do it, since some groups inherit lower ones, is probably test for the "respected" specifically.
Also, in 99% of Gmod lua, when you wish a hook function to continue, don't return "true", just return.
There are exceptions, but, returning anything will often break the hooks of other mods/scripts using the same hook, even if it is only 'true'
(Yes, I know, the Gmod wiki is full of 'return true'...many hooks break others that way.
function GM:PlayerJoinTeam( ply, class, teamid )
if not ply:IsUserGroup("respected") and teamid == TEAM_SWAT then
return false
else
return
end
end
In english, the above statement reads
if player is not in group respected and they want to join Swat team, do nothing, else if they are in group respected and want to join swat, go ahead.