I've been testing with a ~200kb bans.txt file.. Most of those don't have much data associated with it, but so far it's still working. Do you have an approximate count of bans before splosion happens? There is the "View Source Bans" button on the bans tab that allows you to see those bans still remaining in the banned_users.cfg, as well.
Unrelated, but I did manage to make some improvements to XGUI load stuff when the server is under heavy load during startup!