I use the fretta map-vote and map changes are smooth -- usually only takes a few seconds, but I also only have 57 small-ish workshop addons and a few legacy addons.
The amount of addons is not AS important as the size of each of those addons. Say you have 100 addons at 1 KB each, and another server with 100 addons at 1 MB each. The server with the smaller sized files will load faster because it can load through each addon faster.
The amount of addons on your server and the size of each of the addons is causing the server to take longer to mount all of the files.
Error loading cfg/trusted_keys_base.txt
Error loading cfg/pure_server_minimal.txt
Those could refer to a map installed incorrectly (as refered to in
this thread on Facepunch) or something else. If nothing is wrong with those, it very well could just be the amount of large addons you have on the server.