Sending large files from server to client

[Timmy]

Context
I’m working on a module that displays ULX logs in XGUI.

Problem
• Active servers can have large log files (up to several megabytes).
• All data transfers in Garry’s Mod are throttled to 20 kb/s.
• A Net message can only contain up to 64 kb of data.

Current approach
Send compressed contents with xgui.SendDataEvent
+ Simple
- Fails when dealing with large log files

Alternative approaches
"Load more…" button
Send a snippet. Make user press button to load the next ~64 kb.
+ Fast, works with large files
- Clutters UI, inconvenient when dealing with large files

Send in chunks
Split contents into small chunks. Send all the chunks.
+ Works with large files
- Very slow and intense lag when dealing with large files

Remote server
Upload log to a remote server.
+ Access to all of the contents, fast
- Server can go offline, client may not have internet access, access to host may be blocked
Are there any other approaches I should consider? How would you send large files to users?

[JamminR]

Though still a type of "remote server", don't host remotely.
I just came up with this idea,and apparently other people have already made it happen.
Lua web server. (Does Gmod Awesomium limit file size?
Several exist.
https://www.google.com/search?q=lua+web+server
I wasn't overly surprised, as I use a Python CherryPy/Mako based web server, but, yeah, Lua.

[Timmy]

Thanks, JamminR! I appreciate your input.

Though none of those web servers are written in pure Lua. They all require binary modules written in C/C++. I don’t believe it is possible without those.

There is a networking module for Garry’s Mod written by a third party called Bromsock.

I expect this would be fast and work with large files.

But this would reduce accessibility as server owners would have to manually install (and update) the binary module. They would also have to update their firewall to allow traffic on an additional port.

I would also be relying on the maintainer’s ability to keep the project up-to-date and secure.

[JamminR]

Welcome. I didn't look that deep - figures - I'd hoped something would be available in pure lua.
Loading binary in this instance is cheating!

[MrPresident]

You could use an approach I used for sending screenshots from the client to the server. Should work backwards.

Take the log file and convert it to binary, then break it up into chunks and send a net message to the user with a header of the current chunk and the total chunks. Then have the client wait until it receives all the chunks and put them back together.

[Timmy]

Thanks for your input! I’m going to experiment with chunked transfers a bit more.

Garry’s Mod also performs a chunked transfer when sending save files to clients.

First few tests are… interesting. It turns out that you can only send 262112 bytes per tick. I am definitely going to have to limit the amount of data I send in each frame so I don’t overflow the buffer.

[Stickly Man!]

Although not very elegant, XGUI's data transfer system can be set up to send chunks of data, much like Mr. President is describing. I think I only used it for bandata because some servers had so many bans- but we've since changed the bans to be queried on demand for performance reasons.

[Timmy]

Oh, neat! I didn’t know that. I’ll be sure to take another look at the XGUI code related to data transfers. Thanks!