Author Topic: Sending large files from server to client  (Read 3711 times)

0 Members and 1 Guest are viewing this topic.

Offline Timmy

  • Ulysses Team Member
  • Sr. Member
  • *****
  • Posts: 252
  • Karma: 168
  • Code monkey
Sending large files from server to client
« on: May 12, 2018, 03:20:00 PM »
Context
I’m working on a module that displays ULX logs in XGUI.

Problem
• Active servers can have large log files (up to several megabytes).
• All data transfers in Garry’s Mod are throttled to 20 kb/s.
• A Net message can only contain up to 64 kb of data.

Current approach
Send compressed contents with xgui.SendDataEvent
+ Simple
- Fails when dealing with large log files

Alternative approaches
"Load more…" button
Send a snippet. Make user press button to load the next ~64 kb.
+ Fast, works with large files
- Clutters UI, inconvenient when dealing with large files

Send in chunks
Split contents into small chunks. Send all the chunks.
+ Works with large files
- Very slow and intense lag when dealing with large files

Remote server
Upload log to a remote server.
+ Access to all of the contents, fast
- Server can go offline, client may not have internet access, access to host may be blocked

Are there any other approaches I should consider? How would you send large files to users?
« Last Edit: May 12, 2018, 03:45:21 PM by Timmy »

Offline JamminR

  • Ulysses Team Member
  • Hero Member
  • *****
  • Posts: 8096
  • Karma: 390
  • Sertafide Ulysses Jenius
    • Team Ulysses [ULib/ULX, other fine releases]
Re: Sending large files from server to client
« Reply #1 on: May 12, 2018, 09:01:25 PM »
Though still a type of "remote server", don't host remotely.
I just came up with this idea,and apparently other people have already made it happen.
Lua web server. (Does Gmod Awesomium limit file size?
Several exist.
https://www.google.com/search?q=lua+web+server
I wasn't overly surprised, as I use a Python CherryPy/Mako based web server, but, yeah, Lua.
"Though a program be but three lines long, someday it will have to be maintained." -- The Tao of Programming

Offline Timmy

  • Ulysses Team Member
  • Sr. Member
  • *****
  • Posts: 252
  • Karma: 168
  • Code monkey
Re: Sending large files from server to client
« Reply #2 on: May 13, 2018, 05:37:52 AM »
Thanks, JamminR! I appreciate your input.

Though none of those web servers are written in pure Lua. They all require binary modules written in C/C++. I don’t believe it is possible without those. :(

There is a networking module for Garry’s Mod written by a third party called Bromsock.

I expect this would be fast and work with large files.

But this would reduce accessibility as server owners would have to manually install (and update) the binary module. They would also have to update their firewall to allow traffic on an additional port.

I would also be relying on the maintainer’s ability to keep the project up-to-date and secure.

Offline JamminR

  • Ulysses Team Member
  • Hero Member
  • *****
  • Posts: 8096
  • Karma: 390
  • Sertafide Ulysses Jenius
    • Team Ulysses [ULib/ULX, other fine releases]
Re: Sending large files from server to client
« Reply #3 on: May 13, 2018, 12:13:16 PM »
Welcome. I didn't look that deep - figures - I'd hoped something would be available in pure lua.
Loading binary in this instance is cheating! :P :D
"Though a program be but three lines long, someday it will have to be maintained." -- The Tao of Programming

Offline MrPresident

  • Ulysses Team Member
  • Hero Member
  • *****
  • Posts: 2728
  • Karma: 430
    • |G4P| Gman4President
Re: Sending large files from server to client
« Reply #4 on: May 13, 2018, 12:39:51 PM »
You could use an approach I used for sending screenshots from the client to the server. Should work backwards.

Take the log file and convert it to binary, then break it up into chunks and send a net message to the user with a header of the current chunk and the total chunks. Then have the client wait until it receives all the chunks and put them back together.


Offline Timmy

  • Ulysses Team Member
  • Sr. Member
  • *****
  • Posts: 252
  • Karma: 168
  • Code monkey
Re: Sending large files from server to client
« Reply #5 on: May 15, 2018, 02:51:17 AM »
Thanks for your input! I’m going to experiment with chunked transfers a bit more.

Garry’s Mod also performs a chunked transfer when sending save files to clients.

First few tests are… interesting. It turns out that you can only send 262112 bytes per tick. I am definitely going to have to limit the amount of data I send in each frame so I don’t overflow the buffer.
« Last Edit: May 25, 2018, 10:09:47 AM by Timmy »

Offline Stickly Man!

  • Ulysses Team Member
  • Hero Member
  • *****
  • Posts: 1270
  • Karma: 164
  • What even IS software anymore?
    • XGUI
Re: Sending large files from server to client
« Reply #6 on: June 04, 2018, 09:34:03 AM »
Although not very elegant, XGUI's data transfer system can be set up to send chunks of data, much like Mr. President is describing. I think I only used it for bandata because some servers had so many bans- but we've since changed the bans to be queried on demand for performance reasons. :)
Join our Team Ulysses community discord! https://discord.gg/gR4Uye6

Offline Timmy

  • Ulysses Team Member
  • Sr. Member
  • *****
  • Posts: 252
  • Karma: 168
  • Code monkey
Re: Sending large files from server to client
« Reply #7 on: June 04, 2018, 10:14:50 AM »
Oh, neat! I didn’t know that. I’ll be sure to take another look at the XGUI code related to data transfers. Thanks! ;D
« Last Edit: June 04, 2018, 10:17:37 AM by Timmy »

An Error Has Occurred!

array_keys(): Argument #1 ($array) must be of type array, null given