LPMuds.net

LPMuds.net Forums => Drivers => Topic started by: Hilapdatus on January 10, 2018, 02:52:59 PM

Title: Sending large strings through socket efuns
Post by: Hilapdatus on January 10, 2018, 02:52:59 PM
I am dicking around with code which converts LPC constructs to JSON and then uses socket efuns to PUT/POST this content to a document database.   This seems to work great for for smaller constructs but fails miserably as soon as the data size exceeds the value of LARGEST_PRINTABLE_STRING which is 8kB.

I can think of two solutions, perhaps there are more or better solutions?

The first idea is to patch socket_efuns.cc so that socket_write() accepts LPC string arrays.  I'd rather not patch the driver if I can avoid it unless this seems like a reasonable modification?

The second idea is to stuff the JSON data into an LPC buffer type, which is already handled by socket_write(), but building buffers seems kinda klunky if you're not sourcing the data from a file on disk.  Perhaps the read_buffer() efun could/should be modified so as to accept LPC string arrays and return buffer?

Thoughts or opinions?

Cheers!
Hil
Title: Re: Sending large strings through socket efuns
Post by: silenus on January 10, 2018, 09:35:49 PM
I am no expert on the driver but it seems that LARGEST_PRINTABLE_STRING is one of the internal defines which can be increased in size. I am not sure why fallentree separated it out in this manner. It seems that it just governs the size of 3 char arrays (two in simulate.cc and one in outbuf.cc).

Maybe you could first try increasing the size of this? Seems a bit easier than the other two options.   
Title: Re: Sending large strings through socket efuns
Post by: silenus on January 10, 2018, 10:09:22 PM
It seems that the arrays are also all stack allocated as local variables.
Title: Re: Sending large strings through socket efuns
Post by: quixadhal on January 11, 2018, 09:02:03 AM
The LPC way is not to just make things bigger, but break the problem up into small enough chunks to manage.

There'e no reason to keep the entire document web in a single JSON object.  Break it up by category, keyword, pathname, however you like.

I would probably just make the web service accept a pathname as an argument for the POST, and then the document object as a second argument.  That way the web server can place things into file paths, or store them in SQL, or whatever you want to do on that end.  Retrieval would simply be by the pathname.  The only extra legwork would be providing an index to map pathnames to all topics.

You can still load/save the whole document archive, but you just may need to do it in a loop, perhaps across eval ticks via call_out() if needed.
Title: Re: Sending large strings through socket efuns
Post by: silenus on January 11, 2018, 05:43:41 PM
Well nowadays given how large memory is and the bandwidth available with networks I don't see it being much of a problem increasing the 8k size to something a bit higher if one happens to need it.
Title: Re: Sending large strings through socket efuns
Post by: Hilapdatus on January 12, 2018, 07:08:59 PM
Thank you for the input.