Author Topic: proposal for multi-threaded comm changes.  (Read 2141 times)

Offline FallenTree

  • BFF
  • ***
  • Posts: 485
    • View Profile
proposal for multi-threaded comm changes.
« on: March 04, 2013, 05:35:16 pm »
Hi,

there has been some effort in Chinese fluffos forks to utilize epoll instead of select, where I think it's not solving the root problem. The root problem is that all communicates are handled in main thread.  takes precious time for your main mud loop have to finish each heartbeat.

I've been thinking of a way to separate them. so far I think what can be done is:

1) run a libevent dispatch loop in separate thread for all socket communicates and compression etc.  The libevent loop would send out/receive stuff as needed and saved in its own datastructure waiting for main loop to get it.  It's like a messaging queue but actually can be just implemented as lock-protected linked-list, or something like that.

2) main loop still calls proceessio() routinely,but it is merely memcpy back-and-forth the data main loop (interactive_t) and data structure in the communication thread. This can be implemented with locks, but I rather look for a lock-free solution.

The idea is to make minimal changes to current code to first move communication stuff to separate thread first, and no behavior changes.  then when that is done and tested and trusted,  try to unify the two data structures so that no memcpying is needed.

I would like to hear your opinions.

Offline vorpal

  • Acquaintance
  • *
  • Posts: 14
    • View Profile
Re: proposal for multi-threaded comm changes.
« Reply #1 on: March 04, 2013, 07:30:56 pm »
How much latency is process_io() causing in the backend loop, excluding the LPC callbacks it makes?  How many sockets are being managed, where scalability of select() is a problem?

I ask, because while I definitely appreciate the benefit of offloading as many CPU cycles as possible from the main thread, the changes required to do this seem fairly significant at first glance.

Offline wodan

  • BFF
  • ***
  • Posts: 434
  • Drink and code, you know you want to!
    • View Profile
Re: proposal for multi-threaded comm changes.
« Reply #2 on: March 04, 2013, 07:35:01 pm »
How big are those chinese muds? This has never been an issue for dw, not even with 225 players on hardware from around the year 2000!

I suspect epoll would fix any real comms slowness as the actual communications take hardly any time at all and select is known to be slow for large amounts of sockets, but you're talking well over a 1000 players before that should be an issue, are they really that popular?

Offline FallenTree

  • BFF
  • ***
  • Posts: 485
    • View Profile
Re: proposal for multi-threaded comm changes.
« Reply #3 on: March 04, 2013, 08:20:12 pm »
I think it can be around 300+, even more and that many is usually because of mulitple connections each user. (robot).

I agree change select to epoll doesn't seems to be solving the real problem, at least not seems to, but multi-threading would certainly be beneficial.    Especially since there is now compression/iconv involved.

Offline wodan

  • BFF
  • ***
  • Posts: 434
  • Drink and code, you know you want to!
    • View Profile
Re: proposal for multi-threaded comm changes.
« Reply #4 on: March 05, 2013, 04:41:14 am »
300 shouldn't be a problem at all for select(), or any of the other code in the driver, how do they know it's actually the sockets that are slowing them down?

Offline quixadhal

  • BFF
  • ***
  • Posts: 642
    • View Profile
    • WileyMUD
Re: proposal for multi-threaded comm changes.
« Reply #5 on: March 05, 2013, 08:44:40 am »
Their problem isn't due to threading.

Considering that they're trying to run antique mudlibs, it wouldn't surprise me if they're also using underpowered hardware, but more importantly, using OLD versions of linux/unix.  Once upon a time, there was indeed a limit of 256 selectors per socket set, as that field used to be an unsigned char.  I think that was changed over 10 years ago though, and AFAIK, the only practical limit now is the number of sockets you can have (64K -- because port numbers are still limited to 16-bit values).

I don't know the internals of FluffOS, so I don't know what can of worms you're opening by introducing multi-threading at that level.  Since FluffOS's LPC isn't itself multi-threaded (unlike DGD's Hydra), I suppose it wouldn't be too ugly as long as you were careful about mutex-locking the code that's fetching socket input and dumping it into the processing buffers (and likewise for output).

Be aware also that different OS's handle threads very differently.  In linux, a thread is implemented as a full process, which means you'd want to start your threads very early in the startup process to avoid having to copy large amounts of data structures.  Windows may be a whole different ballpark.  Cygwin, OSX, and whatever else people still try to use may also be a bit different.

In any case, you'll probably need to make it optional because people running MUDs on embedded ARM systems won't appreciate having the extra overhead when they're probably already pushing it. *grin*

Oh, and I don't see a way to handle this without locks.  Remember, you can be in the MIDDLE of a TELNET sequence after you've flushed a socket buffer, and you really don't want to process stuff without locking if that's the case.

Offline FallenTree

  • BFF
  • ***
  • Posts: 485
    • View Profile
Re: proposal for multi-threaded comm changes.
« Reply #6 on: March 06, 2013, 03:42:42 pm »
the locking could be handled differently if proper care is taken on reducing shared data structure. Or it could even be offloaded if a message-queue like thing is in use.

Select model have weird effect on things, it is a limit, you are totally right things are good enough when there is low number of users, but it just doesn't scale, you can argue there's no scaling needed, but counter argument is since you can't scale, people never make interesting use of it, because they can't.

Offline quixadhal

  • BFF
  • ***
  • Posts: 642
    • View Profile
    • WileyMUD
Re: proposal for multi-threaded comm changes.
« Reply #7 on: March 07, 2013, 06:47:14 am »
You could always do it the way the big boys do (commercial MMOs).  Use a proxy.

User connects to foo.com 3000.  That's a proxy server which then redirects the connection to foo.com 5000 + random(50).  MUD is actually listening on all 50 ports from 5000 to 5049, but treats them all the same in terms of what happens.. IE: they all make a call to the master object to get the same mudlib connection object.

Result?  You are now using 50 seperate select calls, thus reducing each one's workload.  If select really is a problem with several hundred clients, that will solve the problem, no?

I think you can do this with FluffOS as it stands... if you can specify an arbitrary number of ports for it to open.  And a ruby/python/perl forwarder should be pretty easy to find/code.

Offline Dworkin

  • Acquaintance
  • *
  • Posts: 34
    • View Profile
Re: proposal for multi-threaded comm changes.
« Reply #8 on: March 07, 2013, 06:37:05 pm »
I don't know the internals of FluffOS, so I don't know what can of worms you're opening by introducing multi-threading at that level.  Since FluffOS's LPC isn't itself multi-threaded (unlike DGD's Hydra), I suppose it wouldn't be too ugly as long as you were careful about mutex-locking the code that's fetching socket input and dumping it into the processing buffers (and likewise for output).

Hydra on Linux uses epoll_wait(), but for a single-threaded server select() can be more efficient because it can collect multiple events per call. The higher the activity/connections ratio, and the longer LPC execution takes (making it more likely that input events will be grouped together), the more efficient select() becomes.