Author Topic: WIP: Fluffos 3.0 Alpha 9.0  (Read 3952 times)

Offline DarKWateR

  • BFF
  • ***
  • Posts: 104
    • View Profile
Re: WIP: Fluffos 3.0 Alpha 9.0
« Reply #30 on: November 13, 2015, 11:12:24 AM »
That backtrace is from Fluffos 8.1, because i didn't save bt with last 9.0.
Note: Line Numbers from some files (main.cc) aren't exacts because some changes in my own driver version (I still don't send pull request), but i think that crash it is not due for this changes:

Used Command: addr2line -e ./driver "address"
Trace:
Code: [Select]
/home/mud/rlmud/driver/bin/driver(+0xd5415)[0x7f2db9e27415] => src/main.cc:203
/lib/x86_64-linux-gnu/libc.so.6(+0x35180)[0x7f2db74c3180] => rc/vm/internal/base/object.cc:1230
/home/mud/rlmud/driver/bin/driver(+0xd4a40)[0x7f2db9e26a40] => src/net/telnet.cc:220
/home/mud/rlmud/driver/bin/driver(+0xd1f38)[0x7f2db9e23f38] => src/thirdparty/libtelnet/libtelnet.c:813
/home/mud/rlmud/driver/bin/driver(+0xd365a)[0x7f2db9e2565a] => src/thirdparty/libtelnet/libtelnet.c:1052
/home/mud/rlmud/driver/bin/driver(telnet_recv+0x158)[0x7f2db9e14eb8] => ??:0 ???
/home/mud/rlmud/driver/bin/driver(_Z13get_user_dataP13interactive_t+0x571)[0x7f2db9e3bcd1] => ??:0 ???
/usr/lib/libevent-2.1.so.5(+0x1785c)[0x7f2db807085c]
/usr/lib/libevent-2.1.so.5(+0x209ae)[0x7f2db80799ae]
/usr/lib/libevent-2.1.so.5(event_base_loop+0x49f)[0x7f2db807a32f]
/home/mud/rlmud/driver/bin/driver(_Z7backendP10event_base+0x26e)[0x7f2db9e18f0e]
/home/mud/rlmud/driver/bin/driver(main+0x302)[0x7f2db9d71d42]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f2db74afb45]
/home/mud/rlmud/driver/bin/driver(+0x20475)[0x7f2db9d72475]
******** FATAL ERROR: SIGSEGV: Segmentation fault
FluffOS driver attempting to exit gracefully.
crash() in master called successfully.  Aborting.

With GDB (With "info line"):
Code: [Select]
/home/mud/rlmud/driver/bin/driver(+0xd5415)[0x7f2db9e27415]
Line 203 of "main.cc" starts at address 0xd5415 <attempt_shutdown(int)+149> and ends at 0xd5428 <attempt_shutdown(int)+168>.
Code: [Select]
/lib/x86_64-linux-gnu/libc.so.6(+0x35180)[0x7f2db74c3180]
Line 1230 of "vm/internal/base/object.cc" starts at address 0x3517e <fgv_recurse(program_t*, int*, char*, unsigned short*, int) [clone .lto_priv.364]+1038>
   and ends at 0x35181 <fgv_recurse(program_t*, int*, char*, unsigned short*, int) [clone .lto_priv.364]+1041>.
Code: [Select]
/home/mud/rlmud/driver/bin/driver(+0xd4a40)[0x7f2db9e26a40]
Line 220 of "net/telnet.cc" starts at address 0xd4a40 <telnet_event_handler(telnet_t*, telnet_event_t*, void*) [clone .lto_priv.272]+1184>
   and ends at 0xd4a5d <telnet_event_handler(telnet_t*, telnet_event_t*, void*) [clone .lto_priv.272]+1213>.
Code: [Select]
/home/mud/rlmud/driver/bin/driver(+0xd1f38)[0x7f2db9e23f38]
Line 813 of "thirdparty/libtelnet/libtelnet.c" starts at address 0xd1f38 <_subnegotiate(telnet_t*)+72> and ends at 0xd1f62 <_subnegotiate(telnet_t*)+114>.
Code: [Select]
/home/mud/rlmud/driver/bin/driver(+0xd365a)[0x7f2db9e2565a]
Line 1052 of "thirdparty/libtelnet/libtelnet.c" starts at address 0xd3655 <_process(telnet_t*, char const*, unsigned long) [clone .lto_priv.264]+1301>
   and ends at 0xd3674 <_process(telnet_t*, char const*, unsigned long) [clone .lto_priv.264]+1332>.
Code: [Select]
/home/mud/rlmud/driver/bin/driver(telnet_recv+0x158)[0x7f2db9e14eb8]
Line 1171 of "thirdparty/libtelnet/libtelnet.c" starts at address 0xc2eb8 <telnet_recv+344> and ends at 0xc2ed0 <pop_n_elems(int) [clone .constprop.143]>.
Code: [Select]
/home/mud/rlmud/driver/bin/driver(_Z13get_user_dataP13interactive_t+0x571)[0x7f2db9e3bcd1]
Line 769 of "comm.cc" starts at address 0xe9cd1 <get_user_data(interactive_t*)+1393> and ends at 0xe9ce0 <get_user_data(interactive_t*)+1408>.
Code: [Select]
/home/mud/rlmud/driver/bin/driver(_Z7backendP10event_base+0x26e)[0x7f2db9e18f0e]
Line 259 of "backend.cc" starts at address 0xc6f0e <backend(event_base*)+622> and ends at 0xc6f18 <backend(event_base*)+632>.
Code: [Select]
/home/mud/rlmud/driver/bin/driver(main+0x302)[0x7f2db9d71d42]
Line 168 of "main.cc" starts at address 0x1fd42 <main(int, char**)+770> and ends at 0x1fd58 <main(int, char**)+792>.

Offline FallenTree

  • BFF
  • ***
  • Posts: 476
    • View Profile
Re: WIP: Fluffos 3.0 Alpha 9.0
« Reply #31 on: November 13, 2015, 11:32:01 PM »
are you saying the crash is also happening in 3.0alpha8.1 ?

It looks like a libtelnet bug, which I have yet to figure out how it happens.

Offline DarKWateR

  • BFF
  • ***
  • Posts: 104
    • View Profile
Re: WIP: Fluffos 3.0 Alpha 9.0
« Reply #32 on: November 14, 2015, 01:25:31 AM »
yes, the last backtrace is from fluffos 8.1.
After last crash from 9.0, i came back to fluffos 8.1 and finally crash too.

Offline quixadhal

  • BFF
  • ***
  • Posts: 618
    • View Profile
    • A Waste of Time
Re: WIP: Fluffos 3.0 Alpha 9.0
« Reply #33 on: November 14, 2015, 07:26:55 AM »
I'm not an expert on character set encodings, but my first instinctive thought was... are you using UTF-8 or a similar extended character set, and are there any valid byte sequences that use 255 in them?  TELNET was designed for ASCII transmission, and character 255 was denoted as the special IAC escape code to say "Hey, the next byte is either part of a command sequence, or another 255 escaped"

If libtelnet is scanning data at the byte level for IAC sequences, and one happens to be part of a multi-byte character set in whatever encoding you're using, it's very possible that libtelnet might try to analyze it.  If it happens to be a valid command (but with nonsense data), it might do something unexpected.

That's just a wild guess though.

Offline DarKWateR

  • BFF
  • ***
  • Posts: 104
    • View Profile
Re: WIP: Fluffos 3.0 Alpha 9.0
« Reply #34 on: November 14, 2015, 11:49:09 AM »
You're right, my mud has active UTF-8.

Offline FallenTree

  • BFF
  • ***
  • Posts: 476
    • View Profile
Re: WIP: Fluffos 3.0 Alpha 9.0
« Reply #35 on: November 15, 2015, 12:40:30 AM »
I doubt that's the issue, telnet protocol will escape IAC automatically.

This confirm my suspicion, this is a libtelnet bug, which I have yet to have a grasp at.

Offline DarKWateR

  • BFF
  • ***
  • Posts: 104
    • View Profile
Re: WIP: Fluffos 3.0 Alpha 9.0
« Reply #36 on: November 16, 2015, 02:14:44 AM »
Hello, this night driver crashed again.
I have the coredump, if you say me a mail i'll send you with the driver binary.

Offline FallenTree

  • BFF
  • ***
  • Posts: 476
    • View Profile
Re: WIP: Fluffos 3.0 Alpha 9.0
« Reply #37 on: November 18, 2015, 05:47:46 AM »
sunyucong@gmail.com

There must be some configuration trigger specific problem, because we havn't have much of a issue on many popular chinese muds.

Please send me binary, core dump, and full log.

Offline DarKWateR

  • BFF
  • ***
  • Posts: 104
    • View Profile
Re: WIP: Fluffos 3.0 Alpha 9.0
« Reply #38 on: November 18, 2015, 01:33:33 PM »
With full log, do you mean runtime_driver?
That log was deleted :(
the only useful info in it was stack trace, but that info is visible with gdb, isn't it?

The compress tar.gz core filesize are 56338715 MB.
Which are the max file_size for attachments in your mail? or do you have a ftp or similar?

I saw the core with gdb and in the moment of crash, i saw a telnet petition type TELNET_TELOPT_LINEMODE (34).
Perhaps some combination for some user and telnet client, and crash driver when enters? but i don't know, telnet protocol is new for me.

Thanks!

PD: Sorry my english

Offline FallenTree

  • BFF
  • ***
  • Posts: 476
    • View Profile
Re: WIP: Fluffos 3.0 Alpha 9.0
« Reply #39 on: November 19, 2015, 09:50:22 PM »
How come your core size is so big......How much memory do you have?

hmm, your crash looks different than what I have been hearing. I will take a look early next week.

Offline DarKWateR

  • BFF
  • ***
  • Posts: 104
    • View Profile
Re: WIP: Fluffos 3.0 Alpha 9.0
« Reply #40 on: November 20, 2015, 11:20:57 AM »
In my server, driver is add to systemctl.
That core is generated with systemd, because with ulimit didn't generate core in that situation.

surely i did something wrong, it was my first time with coredumps :D

Offline FallenTree

  • BFF
  • ***
  • Posts: 476
    • View Profile
Re: WIP: Fluffos 3.0 Alpha 9.0
« Reply #41 on: November 23, 2015, 08:31:29 AM »
@DarkWateR

First: I think this is caused by broken client, not a server side problem (although we certainly can do better than crashing..)

This is a very interesting problem: I have a workaround checked in , but I want to you do do something with the core, to verify something for me.

Here's how: use gdb driver core , use bt to find stack that belongs to "comm.cc", and use "f X" to select that frame:

> print ip->local_port
> x/4ub ((in_addr*)&((struct sockaddr_in*)&(ip->addr))->sin_addr)->s_addr
> x/36xb buf

First command will print out the port it connected to,  second command prints out IP address,   the rest will show me the whole packets.

Here is what I found: The port is 8000,  remote ip address is in "208.100.*",  and raw data look like this

(gdb) x/36xb buf
0x7fff4271eff0: 0xff    0xfc    0x22    0xff    0xfa    0x22    0xff    0xf0
0x7fff4271eff8: 0xff    0xff    0xfc    0x03    0xff    0xfc    0x18    0xff
0x7fff4271f000: 0xfc    0x1f    0xff    0xfc    0x27    0xff    0xfe    0x56
0x7fff4271f008: 0xff    0xfc    0x5b    0xff    0xfe    0x46    0xff    0xfe
0x7fff4271f010: 0x5d    0xff    0xfe    0xc9

.

Also, don't forget to try the latest fix :-p



Offline DarKWateR

  • BFF
  • ***
  • Posts: 104
    • View Profile
Re: WIP: Fluffos 3.0 Alpha 9.0
« Reply #42 on: November 23, 2015, 01:09:13 PM »
Hello, output is:
(gdb) print ip->local_port
$1 = 5001

(Incorrect coredump or binary?)
(gdb) x/4ub ((struct in_addr*)&((struct sockaddr_in*)&(ip->addr))->sin_addr)->s_addr
0xe71a64d0:     Cannot access memory at address 0xe71a64d0

(gdb)  x/36xb buf
0x7fffc2aa05a0: 0xff    0xfc    0x22    0xff    0xfa    0x22    0xff    0xf0
0x7fffc2aa05a8: 0xff    0xff    0xfc    0x03    0xff    0xfc    0x18    0xff
0x7fffc2aa05b0: 0xfc    0x1f    0xff    0xfc    0x27    0xff    0xfe    0x56
0x7fffc2aa05b8: 0xff    0xfc    0x5b    0xff    0xfe    0x46    0xff    0xfe
0x7fffc2aa05c0: 0x5d    0xff    0xfe    0xc9

Last output is the same.

i'll update with your path and cross my fingers!

Offline FallenTree

  • BFF
  • ***
  • Posts: 476
    • View Profile
Re: WIP: Fluffos 3.0 Alpha 9.0
« Reply #43 on: November 24, 2015, 05:19:36 AM »
o yeah, the wonder of broken attacker!

Offline DarKWateR

  • BFF
  • ***
  • Posts: 104
    • View Profile
Re: WIP: Fluffos 3.0 Alpha 9.0
« Reply #44 on: November 24, 2015, 12:58:05 PM »
Yeah! :D