Threaded Menu crash

Please post any questions about developing your plugin here. Please use the search function before posting!
decompile
Senior Member
Posts: 416
Joined: Sat Oct 10, 2015 10:37 am
Location: Germany
Contact:

Threaded Menu crash

Postby decompile » Sun Feb 07, 2021 5:24 am

Hey,

I'm currently having a random server crash on sending a threaded menu which I cant replicate. It just happens randomly, mostly it works and then suddenly it crashes.

Syntax: Select all

loading_state = itertools.cycle(['   ', '.  ', '.. ', '...'])
def build_loading_menu(menu, index):
menu.clear()
menu.append('Loading' + next(loading_state))
loading_menu = SimpleMenu(build_callback=build_loading_menu)


def _send_threaded_menu(index, callback, args, kwargs, set_page):
print('DEBUG 1')

# Returns a menu class which is ready to be sent to a player
menu = callback(*args, **kwargs)

print('DEBUG 2')

# New menu has been loaded - We can close the old loading menu
loading_menu.close(index)

print('DEBUG 3')

# Send new menu
menu.send(index)

print('DEBUG 4')

# If set_page is defined - Set player page on new menu
if set_page:
print('DEBUG 5')

menu.set_player_page(index, set_page)

print('DEBUG 6')


def send_threaded_menu(index, callback, args=(), kwargs=None, set_page=None):
kwargs = kwargs if kwargs is not None else dict()

# Send loading menu
loading_menu.send(index)

# Add function to WorkerThread
ThreadedClass.function_call(_send_threaded_menu, (index, callback, args, kwargs, set_page))


Code: Select all

DEBUG 1
DEBUG 2
Could not find steamerrorreporter binary. Any minidumps will be uploaded in-processUserMessageBegin:  New message started before matching call to EndMessage.
 
L 02/07/2021 - 02:02:13: Engine error: UserMessageBegin:  New message started before matching call to EndMessage.
 
Segmentation fault (core dumped)
Add "-debug" to the ./srcds_run command line to generate a debug.log to help with solving this problem


The ThreadedClass has the base of a WorkerThread from here:
viewtopic.php?p=13740#p13740

I also uncommented the loading_menu parts, where I got the same message, but for sending the actual menu.

Syntax: Select all

loading_state = itertools.cycle(['   ', '.  ', '.. ', '...'])
def build_loading_menu(menu, index):
menu.clear()
menu.append('Loading' + next(loading_state))
loading_menu = SimpleMenu(build_callback=build_loading_menu)


def _send_threaded_menu(index, callback, args, kwargs, set_page):
print('DEBUG 1')

# Returns a menu class which is ready to be sent to a player
menu = callback(*args, **kwargs)

print('DEBUG 2')

# New menu has been loaded - We can close the old loading menu
# loading_menu.close(index)

print('DEBUG 3')

# Send new menu
menu.send(index)

print('DEBUG 4')

# If set_page is defined - Set player page on new menu
if set_page:
print('DEBUG 5')

menu.set_player_page(index, set_page)

print('DEBUG 6')


def send_threaded_menu(index, callback, args=(), kwargs=None, set_page=None):
kwargs = kwargs if kwargs is not None else dict()

# Send loading menu
# loading_menu.send(index)

# Add function to WorkerThread
ThreadedClass.function_call(_send_threaded_menu, (index, callback, args, kwargs, set_page))


Code: Select all

DEBUG 1
DEBUG 2
DEBUG 3
Could not find steamerrorreporter binary. Any minidumps will be uploaded in-processUserMessageBegin:  New message started before matching call to EndMessage.
 
L 02/07/2021 - 04:10:34: Engine error: UserMessageBegin:  New message started before matching call to EndMessage.
 
Segmentation fault (core dumped)
User avatar
Ayuto
Project Leader
Posts: 2193
Joined: Sat Jul 07, 2012 8:17 am
Location: Germany

Re: Threaded Menu crash

Postby Ayuto » Sun Feb 07, 2021 6:25 am

Please post the full test code to reproduce the crash.
decompile
Senior Member
Posts: 416
Joined: Sat Oct 10, 2015 10:37 am
Location: Germany
Contact:

Re: Threaded Menu crash

Postby decompile » Sun Feb 07, 2021 7:08 am

As stated in the second line, I have no way to reproduce this code. I cant explain why it happens.
User avatar
Ayuto
Project Leader
Posts: 2193
Joined: Sat Jul 07, 2012 8:17 am
Location: Germany

Re: Threaded Menu crash

Postby Ayuto » Sun Feb 07, 2021 8:22 am

Well, I have an assumption, but I would need to see the full test code (especially your Thread class). I guess it doesn't execute the menu.send() line atomically. So, a user message is started, but then the thread is being switched and another one is started.

https://en.m.wikipedia.org/wiki/Context_switch
User avatar
Ayuto
Project Leader
Posts: 2193
Joined: Sat Jul 07, 2012 8:17 am
Location: Germany

Re: Threaded Menu crash

Postby Ayuto » Sun Feb 07, 2021 3:00 pm

Additionally, to your PM, this might help understanding the problem:

Syntax: Select all

# Only an empty CS:S server is required for this test/explanation plugin.
# No bots; no players. Just load this test plugin with test1 uncommented.
# Then comment out test1, uncomment test2 and reload. Continue like that with test3.

from engines.server import engine_server
from memory import get_virtual_function
from memory import make_object
from bitbuffers import BitBufferWrite
from filters.recipients import RecipientFilter

# Import this snippet: https: //pastebin.com/raw/wbHxTt6d
from test6.worker import WorkerThread

# virtual bf_write *UserMessageBegin( IRecipientFilter *filter, int msg_type ) = 0;
UserMessageBegin = get_virtual_function(engine_server, 'UserMessageBegin')

# virtual void MessageEnd( void ) = 0;
MessageEnd = get_virtual_function(engine_server, 'MessageEnd')

recipients = RecipientFilter()

def test1(arg):
print(arg, 'START')
# Simply send a user message
buffer = make_object(
BitBufferWrite,
UserMessageBegin(engine_server, recipients, 0))

buffer.write_byte(0) # Required for user message 0

# This will cause the thread do be busy and might cause a context switch.
# Alternativly, time.sleep()
y = 0
for x in range(100000):
y += x % 2

MessageEnd(engine_server)

print(arg, 'END')

def test2():
buffer = UserMessageBegin(engine_server, recipients, 0)

# Crashes here, because a second user message is started before the first
# one is finished
buffer = UserMessageBegin(engine_server, recipients, 0)

def test3():
worker1 = WorkerThread()
worker2 = WorkerThread()

for x in range(10):
worker1.add_job(test1, ['worker1'])
worker2.add_job(test1, ['worker2'])

# There is a very high chance that you crash. Even with only one worker
# thread, because the main thread also continously starts user messages.
# Context switching causes the function calls to test1() to stop somewhere
# during the execution. It can be before line 1, in line 1, after line 1,
# but before line 2. Basically, anywhere in that function, because it's not
# atomic/thread safe.
worker1.start()
worker2.start()


# TEST 1 - Demonstrate a working user message. You server won't crash and the
# plugin will load fine.
#test1('main thread')

# TEST 2 - Demonstrate two calls to UserMessageBegin without finishing the
# first one. This will crash your server.
#test2()

# TEST 3 - Demonstrate a high chance of crashing. You might need to reload the
# plugin multiple times if you are lucky.
#test3()

TEST1

Code: Select all

main thread START
main thread END

TEST3

Code: Select all

worker1 START
worker2 START

--> Crash due to context switch from worker1 to worker2
User avatar
L'In20Cible
Project Leader
Posts: 1533
Joined: Sat Jul 14, 2012 9:29 pm
Location: Québec

Re: Threaded Menu crash

Postby L'In20Cible » Sun Feb 07, 2021 7:59 pm

Using a reentrant lock to send the menu would likely fix the context switch, however, the main unlocked thread (aka the game) could still send a message at any time during game-play likely causing the exact same crash if a worker is currently executing regardless of its locking state. Threading bitbuf messages is risky, and I don't think there is an easy solution -- if any at all because we have no control over the original (or external) calls.
User avatar
Ayuto
Project Leader
Posts: 2193
Joined: Sat Jul 07, 2012 8:17 am
Location: Germany

Re: Threaded Menu crash

Postby Ayuto » Mon Feb 08, 2021 12:41 pm

Yeah, I thought about a mutex as well, but discarded the idea, because the best solution is to simply send the user message in the main thread. A generic solution might be to pre-hook UserMessageBegin and set the lock and post-hook MessageEnd and unlock.
decompile
Senior Member
Posts: 416
Joined: Sat Oct 10, 2015 10:37 am
Location: Germany
Contact:

Re: Threaded Menu crash

Postby decompile » Sat Feb 13, 2021 12:28 pm

Hi,

sorry for the late response, I haven't found the time to reply yet.

My crashing issue has been resolved, after Ayuto checking out the functions I was using. The reason for the crash was, that I was building the menu in a seperate thread, which allowed the main thread & seperate thread to collide with each other while building bitbuf messages. As the engine error said, starting a new bitbuff message before the other one was closed. To resolve this issue, I just had to pass the data to the main thread, and build the menu from there.

Thank you guys for your help.

Return to “Plugin Development Support”

Who is online

Users browsing this forum: No registered users and 20 guests