GameThread slow

Please post any questions about developing your plugin here. Please use the search function before posting!
decompile
Senior Member
Posts: 416
Joined: Sat Oct 10, 2015 10:37 am
Location: Germany
Contact:

Re: GameThread slow

Postby decompile » Tue Jul 25, 2023 11:38 pm

Could you please leave me a downloadable version of your InfiniteThread changes? I would like to try it out.


I've now updated the run function to your improved one:

Syntax: Select all

def run(self):
while self.connected:
while not self.queue.empty():
with ThreadPoker():
callback, args, kwargs = self.queue.get_nowait()
callback(*args, **kwargs)

time.sleep(0.1) # Reduce CPU Load


Thank you, the server is now playable, but everytime the ThreadPoker is being triggered, e.g. by adding a function call, the server lags (loses tickrate) for a very short duration, as if it would been executed in the main thread. (I have tested it while being on the server, and while moving, you instantly notice the lag)

You can replicate it yourself by using:

test.py

Syntax: Select all

import time
from listeners.tick import GameThread


def test_function():
time.sleep(1.0)

print("test function")


thread = GameThread(target=test_function)
thread.start()
User avatar
L'In20Cible
Project Leader
Posts: 1534
Joined: Sat Jul 14, 2012 9:29 pm
Location: Québec

Re: GameThread slow

Postby L'In20Cible » Wed Jul 26, 2023 9:16 am

decompile wrote:You can replicate it yourself by using:

I was able to reproduce it with (which I believe reproduce your tickrate_enabler plugin):

Syntax: Select all

from memory import *
from memory.hooks import *
from engines.server import *

@PreHook(
get_object_pointer(server_game_dll).make_virtual_function(
10, # GetTickInterval
Convention.CDECL,
(),
DataType.FLOAT
)
)
def _(stack):
return 1.0 / 128

This causes the server to skip ticks because poking the system 128 times per seconds may takes longer than 1.0 / 128. I will push a different approach to prevent thread starvation, but results will be quite slower. I will also re-organize some stuff while I'm at it.
User avatar
L'In20Cible
Project Leader
Posts: 1534
Joined: Sat Jul 14, 2012 9:29 pm
Location: Québec

Re: GameThread slow

Postby L'In20Cible » Wed Jul 26, 2023 10:47 am

d6580e1 should address the issue. It also add some utilities (have a look at ../threads.py for examples). You will also need to edit your core_settings.ini to enable the yielding.

Test build (Windows): 834f711

For your use case here, you could simply use queued without having to worry about managing a thread yourself:

Syntax: Select all

import time
from threads import queued

def done_sleeping(start, result):
print(f'Done waiting and sleeping for {time.time() - start} seconds!')

queued(time.sleep, 3).with_callback(done_sleeping, time.time())
decompile
Senior Member
Posts: 416
Joined: Sat Oct 10, 2015 10:37 am
Location: Germany
Contact:

Re: GameThread slow

Postby decompile » Wed Jul 26, 2023 11:38 pm

Hey, thank you for the changes. They look promising and I'm currently trying it out, thanks alot.

Just noticed this here when unloading my test.py from yesterday.

Code: Select all

[SP] Successfully loaded plugin 'test'.
test_function
sp plugin unload test
[SP] Unloading plugin 'test'...
[Source.Python]
[SP] Caught an Exception:
Traceback (most recent call last):
  File "..\addons\source-python\packages\source-python\plugins\manager.py", line 437, in _unload_auto_unload_instances
    instance._unload_instance()
  File "..\addons\source-python\packages\source-python\threads.py", line 88, in _unload_instance
    del self._target, self._args, self._kwargs

AttributeError: _target


[SP] Successfully unloaded plugin 'test'.


Imported from threads import GameThread


Edit: Your changes and additions look amazing. The server and gameplay is now smooth without any lags or tickrate loss. Thank you so much. If I could suggest something, that it would may be worth to call the callback in the main thread, or if possible in the original thread. (I had alot of issues where I sent menus inside a thread callback, which caused my server to crash alot)
User avatar
L'In20Cible
Project Leader
Posts: 1534
Joined: Sat Jul 14, 2012 9:29 pm
Location: Québec

Re: GameThread slow

Postby L'In20Cible » Thu Jul 27, 2023 4:18 am

decompile wrote:Just noticed this here when unloading my test.py from yesterday.

Code: Select all

AttributeError: _target
Thanks, will get that fixed.

decompile wrote:The server and gameplay is now smooth without any lags or tickrate loss.
What about execution time? Did you notice any difference in timing?

On my side, I just timed the following:

Syntax: Select all

list(GAME_PATH.walkfiles()) # I/O
[sum([sqrt(i) for i in range(1000)]) for i in range(10000)] # CPU
And got the following result:

Code: Select all

Direct: 2.585948872566223
Threaded: 469.47447347640986
Threaded (w/ Poker): 3.258913373947143
Threaded (w/ Yielder): 28.63925561904907

decompile wrote:If I could suggest something, that it would may be worth to call the callback in the main thread, or if possible in the original thread. (I had alot of issues where I sent menus inside a thread callback, which caused my server to crash alot)
While I think resource locking should be the responsibility of the callback, I also think we should probably make bitbuf messages thread-safe since they work in paired calls.

EDIT: Done into 76a1006. Updated folder: 11wjA_hu9J87k2VBcH-zKAjWJMFGt4rOc
decompile
Senior Member
Posts: 416
Joined: Sat Oct 10, 2015 10:37 am
Location: Germany
Contact:

Re: GameThread slow

Postby decompile » Thu Jul 27, 2023 6:52 pm

Thanks for the update! I'm currently trying it out and I feel no loss in performance. Its very smooth and the results in threads are amazingly fast with enable_thread_yielding enabled.

Just for example:

Code: Select all

2640
Time: 0.26081013679504395


Edit: I've tested some I/O things aswell. Opening, reading and analyzing the same data file in another thread took me around 0.5s, now its 0.04s (wow)
User avatar
L'In20Cible
Project Leader
Posts: 1534
Joined: Sat Jul 14, 2012 9:29 pm
Location: Québec

Re: GameThread slow

Postby L'In20Cible » Tue Aug 01, 2023 8:50 pm

Try this: 1wAeyyUgdGSAENV80BeWSIN0Sy3w4W5Jb

It should not only be faster but overall smoother. Basically, instead of yielding an extra 1ms every frame and exceed the intended interval, it now yields the entire time the main thread goes inactive after it's done running the frame. You can see how many ms with developer 2.

Syntax: Select all

from threads import InfiniteThread, yielder
from time import time
from paths import GAME_PATH
from math import sqrt

class Test(InfiniteThread):
def __call__(self):
def run():
t = time()
list(GAME_PATH.walkfiles()) # I/O
[sum([sqrt(i) for i in range(1000)]) for i in range(10000)] # CPU
print(time() - t)

run()
with yielder:
run()

raise SystemExit

Test().start()

Code: Select all

...
[Source.Python] Yielded 14ms on tick 47683!
[Source.Python] Yielded 9ms on tick 47684!
[Source.Python] Yielded 12ms on tick 47685!
[Source.Python] Yielded 14ms on tick 47686!
[Source.Python] Yielded 8ms on tick 47687!
464.5280984163284
2.3814141035079955
[Source.Python] Yielded 7ms on tick 47688
decompile
Senior Member
Posts: 416
Joined: Sat Oct 10, 2015 10:37 am
Location: Germany
Contact:

Re: GameThread slow

Postby decompile » Wed Aug 02, 2023 11:32 pm

Hey, I tried it out and it feels as good as before. Is the change noticeable?
User avatar
L'In20Cible
Project Leader
Posts: 1534
Joined: Sat Jul 14, 2012 9:29 pm
Location: Québec

Re: GameThread slow

Postby L'In20Cible » Thu Aug 03, 2023 12:01 am

decompile wrote:Hey, I tried it out and it feels as good as before. Is the change noticeable?

Let's take for example a server with default 67 tickrate. Each tick would be at ~15ms interval. If running the frame takes 5ms, it then goes to sleep for 10ms. The previous workaround was going to sleep an extra 1ms on top no matter how long running the frame took. For the same time, it was becoming 5ms to run the frame, 1ms to yield, and 9ms to sleep. This is fine, until the frame takes 14ms or longer (lot of players, lot of plugins, etc.). In such case, it was still going to sleep an extra 1ms and exceeding the interval (dumplongticks was very noisy about it). Now, we only yield when the server goes to sleep (if it sleeps for 10ms, we yield for 10ms, if it does not sleep, we don't, etc.). Not only this means we never accidentally exceed the interval, but we also make use of every bit of resource the server is not using yielding many time longer than before. Most of the time, this means we get close to direct execution time. For example, run the following with developer 2:

Syntax: Select all

from paths import GAME_PATH
from threads import queued
from math import sqrt
from time import time

def foo():
t = time()
list(GAME_PATH.walkfiles()) # I/O
[sum([sqrt(i) for i in range(1000)]) for i in range(100000)] # CPU
print(time() - t)

foo()
queued(foo)

Then run it again with sp_thread_yielding 0. For instance, if a routine takes 1 second when called directly, it would then takes (1 / 0.001) / <tickrate> (~15s at 67 tickrate) to run when limited at 1ms per tick. Now, if the server goes to sleep 10ms per tick in average, it will execute 10 times faster, etc. So, you are not going to see any difference for calls that take < 1ms because we force a context switch and get it done in one frame, but for long calls it will be exponentially faster (depending of server activity at the time, of course).
decompile
Senior Member
Posts: 416
Joined: Sat Oct 10, 2015 10:37 am
Location: Germany
Contact:

Re: GameThread slow

Postby decompile » Fri Aug 04, 2023 7:34 pm

Thank you for this detailed explanation. Thats an explanation if you ask me! :)
User avatar
L'In20Cible
Project Leader
Posts: 1534
Joined: Sat Jul 14, 2012 9:29 pm
Location: Québec

Re: GameThread slow

Postby L'In20Cible » Mon Aug 07, 2023 4:18 am

Added support for Linux and CS:GO: 1wAeyyUgdGSAENV80BeWSIN0Sy3w4W5Jb
decompile
Senior Member
Posts: 416
Joined: Sat Oct 10, 2015 10:37 am
Location: Germany
Contact:

Re: GameThread slow

Postby decompile » Fri Aug 18, 2023 10:45 pm

Hey,

thank you for all the work on this so far. The changes are literally day and night in a very positive way, and makes developing way easier. :) Is there any update about this?
User avatar
L'In20Cible
Project Leader
Posts: 1534
Joined: Sat Jul 14, 2012 9:29 pm
Location: Québec

Re: GameThread slow

Postby L'In20Cible » Sat Aug 19, 2023 10:10 am

decompile wrote:Is there any update about this?
I don't plan on iterating on this further at this time but progress (if any) can be followed into #491.

Return to “Plugin Development Support”

Who is online

Users browsing this forum: No registered users and 22 guests