post json data lag

Please post any questions about developing your plugin here. Please use the search function before posting!
Banz
Junior Member
Posts: 10
Joined: Tue Sep 06, 2016 3:40 pm

post json data lag

Postby Banz » Mon Sep 12, 2016 7:09 pm

Okay so the problem is whenever I try to send json data server lags for a second (var spike) but the information gets succesfully sent so thats good.



Here is an example code

Syntax: Select all

from commands.say import SayCommand
from messages import SayText2
import json, urllib.request

def post_request(data):
url = "https://example.com"
req = urllib.request.Request(url)
req.add_header("Content-Type", "application/json; charset=utf-8")
jsondata = json.dumps(data)
jsondataasbytes = jsondata.encode("utf-8")
req.add_header("Content-Length", len(jsondataasbytes))
print(jsondataasbytes)
response = urllib.request.urlopen(req, jsondataasbytes)
@SayCommand("!postd")
def on_say_post(command, index, team):
whatever = {"string": command.command_string[7:]}
SayText2(whatever["string"]).send(index)
post_request(whatever)

once again thanks in advance
Last edited by Ayuto on Mon Sep 12, 2016 8:00 pm, edited 1 time in total.
Reason: code -> python
User avatar
Ayuto
Project Leader
Posts: 2193
Joined: Sat Jul 07, 2012 8:17 am
Location: Germany

Re: post json data lag

Postby Ayuto » Mon Sep 12, 2016 8:10 pm

That's normal. If you don't need the response immediately, you can fix that issue by creating a thread for the request:
https://github.com/Source-Python-Dev-Te ... ick.py#L50
Banz
Junior Member
Posts: 10
Joined: Tue Sep 06, 2016 3:40 pm

Re: post json data lag

Postby Banz » Mon Sep 12, 2016 8:24 pm

Syntax: Select all

from commands.say import SayCommand
from messages import SayText2
from listeners.tick import GameThread
import json, urllib.request

def post_request(data):
url = "https://example.com"
req = urllib.request.Request(url)
req.add_header("Content-Type", "application/json; charset=utf-8")
jsondata = json.dumps(data)
jsondataasbytes = jsondata.encode("utf-8")
req.add_header("Content-Length", len(jsondataasbytes))
print(jsondataasbytes)
response = urllib.request.urlopen(req, jsondataasbytes)
@SayCommand("!postd")
def on_say_post(command, index, team):
whatever = {"string": command.command_string[7:]}
SayText2(whatever["string"]).send(index)
t = GameThread(target=post_request(whatever))
t.start()


Is this how it should be done? Still seems to have the spike so I guess not.
Last edited by Banz on Mon Sep 12, 2016 8:39 pm, edited 1 time in total.
User avatar
satoon101
Project Leader
Posts: 2697
Joined: Sat Jul 07, 2012 1:59 am

Re: post json data lag

Postby satoon101 » Mon Sep 12, 2016 8:30 pm

Just a small tip, but if you just want the argument string, instead of this:

Syntax: Select all

whatever = {"string": command.command_string[7:]}

use this:

Syntax: Select all

whatever = {"string": command.arg_string}
Image
Banz
Junior Member
Posts: 10
Joined: Tue Sep 06, 2016 3:40 pm

Re: post json data lag

Postby Banz » Mon Sep 12, 2016 8:33 pm

satoon101 wrote:Just a small tip, but if you just want the argument string, instead of this:

Syntax: Select all

whatever = {"string": command.command_string[7:]}

use this:

Syntax: Select all

whatever = {"string": command.arg_string}

Thanks was wondering if there was better way to do that :D
User avatar
satoon101
Project Leader
Posts: 2697
Joined: Sat Jul 07, 2012 1:59 am

Re: post json data lag

Postby satoon101 » Mon Sep 12, 2016 8:45 pm

I am no expert with threading, but I think it's something more like:

Syntax: Select all

from commands.say import SayCommand
from messages import SayText2
from listeners.tick import GameThread
import json, urllib.request

def post_request(data):
url = "https://example.com"
req = urllib.request.Request(url)
req.add_header("Content-Type", "application/json; charset=utf-8")
jsondata = json.dumps(data)
jsondataasbytes = jsondata.encode("utf-8")
req.add_header("Content-Length", len(jsondataasbytes))
print(jsondataasbytes)
response = urllib.request.urlopen(req, jsondataasbytes)

@SayCommand("!postd")
def on_say_post(command, index, team):
whatever = {"string": command.arg_string}
SayText2(whatever["string"]).send(index)
t = GameThread(target=post_request, args=(whatever, ))
t.start()
Image
Banz
Junior Member
Posts: 10
Joined: Tue Sep 06, 2016 3:40 pm

Re: post json data lag

Postby Banz » Mon Sep 12, 2016 8:52 pm

satoon101 wrote:I am no expert with threading, but I think it's something more like:

Syntax: Select all

from commands.say import SayCommand
from messages import SayText2
from listeners.tick import GameThread
import json, urllib.request

def post_request(data):
url = "https://example.com"
req = urllib.request.Request(url)
req.add_header("Content-Type", "application/json; charset=utf-8")
jsondata = json.dumps(data)
jsondataasbytes = jsondata.encode("utf-8")
req.add_header("Content-Length", len(jsondataasbytes))
print(jsondataasbytes)
response = urllib.request.urlopen(req, jsondataasbytes)

@SayCommand("!postd")
def on_say_post(command, index, team):
whatever = {"string": command.arg_string}
SayText2(whatever["string"]).send(index)
t = GameThread(target=post_request, args=(whatever, ))
t.start()

That seemed to do the trick thank you for both of you :)
User avatar
Doldol
Senior Member
Posts: 200
Joined: Sat Jul 07, 2012 7:09 pm
Location: Belgium

Re: post json data lag

Postby Doldol » Mon Sep 12, 2016 9:40 pm

Be careful, starting a Python thread can introduce lag too. (if someone were to spam !postd, 100% guaranteed the server will start to lag.). My solution would be to create one thread when your plugin starts that you can pass functions to call to, like this:

Syntax: Select all

import time
import threading
import queue
from queue import Queue
from contextlib import suppress

from core import AutoUnload
from hooks.exceptions import except_hooks
from listeners import on_tick_listener_manager

from commands.say import SayCommand
from messages import SayText2
import json, urllib.request

class StoppableSPThread(threading.Thread, AutoUnload):
def __init__(self, accuracy=1, *args, **kwargs):
super().__init__(*args, **kwargs)
self.accuracy = accuracy
on_tick_listener_manager.register_listener(self._tick) # Automatically wake up every tick
self._stop = threading.Event()

def run(self):
while not self.stopped:
try:
self.do()
except Exception:
except_hooks.print_exception()
time.sleep(self.accuracy)

def do(self):
raise NotImplementedError("Override me!")

def _tick(self):
pass

def stop(self):
self._stop.set()
on_tick_listener_manager.unregister_listener(self._tick)

@property
def stopped(self):
return self._stop.is_set()

_unload_instance = stop

class ThreadedCaller(StoppableSPThread):
def __init__(self, *args, **kwargs):
self.queue = Queue()
super().__init__(*args, **kwargs)

def do(self):
with suppress(queue.Empty):
to_call = self.queue.get(block=True, timeout=10)
to_call["func"](*to_call["args"], **to_call["kwargs"])

threaded_caller = ThreadedCaller()
threaded_caller.start()

def post_request(data):
url = "https://example.com"
req = urllib.request.Request(url)
req.add_header("Content-Type", "application/json; charset=utf-8")
jsondata = json.dumps(data)
jsondataasbytes = jsondata.encode("utf-8")
req.add_header("Content-Length", len(jsondataasbytes))
print(jsondataasbytes)
response = urllib.request.urlopen(req, jsondataasbytes)

@SayCommand("!postd")
def on_say_post(command, index, team):
whatever = {"string": command.arg_string}
SayText2(whatever["string"]).send(index)
threaded_caller.queue.put_nowait({"func:":post_request, "args":(whatever,), "kwargs":{}})

(Not all of it is tested so it may have some typos.)

Btw, imo it may be a good idea to implement a worker like this in the SP library.
Last edited by Doldol on Mon Jan 23, 2017 10:19 pm, edited 1 time in total.
User avatar
L'In20Cible
Project Leader
Posts: 1533
Joined: Sat Jul 14, 2012 9:29 pm
Location: Québec

Re: post json data lag

Postby L'In20Cible » Mon Sep 12, 2016 10:49 pm

No need to handle the tick listener yourself, simply inherit from GameThread.
decompile
Senior Member
Posts: 416
Joined: Sat Oct 10, 2015 10:37 am
Location: Germany
Contact:

Re: post json data lag

Postby decompile » Tue Jan 17, 2017 7:43 pm

Doldol wrote:Be careful, starting a Python thread can introduce lag too. (if someone were to spam !postd, 100% guaranteed the server will start to lag.). My solution would be to create one thread when your plugin starts that you can pass functions to call to, like this:

Syntax: Select all

import time
import threading
import queue
from queue import Queue
from contextlib import suppress

from core import AutoUnload
from hooks.exceptions import except_hooks
from listeners import on_tick_listener_manager

from commands.say import SayCommand
from messages import SayText2
import json, urllib.request

class StoppableSPThread(threading.Thread, AutoUnload):
def __init__(self, accuracy=1, *args, **kwargs):
super().__init__(*args, **kwargs)
self.accuracy = accuracy
on_tick_listener_manager.register_listener(self._tick) # Automatically wake up every tick
self._stop = threading.Event()

def run(self):
while not self.stopped:
try:
self.do()
except Exception:
except_hooks.print_exception()
time.sleep(self.accuracy)

def do(self):
raise NotImplementedError("Override me!")

def _tick(self):
pass

def stop(self):
self._stop.set()
on_tick_listener_manager.unregister_listener(self._tick)

@property
def stopped(self):
return self._stop.is_set()

_unload_instance = stop

class ThreadedCaller(StoppableSPThread):
def __init__(self, *args, **kwargs):
self.queue = Queue()
super().__init__(*args, **kwargs)

def do(self):
with suppress(queue.Empty):
to_call = self.queue.get(block=True, timeout=10)
to_call["func"](*to_call["args"], **to_call["kwargs"])

threaded_caller = ThreadedCaller()
threaded_caller.start()

def post_request(data):
url = "https://example.com"
req = urllib.request.Request(url)
req.add_header("Content-Type", "application/json; charset=utf-8")
jsondata = json.dumps(data)
jsondataasbytes = jsondata.encode("utf-8")
req.add_header("Content-Length", len(jsondataasbytes))
print(jsondataasbytes)
response = urllib.request.urlopen(req, jsondataasbytes)

@SayCommand("!postd")
def on_say_post(command, index, team):
whatever = {"string": command.arg_string}
SayText2(whatever["string"]).send(index)
threaded_caller.put_nowait({"func:":post_request, "args":(whatever,), "kwargs":{}})

(Not all of it is tested so it may have some typos.)

Btw, imo it may be a good idea to implement a worker like this in the SP library.


Sorry for bumping older threads,
but Im kinda working with that issue right now.

When I read the code, Im pretty sure that you will get an error since the class "threaded_caller" has no "put_nowait".

Imo it would be good to have an updated and most recent version out of a super GameThread which can be used for http POST or whatever.
Last edited by decompile on Tue Jan 17, 2017 9:26 pm, edited 1 time in total.
User avatar
L'In20Cible
Project Leader
Posts: 1533
Joined: Sat Jul 14, 2012 9:29 pm
Location: Québec

Re: post json data lag

Postby L'In20Cible » Tue Jan 17, 2017 8:51 pm

decompile wrote:Imo it would be good to have an updated and most recent version out of a super GameThread which can be used for http POST or whatever.

Would be good, yes. How about you propose a PR? :)
User avatar
Doldol
Senior Member
Posts: 200
Joined: Sat Jul 07, 2012 7:09 pm
Location: Belgium

Re: post json data lag

Postby Doldol » Mon Jan 23, 2017 10:17 pm

decompile wrote:
Doldol wrote:Be careful, starting a Python thread can introduce lag too. (if someone were to spam !postd, 100% guaranteed the server will start to lag.). My solution would be to create one thread when your plugin starts that you can pass functions to call to, like this:

Syntax: Select all

import time
import threading
import queue
from queue import Queue
from contextlib import suppress

from core import AutoUnload
from hooks.exceptions import except_hooks
from listeners import on_tick_listener_manager

from commands.say import SayCommand
from messages import SayText2
import json, urllib.request

class StoppableSPThread(threading.Thread, AutoUnload):
def __init__(self, accuracy=1, *args, **kwargs):
super().__init__(*args, **kwargs)
self.accuracy = accuracy
on_tick_listener_manager.register_listener(self._tick) # Automatically wake up every tick
self._stop = threading.Event()

def run(self):
while not self.stopped:
try:
self.do()
except Exception:
except_hooks.print_exception()
time.sleep(self.accuracy)

def do(self):
raise NotImplementedError("Override me!")

def _tick(self):
pass

def stop(self):
self._stop.set()
on_tick_listener_manager.unregister_listener(self._tick)

@property
def stopped(self):
return self._stop.is_set()

_unload_instance = stop

class ThreadedCaller(StoppableSPThread):
def __init__(self, *args, **kwargs):
self.queue = Queue()
super().__init__(*args, **kwargs)

def do(self):
with suppress(queue.Empty):
to_call = self.queue.get(block=True, timeout=10)
to_call["func"](*to_call["args"], **to_call["kwargs"])

threaded_caller = ThreadedCaller()
threaded_caller.start()

def post_request(data):
url = "https://example.com"
req = urllib.request.Request(url)
req.add_header("Content-Type", "application/json; charset=utf-8")
jsondata = json.dumps(data)
jsondataasbytes = jsondata.encode("utf-8")
req.add_header("Content-Length", len(jsondataasbytes))
print(jsondataasbytes)
response = urllib.request.urlopen(req, jsondataasbytes)

@SayCommand("!postd")
def on_say_post(command, index, team):
whatever = {"string": command.arg_string}
SayText2(whatever["string"]).send(index)
threaded_caller.put_nowait({"func:":post_request, "args":(whatever,), "kwargs":{}})

(Not all of it is tested so it may have some typos.)

Btw, imo it may be a good idea to implement a worker like this in the SP library.


Sorry for bumping older threads,
but Im kinda working with that issue right now.

When I read the code, Im pretty sure that you will get an error since the class "threaded_caller" has no "put_nowait".

Imo it would be good to have an updated and most recent version out of a super GameThread which can be used for http POST or whatever.


I'm really late too ^^

You're right, typo: threaded_caller.put_nowait should be threaded_caller.queue.put_nowait

I'll correct it if I can still edit the post :3
User avatar
Doldol
Senior Member
Posts: 200
Joined: Sat Jul 07, 2012 7:09 pm
Location: Belgium

Re: post json data lag

Postby Doldol » Mon Jan 23, 2017 10:21 pm

L'In20Cible wrote:No need to handle the tick listener yourself, simply inherit from GameThread.


I haven't found __del__ to be very reliable, especially while working with threads. http://stackoverflow.com/a/1481512

I saw bootstrap_inner, but I prefer the finer grained control writing the logic myself gives me while working with threads.

Return to “Plugin Development Support”

Who is online

Users browsing this forum: No registered users and 38 guests