Port of IPToCountry and some questions to port from ES To SP

Please post any questions about developing your plugin here. Please use the search function before posting!
decompile
Senior Member
Posts: 416
Joined: Sat Oct 10, 2015 10:37 am
Location: Germany
Contact:

Port of IPToCountry and some questions to port from ES To SP

Postby decompile » Mon Dec 14, 2015 5:56 pm

Hello again!

I tried to port the ES IPToCountry to SP but it seems like it does not include StringIO, so could someone might port that for us to SP please? (Never used it so i have no clear about that)

Questions:

1. Whats the function for sending a userid a message to CHAT with colors like \x07..? (ES: esc.tell)

2. Whats the function for sending a message to all users to CHAT with colors like \x07..? (ES: esc.msg)

3. Whats the replace for es.getUseridList()?

4. Is there something for es.registerClientCommandFilter ?

5. How can I execute something through the server ? es.server.queuecmd

And last question:

Can someone might explain me why we are using indexes from players instead of the userid like on es (If I understood that correctly)


That would be the questions for porting my first "easier" files to SP.

Thank you!
User avatar
satoon101
Project Leader
Posts: 2697
Joined: Sat Jul 07, 2012 1:59 am

Postby satoon101 » Mon Dec 14, 2015 6:34 pm

For 1 and 2, use messages.SayText2. There are lots of examples on the forums already.

For player iteration, use filters.players.PlayerIter.

To register a client command filter, use commands.client.ClientCommandFilter. If you wish to only listen for a particular command or a few particular commands, you should use commands.client.ClientCommand instead.

To execute a command, use engines.server.engine_server.server_command (sorry, it isn't on the wiki, yet).

Syntax: Select all

from engines.server import engine_server

engine_server.server_command('changelevel de_dust2;')


Note that currently you must end the server_command string with either ; or \n. We are going to add that internally here soon, so that it won't be required anymore.

Also, if you need the command to be executed immediately, you can use engine_server.server_execute() to force all currently queued commands to execute.

You can still use userids instead of indexes for your own purposes and probably should in some cases. The main reason we are using indexes for a lot of things is that our Entity implementation uses them. The Player class inherits from Entity, and as such uses the index, as well.

*Edit: also, we are working very close to the engine, and pretty much nothing in the engine uses userids, only indexes (and other object types like PlayerInfo/edict_t).
Image
decompile
Senior Member
Posts: 416
Joined: Sat Oct 10, 2015 10:37 am
Location: Germany
Contact:

Postby decompile » Mon Dec 14, 2015 10:07 pm

Thank you!
decompile
Senior Member
Posts: 416
Joined: Sat Oct 10, 2015 10:37 am
Location: Germany
Contact:

Postby decompile » Wed Dec 16, 2015 10:39 pm

Short question towards IPToCountry SP port..

I have tried to port

Syntax: Select all

def _download_database(self, url):
try:
with closing(urllib.urlopen(url)) as u:
return StringIO(u.read())
except IOError:
self.__show_exception(sys.exc_info())
return None


to

Syntax: Select all

def _download_database(self, url):
try:
with closing(urllib.request.urlopen(url)) as u:
response = u.read().decode('utf-8')
return StringIO(response)
except IOError:
self.__show_exception(sys.exc_info())
return None


But im still getting

utf-8 codec can't decode byte 0x8f in position 12: invalid start byte


It tells me that it needs to be Str or None, not bytes but seems like decode doesnt work for that

I tried googling it and found actually almost always the same answer...
User avatar
L'In20Cible
Project Leader
Posts: 1533
Joined: Sat Jul 14, 2012 9:29 pm
Location: Québec

Postby L'In20Cible » Thu Dec 17, 2015 6:36 am

Please, post full code to reproduce and full traceback.
User avatar
iPlayer
Developer
Posts: 590
Joined: Sat Nov 14, 2015 8:37 am
Location: Moscow
Contact:

Postby iPlayer » Thu Dec 17, 2015 9:45 am

You sure you're getting UTF-8 from u.read()?

I can assume that you're actually getting binary data from that, and StringIO won't work in this case.
Try adapting your code to ByteIO.
User avatar
Doldol
Senior Member
Posts: 200
Joined: Sat Jul 07, 2012 7:09 pm
Location: Belgium

Postby Doldol » Thu Dec 17, 2015 8:41 pm

Why don't you just use a readily-available geoip library? e.g. https://pypi.python.org/pypi/geoip2
decompile
Senior Member
Posts: 416
Joined: Sat Oct 10, 2015 10:37 am
Location: Germany
Contact:

Postby decompile » Sat Dec 19, 2015 3:27 pm

Doldol wrote:Why don't you just use a readily-available geoip library? e.g. https://pypi.python.org/pypi/geoip2


I could do that too, but i see someone already ported it over to es and porting it to SP shouldnt be that hard at all since im already used to that for years now.
User avatar
satoon101
Project Leader
Posts: 2697
Joined: Sat Jul 07, 2012 1:59 am

Postby satoon101 » Sun Dec 20, 2015 3:52 pm

Could you please do this so we can better assist you:
L'In20Cible wrote:Please, post full code to reproduce and full traceback.
Image
User avatar
satoon101
Project Leader
Posts: 2697
Joined: Sat Jul 07, 2012 1:59 am

Postby satoon101 » Sun Dec 20, 2015 4:48 pm

Actually, had a bit of time to do some testing this morning. Haven't figured out myself, but here is full reproducible code:

Syntax: Select all

>>> from contextlib import closing
>>> from urllib.request import urlopen
>>> from io import StringIO
>>> with closing(urlopen('http://software77.net/geo-ip?DL=2')) as u:
x = StringIO(u.read())


Traceback (most recent call last):
File "<pyshell#5>", line 2, in <module>
x = StringIO(u.read())
TypeError: initial_value must be str or None, not bytes
>>> with closing(urlopen('http://software77.net/geo-ip?DL=2')) as u:
response = u.read().decode('utf-8')


Traceback (most recent call last):
File "<pyshell#7>", line 2, in <module>
response = u.read().decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x94 in position 12: invalid start byte
Image
User avatar
Doldol
Senior Member
Posts: 200
Joined: Sat Jul 07, 2012 7:09 pm
Location: Belgium

Postby Doldol » Sun Dec 20, 2015 6:45 pm

satoon101 wrote:Actually, had a bit of time to do some testing this morning. Haven't figured out myself, but here is full reproducible code:


You'll have to use BytesIO instead of StringIO, since you're downloading binary data.

Edit: But that doesn't make sense as u.read() should already return a bytes object, I tried the code out and I think no one noticed you're downloading a zip archive, which obviously can't be handled by StringIO.
User avatar
satoon101
Project Leader
Posts: 2697
Joined: Sat Jul 07, 2012 1:59 am

Postby satoon101 » Sun Dec 20, 2015 6:57 pm

The base of this code comes directly from SuperDave's IP to Country addon for ES. That is where the url I added comes from. As far as I know, it still works fine on ES using Python2.5. It then takes the StringIO object and parses the zip from that using zipfile.ZipFile.

*Edit: also, using this code in Python2.7 works fine (obviously with urlopen and StringIO imports being their 2.7 versions).
Image
User avatar
iPlayer
Developer
Posts: 590
Joined: Sat Nov 14, 2015 8:37 am
Location: Moscow
Contact:

Postby iPlayer » Sun Dec 20, 2015 7:22 pm

Because in Python 2 str can be used for both binary and text data, in Python 3 it can't.
User avatar
Doldol
Senior Member
Posts: 200
Joined: Sat Jul 07, 2012 7:09 pm
Location: Belgium

Postby Doldol » Sun Dec 20, 2015 7:27 pm

Working example:

Syntax: Select all

from pprint import pprint
from zipfile import ZipFile
from contextlib import closing
from urllib.request import urlopen
from io import StringIO, BytesIO
import csv

with closing(urlopen('http://software77.net/geo-ip?DL=2')) as u:
data = u.read()

zip_file = ZipFile(BytesIO(data))
csv_file = zip_file.read(zip_file.namelist()[0])
csv_reader = csv.reader(StringIO(csv_file.decode("utf-8", "ignore")))

pprint(list(csv_reader)[:1000])
User avatar
Doldol
Senior Member
Posts: 200
Joined: Sat Jul 07, 2012 7:09 pm
Location: Belgium

Postby Doldol » Sun Dec 20, 2015 9:10 pm

Btw, I threw this together kind of quickly, it's a wrapper around geoip2, it downloads and keeps the database updated (uses threading to do this) & contains an example.

The attachment geoip2sp.zip is no longer available
Attachments
geoip2sp.zip
(2.86 KiB) Downloaded 398 times
decompile
Senior Member
Posts: 416
Joined: Sat Oct 10, 2015 10:37 am
Location: Germany
Contact:

Postby decompile » Sun Dec 20, 2015 11:29 pm

Doldol wrote:Btw, I threw this together kind of quickly, it's a wrapper around geoip2, it downloads and keeps the database updated (uses threading to do this) & contains an example.



Thank you! Im gonna try that tomorrow After work. Thank you for everyone who replied.
decompile
Senior Member
Posts: 416
Joined: Sat Oct 10, 2015 10:37 am
Location: Germany
Contact:

Postby decompile » Mon Dec 21, 2015 8:07 pm

Okay, i tried it out now.. It seems to work except it only shows "None" for almost every player who connects.. Always getting

geoip2.errors.AddressNotFoundError: The address XXX.XXX.XXX.XX is not in the dat
abase.
User avatar
satoon101
Project Leader
Posts: 2697
Joined: Sat Jul 07, 2012 1:59 am

Postby satoon101 » Mon Dec 21, 2015 8:54 pm

Using the old IP-to-Country way, this seems to work just fine:

Syntax: Select all

>>> from zipfile import ZipFile
>>> from contextlib import closing
>>> from urllib.request import urlopen
>>> from io import BytesIO
>>> from io import StringIO
>>> with closing(urlopen('http://software77.net/geo-ip?DL=2')) as u:
data = u.read()


>>> z = ZipFile(BytesIO(data))
>>> csv = z.read(z.namelist()[0])
>>> csv = StringIO(csv.decode('utf-8', 'ignore'))
>>> database = list()
>>> for line in csv.readlines():
if line.startswith('#'):
continue
if line.count(',') < 6:
continue
values = line.strip().replace(r'"', '').split(',')
database.append({'ip_to': float(values[1]), 'country_3': values[5].strip(), 'country_long': values[6]})


>>> def get_country(ip):
new = ip.split(':', 1)[0]
ip_list = [int(x) for x in ip.split('.')]
if len(ip_list) != 4:
raise ValueError('Invalid IP address "{0}".'.format(ip))
long_ip = ip_list[3] + (ip_list[2] << 8) + (ip_list[1] << 16) + (ip_list[0] << 24)
for data in database:
if long_ip <= data['ip_to']:
return data['country_long'], data['country_3']


>>> l = get_country('123.123.123.123')
>>> print(l)
('China', 'CHN')


I really don't know how often you would ideally want to update the database, but I would guess not very often would it really be necessary.
Image
decompile
Senior Member
Posts: 416
Joined: Sat Oct 10, 2015 10:37 am
Location: Germany
Contact:

Postby decompile » Sun Dec 27, 2015 7:43 pm

Should i just use that as a new package or just update it into the "iptocountry" file?
User avatar
Painkiller
Senior Member
Posts: 725
Joined: Sun Mar 01, 2015 8:09 am
Location: Germany
Contact:

Re:

Postby Painkiller » Mon May 15, 2017 8:08 am

satoon101 wrote:Using the old IP-to-Country way, this seems to work just fine:

Syntax: Select all

>>> from zipfile import ZipFile
>>> from contextlib import closing
>>> from urllib.request import urlopen
>>> from io import BytesIO
>>> from io import StringIO
>>> with closing(urlopen('http://software77.net/geo-ip?DL=2')) as u:
data = u.read()


>>> z = ZipFile(BytesIO(data))
>>> csv = z.read(z.namelist()[0])
>>> csv = StringIO(csv.decode('utf-8', 'ignore'))
>>> database = list()
>>> for line in csv.readlines():
if line.startswith('#'):
continue
if line.count(',') < 6:
continue
values = line.strip().replace(r'"', '').split(',')
database.append({'ip_to': float(values[1]), 'country_3': values[5].strip(), 'country_long': values[6]})


>>> def get_country(ip):
new = ip.split(':', 1)[0]
ip_list = [int(x) for x in ip.split('.')]
if len(ip_list) != 4:
raise ValueError('Invalid IP address "{0}".'.format(ip))
long_ip = ip_list[3] + (ip_list[2] << 8) + (ip_list[1] << 16) + (ip_list[0] << 24)
for data in database:
if long_ip <= data['ip_to']:
return data['country_long'], data['country_3']


>>> l = get_country('123.123.123.123')
>>> print(l)
('China', 'CHN')


I really don't know how often you would ideally want to update the database, but I would guess not very often would it really be necessary.


I have tested this but it did not come to the desired display.
Example what it should show:

Image

Return to “Plugin Development Support”

Who is online

Users browsing this forum: No registered users and 24 guests