Trouble with psycopg2-binary

Please post any questions about developing your plugin here. Please use the search function before posting!
User avatar
Hymns For Disco
Member
Posts: 32
Joined: Wed Nov 22, 2017 7:18 am
Contact:

Trouble with psycopg2-binary

Postby Hymns For Disco » Mon Jan 20, 2020 7:50 am

I'm trying to use psycopg-binary in my source-python plugin, as its the database driver used by Django for my models. The trouble is that the version of psycopg2 installed in the django project is a 64 bit version, and the embedded python installation of source-python is 32 bit, and so source-python is not able to import this module.

Also unfortunately, source-python's embedded installation does not have the pip module, and so I don't know how to install the version of psycopg2 that is appropriate for this environment. The default python installations on my server are all 64 bit, and so using one of those pips to install to the site-packages folder of source-python isn't guaranteed to get a usable result for binary packages such as psycopg2-binary.

Any thoughts on how to solve this and get psycopg2-binary or other binary packages installed and usable by source-python while operating on a 64 bit machine?
User avatar
L'In20Cible
Project Leader
Posts: 1533
Joined: Sat Jul 14, 2012 9:29 pm
Location: Québec

Re: Trouble with psycopg2-binary

Postby L'In20Cible » Mon Jan 20, 2020 7:59 am

You need the psycopg2-2.8.4-cp36-cp36m-win32 build.

Syntax: Select all

# ../addons/source-python/plugins/testing/testing.py

import psycopg2
print(psycopg2)


Code: Select all

[SP] Loading plugin 'testing'...
<module 'psycopg2' from '..\addons\source-python\packages\site-packages\psycopg2\__init__.py'>
[SP] Successfully loaded plugin 'testing'.
User avatar
Hymns For Disco
Member
Posts: 32
Joined: Wed Nov 22, 2017 7:18 am
Contact:

Re: Trouble with psycopg2-binary

Postby Hymns For Disco » Mon Jan 20, 2020 8:04 am

L'In20Cible wrote:You need the psycopg2-2.8.4-cp36-cp36m-win32 build.

Thank you, I notice this package name contains win32. I'm running my server on Debian 10. Is this correct, does the embedded Python of source-python allow running win32 versions of packages while on a linux os, or will I have to look for something else?
User avatar
L'In20Cible
Project Leader
Posts: 1533
Joined: Sat Jul 14, 2012 9:29 pm
Location: Québec

Re: Trouble with psycopg2-binary

Postby L'In20Cible » Mon Jan 20, 2020 8:07 am

Hymns For Disco wrote:
L'In20Cible wrote:You need the psycopg2-2.8.4-cp36-cp36m-win32 build.

Thank you, I notice this package name contains win32. I'm running my server on Debian 10. Is this correct, does the embedded Python of source-python allow running win32 versions of packages while on a linux os, or will I have to look for something else?

Try this build for Linux: psycopg2_binary-2.8.4-cp36-cp36m-manylinux1_i686
User avatar
Hymns For Disco
Member
Posts: 32
Joined: Wed Nov 22, 2017 7:18 am
Contact:

Re: Trouble with psycopg2-binary

Postby Hymns For Disco » Tue Jan 21, 2020 12:49 am

Thank you, that wheel seemed to work. I've updated my install plugins script to act as a "pip for source-python" effectively.

Code: Select all

#!/bin/bash

SOURCEPYTHON="/home/csgoserver/serverfiles/csgo/addons/source-python"

echo Installing package requirements to game server
pip3 install -q --no-deps --abi cp36m --python-version 36 --platform manylinux1_i686 --target=$SOURCEPYTHON/packages/site-packages -r /home/csgoserver/plugins/requirements.txt


echo Uninstalling all plugins
rm -r $SOURCEPYTHON/plugins/*

for f  in /home/csgoserver/plugins/*; do
        if [ -d ${f} ]; then
                plugin="$(basename $f)"
                echo Installing $plugin
                pluginfolder=$SOURCEPYTHON/plugins/$plugin
                #mkdir $pluginfolder
                cp -r $f $SOURCEPYTHON/plugins/
        fi
done


Still having some issues with psycopg2 segfaulting while attempting SSL, not sure if related to this issue or not. I'll try to update in the future if I figure out relevant details there
User avatar
Hymns For Disco
Member
Posts: 32
Joined: Wed Nov 22, 2017 7:18 am
Contact:

Re: Trouble with psycopg2-binary

Postby Hymns For Disco » Tue Jan 21, 2020 3:28 am

So here's the trouble:
The _psycopg2 C module must be built in order to be loaded by django to access my database. Installing it with my builtin pip3 (3.7 64 bit), will result in it building a _psycopg.cpython-37m-x86_64-linux-gnu.so, which sourcepython will fail to import. If I install psycopg2 from the wheel you have linked, it will create a _psycopg.cpython-36m-i386-linux-gnu.so which is now loaded by source-python's 32 bit python no problem.
Here's the catch though:
This _psycopg.cpython-36m-i386-linux-gnu.so C module is compiled against OpenSSL, this allows the package to connect to the database in SSL mode, which you will want to do if connecting to a remote machine. Whoever has compiled this wheel, it is somehow incompatible with openssl on my machine. I am running OpenSSL 1.1.1d 10 Sep 2019, which I think is the correct one for latest pyscopg2 wheels.

I think building the psycopg2 package from source would be the best bet, but I can't do this as I don't have a python installation on my server compatible with source-python's 32 bit 3.6. I'm running Debian 10 64 bit which comes with a 64 bit installation of Python 3.7.

Are there any known solutions or plans to address this? It seems a pretty big problem to me that there is no obvious way to actually install packages in a way that's guaranteed to by compatible with source-python. I seem to be unable to get python3.6 installed, let alone python3.6 in 32 bit on my 64 bit system.

Edit: For the time being, I will be running my database and game server on the same machine, so connecting with the ssl mode disabled is fine. Still, this does make me worry about the future of my project if I am unable to safely use common packages like this. So it would help to know from the project leaders, is there any known method or plan for resolving these kinds of issues?

Return to “Plugin Development Support”

Who is online

Users browsing this forum: No registered users and 37 guests