Issue #359: Cannot clone repository with Kallithea in Python 3 with Gunicorn
|Created on:||2020-02-13 15:00|
|Updated on:||2020-02-18 02:52|
After moving to new Kallithea version with Python 3, unable to clone repository, it gives worker TIMEOUT error and operations are rolled back, the same clone works with Python 2 and older version. In logs there is no error.
http authorization required for https://hg.xxxxxxxx.com/test/test-repository realm: Kallithea user: kallithea password: files [====> ] 358/6509 7m18sdestination directory: test-repository requesting all changes adding changesets adding manifests adding file changes transaction abort! rollback completed abort: HTTP request error (incomplete response) (this may be an intermittent network failure; if the error persists, consider contacting the network or server operator)
The Gunicorn service log is:
Feb 13 22:37:23 kallithea-5-2 gunicorn: [2020-02-13 22:37:23 +0800]  [INFO] Starting gunicorn 20.0.4 Feb 13 22:37:23 kallithea-5-2 gunicorn: [2020-02-13 22:37:23 +0800]  [INFO] Listening at: http://0.0.0.0:5000 (7755) Feb 13 22:37:23 kallithea-5-2 gunicorn: [2020-02-13 22:37:23 +0800]  [INFO] Using worker: sync Feb 13 22:37:23 kallithea-5-2 gunicorn: [2020-02-13 22:37:23 +0800]  [INFO] Booting worker with pid: 7758 Feb 13 22:42:05 kallithea-5-2 gunicorn: [2020-02-13 22:42:05 +0800]  [CRITICAL] WORKER TIMEOUT (pid:7758) Feb 13 22:42:05 kallithea-5-2 gunicorn: [2020-02-13 22:42:05 +0800]  [INFO] Worker exiting (pid: 7758) Feb 13 22:42:05 kallithea-5-2 gunicorn: [2020-02-13 22:42:05 +0800]  [INFO] Booting worker with pid: 8876
Comment by vyom, on 2020-02-13 15:01
Comment by vyom, on 2020-02-13 15:38
Resolved the issue by changing Gunicorn settings. Gunicorn version 20 has many breaking changes and one of them is it does not pick up configuration from paste ini.
So after changing few more settings in Gunicorn it works now.
[Unit] Description=kallithea-scm After=network.target [Service] PIDFile=/run/gunicorn/pid User=www-data Group=www-data RuntimeDirectory=gunicorn WorkingDirectory=/home/kallithea/projects/kallithea Environment=VIRTUAL_ENV=/home/kallithea/projects/kallithea/venv3 Environment=PATH=$VIRTUAL_ENV/bin:$PATH ExecStart=/home/kallithea/projects/kallithea/venv3/bin/gunicorn --pid /run/gunicorn/pid --paste new.ini -b 0.0.0.0:5000 -t 600 --threads 3 ExecReload=/bin/kill -s HUP $MAINPID ExecStop=/bin/kill -s TERM $MAINPID PrivateTmp=true [Install] WantedBy=multi-user.target
Comment by Mads Kiilerich, on 2020-02-13 15:48
Thanks for the thorough testing.
What gunicorn version were you using before, and which setting made the crucial difference?
Is it something we should change in the documentation or
kallithea-cli config-create new.ini http_server=gunicorn ?
Comment by vyom, on 2020-02-13 15:55
I was using version 19 which worked with Python 2. In version 20 they had breaking changes and one of them was paste deploy. So all the settings related to gunicorn in new.ini like host, port, threads did not work and need specify them in my systemd file in command line.
[server:main] ## WAITRESS ## use = egg:gunicorn#main ## number of worker threads threads = 2 ## MAX BODY SIZE 100GB max_request_body_size = 107374182400 ## use poll instead of select, fixes fd limits, may not work on old ## windows systems. #asyncore_use_poll = True ## COMMON ## host = 0.0.0.0 port = 5000
Based on gunicorn documentation I think the issue is:
Breaking Change Removed gaiohttp worker Drop support for Python 2.x Drop support for EOL Python 3.2 and 3.3 Drop support for Paste Deploy server blocks
So Paste Deploy server blocks are dropped so may be all the settings need to define in separate configuration file or on command line.
Comment by Thomas De Schampheleire, on 2020-02-15 20:20
Out of curiosity, for which reasons have you selected gunicorn? Which other alternatives did you consider and why did you not select them?
Comment by vyom, on 2020-02-16 00:28
In case of Kallithea I used gunicorn to see if there is a performance increase and remove apache2, also support of gunicorn to load paste deployment ini was straight forward. Didn’t choose uwsgi, cherrypy I need to write my own config files. Before gunicorn I was using kallithea with mod_wsgi.
Following is the old configuration with name based virtual hosting:
<VirtualHost *:443> ServerAdmin email@example.com ServerName kallithea.test.com DocumentRoot /home/kallithea/projects/kallithea ErrorLog /var/log/apache2/error.kallithea.log CustomLog /var/log/apache2/access.kallithea.log combined SSLEngine On SSLCertificateFile /etc/apache2/ssl/kallithea.test.com.crt SSLCertificateKeyFile /etc/apache2/ssl/kallithea.test.com.key WSGIDaemonProcess kallithea processes=1 threads=4 WSGIScriptAlias / /home/kallithea/projects/kallithea/dispatch.wsgi WSGIPassAuthorization On <Directory /home/kallithea/projects/kallithea/dispatch.wsgi> Require all granted </Directory> <Location /> Require all granted </Location> # Error handlers ErrorDocument 500 /errordocs/500.html </VirtualHost>
This is dispatch.wsgi (now fileConfig has been deprecated in new paste).
activate_this = '/home/kallithea/projects/kallithea/venv/bin/activate_this.py' execfile(activate_this,dict(__file__=activate_this)) import os os.environ['HOME'] = '/home/kallithea/projects/kallithea' ini = '/home/kallithea/projects/kallithea/kallithea.ini' from paste.script.util.logging_config import fileConfig fileConfig(ini) from paste.deploy import loadapp application = loadapp('config:' + ini)
I did a test on following for my other project with WSGI apps using Greenlets (via Eventlet and Gevent):
- cherrypy (now its cheroot)
- Cherrypy was the most reliable one but under heavy load was bit slow. It will timeout requests but will not return a wrong response.
- Gunicorn and uwsgi were neck to neck but the error response under heavy load was not very good for both. The only thing I did not like about both of them was under heavy stress they will return wrong responses.
- mod_wsgi I use when I am on apache, it’s easier to setup and work with. It’s performance wise was slower than gunicorn and uwsgi (I did that a while ago so might have change by now).
In our team for developers we use cherrypy (now cheroot) as it’s production quality pure python wsgi server works well across linux, mac and windows. In production use either uwsgi or gunicorn.
Comment by vyom, on 2020-02-18 02:52
Could clone after making changes in gunicorn startup with systemd services.