Kallithea issues archive

Issue #359: Cannot clone repository with Kallithea in Python 3 with Gunicorn

Reported by: vyom
State: resolved
Created on: 2020-02-13 15:00
Updated on: 2020-02-18 02:52


After moving to new Kallithea version with Python 3, unable to clone repository, it gives worker TIMEOUT error and operations are rolled back, the same clone works with Python 2 and older version. In logs there is no error.

http authorization required for https://hg.xxxxxxxx.com/test/test-repository
realm: Kallithea
user: kallithea
files [====>                                                                                        ]  358/6509 7m18sdestination directory: test-repository
requesting all changes
adding changesets
adding manifests
adding file changes
transaction abort!
rollback completed
abort: HTTP request error (incomplete response)
(this may be an intermittent network failure; if the error persists, consider contacting the network or server operator)

The Gunicorn service log is:

Feb 13 22:37:23 kallithea-5-2 gunicorn[7755]: [2020-02-13 22:37:23 +0800] [7755] [INFO] Starting gunicorn 20.0.4
Feb 13 22:37:23 kallithea-5-2 gunicorn[7755]: [2020-02-13 22:37:23 +0800] [7755] [INFO] Listening at: (7755)
Feb 13 22:37:23 kallithea-5-2 gunicorn[7755]: [2020-02-13 22:37:23 +0800] [7755] [INFO] Using worker: sync
Feb 13 22:37:23 kallithea-5-2 gunicorn[7755]: [2020-02-13 22:37:23 +0800] [7758] [INFO] Booting worker with pid: 7758
Feb 13 22:42:05 kallithea-5-2 gunicorn[7755]: [2020-02-13 22:42:05 +0800] [7755] [CRITICAL] WORKER TIMEOUT (pid:7758)
Feb 13 22:42:05 kallithea-5-2 gunicorn[7755]: [2020-02-13 22:42:05 +0800] [7758] [INFO] Worker exiting (pid: 7758)
Feb 13 22:42:05 kallithea-5-2 gunicorn[7755]: [2020-02-13 22:42:05 +0800] [8876] [INFO] Booting worker with pid: 8876



Comment by vyom, on 2020-02-13 15:01

Comment by vyom, on 2020-02-13 15:38

Resolved the issue by changing Gunicorn settings. Gunicorn version 20 has many breaking changes and one of them is it does not pick up configuration from paste ini.

So after changing few more settings in Gunicorn it works now.


ExecStart=/home/kallithea/projects/kallithea/venv3/bin/gunicorn --pid /run/gunicorn/pid --paste new.ini -b -t
600 --threads 3
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID


Comment by Mads Kiilerich, on 2020-02-13 15:48

Thanks for the thorough testing.

What gunicorn version were you using before, and which setting made the crucial difference?

Is it something we should change in the documentation or kallithea-cli config-create new.ini http_server=gunicorn ?

Comment by vyom, on 2020-02-13 15:55

I was using version 19 which worked with Python 2. In version 20 they had breaking changes and one of them was paste deploy. So all the settings related to gunicorn in new.ini like host, port, threads did not work and need specify them in my systemd file in command line.

use = egg:gunicorn#main
## number of worker threads
threads = 2
max_request_body_size = 107374182400
## use poll instead of select, fixes fd limits, may not work on old
## windows systems.
#asyncore_use_poll = True

## COMMON ##
host =
port = 5000

Based on gunicorn documentation I think the issue is:

Breaking Change
    Removed gaiohttp worker
    Drop support for Python 2.x
    Drop support for EOL Python 3.2 and 3.3
    Drop support for Paste Deploy server blocks

So Paste Deploy server blocks are dropped so may be all the settings need to define in separate configuration file or on command line.

Comment by Thomas De Schampheleire, on 2020-02-15 20:20

Out of curiosity, for which reasons have you selected gunicorn? Which other alternatives did you consider and why did you not select them?

Comment by vyom, on 2020-02-16 00:28

In case of Kallithea I used gunicorn to see if there is a performance increase and remove apache2, also support of gunicorn to load paste deployment ini was straight forward. Didn’t choose uwsgi, cherrypy I need to write my own config files. Before gunicorn I was using kallithea with mod_wsgi.

Following is the old configuration with name based virtual hosting:

<VirtualHost *:443>
        ServerAdmin support@test.com
        ServerName kallithea.test.com
        DocumentRoot /home/kallithea/projects/kallithea
        ErrorLog /var/log/apache2/error.kallithea.log
        CustomLog /var/log/apache2/access.kallithea.log combined
    SSLEngine On
    SSLCertificateFile /etc/apache2/ssl/kallithea.test.com.crt
    SSLCertificateKeyFile /etc/apache2/ssl/kallithea.test.com.key

    WSGIDaemonProcess kallithea processes=1 threads=4
    WSGIScriptAlias / /home/kallithea/projects/kallithea/dispatch.wsgi
    WSGIPassAuthorization On
    <Directory /home/kallithea/projects/kallithea/dispatch.wsgi>
     Require all granted
       <Location />
     Require all granted

    # Error handlers
    ErrorDocument 500 /errordocs/500.html


This is dispatch.wsgi (now fileConfig has been deprecated in new paste).

activate_this = '/home/kallithea/projects/kallithea/venv/bin/activate_this.py'
import os
os.environ['HOME'] = '/home/kallithea/projects/kallithea'
ini = '/home/kallithea/projects/kallithea/kallithea.ini'
from paste.script.util.logging_config import fileConfig
from paste.deploy import loadapp
application = loadapp('config:' + ini)

I did a test on following for my other project with WSGI apps using Greenlets (via Eventlet and Gevent):

  1. uwsgi
  2. cherrypy (now its cheroot)
  3. gunicorn
  4. mod_wsgi

I use all of them in my different projects. In benchmark tests with WRK benchmark tool and Locust - A modern load testing framework.

  1. Cherrypy was the most reliable one but under heavy load was bit slow. It will timeout requests but will not return a wrong response.
  2. Gunicorn and uwsgi were neck to neck but the error response under heavy load was not very good for both. The only thing I did not like about both of them was under heavy stress they will return wrong responses.
  3. mod_wsgi I use when I am on apache, it’s easier to setup and work with. It’s performance wise was slower than gunicorn and uwsgi (I did that a while ago so might have change by now).

In our team for developers we use cherrypy (now cheroot) as it’s production quality pure python wsgi server works well across linux, mac and windows. In production use either uwsgi or gunicorn.

Comment by vyom, on 2020-02-18 02:52

Could clone after making changes in gunicorn startup with systemd services.