Issue #269: KALLITHEA_EXTRAS has race conditions
|Reported by:||André Klitzing|
|Created on:||2017-02-16 11:02|
|Updated on:||2018-01-20 21:13|
we use KALLITHEA_EXTRAS to fetch authenticated data for a hook. It seems that there is race condition if multiple people push at the same time.
Sometimes the hook is called with the anonymous "default" user even it is not possible to push to that repository as "anonymous". Also if two people push at the same time it is possible that the KALLITHEA_EXTRAS of both pushes are the SAME eben it is another user and another repository. This is really rarely and looks like a race condition.
Example: User A and B pushes the same time to REPO_1 and REPO_2.
Push of A has KALLITHEA_EXTRAS with user A and REPO_1. Push of B has KALLITHEA_EXTRAS with user A and REPO_1, too. But this should be B and REPO_2.
Comment by André Klitzing, on 2017-02-16 11:03
Comment by Mads Kiilerich, on 2017-02-16 12:13
How do you launch Kallithea? Are you running with a single or multi threaded server?
Comment by André Klitzing, on 2017-02-16 12:53
We use Debian Jessie with Apache2 (2.4.10-10+deb8u7) and a virtualhost with mod_wsgi for kallithea.
SSLEngine on SSLCertificateFile /etc/apache2/ssl/kallithea.cer SSLCertificateKeyFile /etc/apache2/ssl/kallithea.key WSGIPassAuthorization On WSGIScriptAlias "/" "/opt/kallithea/dispatch.wsgi/" WSGIDaemonProcess kallithea threads=4 python-path=/opt/kallithea/lib/python2.7/site-packages/ WSGIProcessGroup kallithea
Do we need to set "threads=4" to 1?
Comment by Mads Kiilerich, on 2017-02-16 12:59
Yes, i suggest that. https://kallithea-scm.org/repos/kallithea/changeset/d6942b2b421c also mostly applies to the stable branch.
But KALLITHEA_EXTRAS passed in the global environment is indeed inherently racy in a multi threaded environment. That could perhaps be done in a smarter way.
Comment by André Klitzing, on 2017-02-16 13:35
Ok, I switch to 1. I will report if it will happen again.
Comment by Mads Kiilerich, on 2017-02-16 14:09
You might want more worker processes, but they will only process one request at a time and nothing can race each other.
Comment by André Klitzing, on 2017-03-21 13:36
just for information....
There was no race condition since we reported and adjust it.
We used "processes=5" ... and saw another problem: Sometimes someone pushed new changesets. If you look with the browser to the WebUI there is a new changeset. If you refresh the page... the changeset isn't there. Another refresh... it is there. So the different processes seems to use an invalid cache here.
Comment by Mads Kiilerich, on 2017-03-21 23:42
Do you follow the documentation in setup.html:
.. note:: If running Kallithea in multiprocess mode, make sure you set ``instance_id = *`` in the configuration so each process gets it's own cache invalidation key.
Comment by André Klitzing, on 2017-03-22 07:50
Oops, sorry... I added the * to it. Hopefully it will be gone now.
Comment by Thomas De Schampheleire, on 2018-01-20 21:13
Please reopen if there is still an issue.