Issue #229: Bad File Descriptor
|Created on:||2016-07-07 06:32|
|Updated on:||2018-06-15 19:51|
Happens from time to time and it shuts down kallithea
Using kallithea 0.3.2 on Ubuntu 16.04 LTS
2016-07-07 02:27:01.500 INFO [kallithea.RequestWrapper] IP: 192.168.20.1 Request to /XXXX/YYYYYYY/ time: 0.213s Traceback (most recent call last): File "/usr/local/bin/paster", line 9, in <module> load_entry_point('PasteScript', 'console_scripts', 'paster')() File "/Kallithea-0.3.2/.eggs/PasteScript-2.0.2-py2.7.egg/paste/script/command.py", line 102, in run invoke(command, command_name, options, args[1:]) File "/Kallithea-0.3.2/.eggs/PasteScript-2.0.2-py2.7.egg/paste/script/command.py", line 141, in invoke exit_code = runner.run(args) File "/Kallithea-0.3.2/.eggs/PasteScript-2.0.2-py2.7.egg/paste/script/command.py", line 236, in run result = self.command() File "/Kallithea-0.3.2/.eggs/PasteScript-2.0.2-py2.7.egg/paste/script/serve.py", line 319, in command serve() File "/Kallithea-0.3.2/.eggs/PasteScript-2.0.2-py2.7.egg/paste/script/serve.py", line 303, in serve server(app) File "/Kallithea-0.3.2/.eggs/PasteDeploy-1.5.2-py2.7.egg/paste/deploy/loadwsgi.py", line 189, in server_wrapper **context.local_conf) File "/Kallithea-0.3.2/.eggs/PasteDeploy-1.5.2-py2.7.egg/paste/deploy/util.py", line 55, in fix_call val = callable(*args, **kw) File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/__init__.py", line 21, in serve_paste serve(app, **kw) File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/__init__.py", line 18, in serve server.run() File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/server.py", line 154, in run use_poll=self.adj.asyncore_use_poll, File "/usr/lib/python2.7/asyncore.py", line 216, in loop poll_fun(timeout, map) File "/usr/lib/python2.7/asyncore.py", line 145, in poll r, w, e = select.select(r, w, e, timeout) select.error: (9, 'Bad file descriptor')
Comment by Thomas De Schampheleire, on 2016-07-07 09:01
Could you try with more recent versions of waitress? For example 0.8.9, 0.8.10, 0.9.0. (http://docs.pylonsproject.org/projects/waitress/en/latest/#change-history) Assuming you're using pip, you could do: pip install --upgrade waitress==0.8.9 (and similar for the other versions)
I wonder if it could be a bug in waitress that is meanwhile fixed.
If it's the case, we need to update our setup.py dependencies.
Comment by codingtony, on 2016-07-08 04:41
The issue occurs perhaps once a week. I'll update to 0.9.0 and see.
Comment by almkuzmin, on 2016-08-19 10:20
Upgrading to waitress 0.9.0 didn't help.
2016-08-19 10:16:03.816 INFO [kallithea.lib.middleware.simplegit] push action on Git repo "rancid/net-cvs/xxx" by "rancid" from 10.x.x.x 2016-08-19 10:16:06.512 INFO [kallithea.RequestWrapper] IP: 10.x.x.x Request to /rancid/net-cvs/xxx/git-receive-pack time: 2.989s 2016-08-19 10:16:06.685 ERROR [waitress] Exception when servicing <waitress.channel.HTTPChannel connected 127.0.0.1:40082 at 0x7f9fe5ab1790> Traceback (most recent call last): File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/task.py", line 78, in handler_thread task.service() File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/channel.py", line 369, in service request.close() File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/parser.py", line 249, in close body_rcv.getbuf().close() File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/buffers.py", line 298, in close buf.close() File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/buffers.py", line 108, in close self.file.close() IOError: [Errno 9] Bad file descriptor
Comment by almkuzmin, on 2016-08-19 10:25
neither did help upgrading to 1.0a2
Comment by Mads Kiilerich, on 2016-08-19 12:30
There is no Kallithea code in the stack trace. It could seem like you are hitting a waitress bug.
Comment by almkuzmin, on 2016-08-19 12:53
Does that mean there will be no fix/workaround?
Comment by almkuzmin, on 2016-08-19 13:06
As far as I can see this happens on huge updates. And it's easy reproducible. What I'm afraid most of is that repo could get broken by this unsuccessful update.
Comment by Mads Kiilerich, on 2016-08-19 13:07
I hope there will. I just doubt it is something that can be fixed in Kallithea - not because we won't but because that isn't where the problem is. But it is possible that some Kallithea user can reproduce the problem and track down what is going on and get it fixed.
@andrew_shadura - you are using git and paster serve ... with waitress? Have you seen this problem?
I guess the simplest workaround is to just not use waitress - see http://kallithea.readthedocs.io/en/latest/overview.html#web-server for the big picture.
Comment by codingtony, on 2016-08-19 13:07
And went it happens, kallithea shutdowns...
Comment by Mads Kiilerich, on 2016-08-19 13:09
Clarification: it is not really "Kallithea shuts down". It is the web server you use for serving Kallithea that crashes.
Comment by Andrej Shadura, on 2016-08-19 13:45
@kiilerix, I’m not exactly using Kallithea with Git, I’m mostly working on improving Git support but I’m not using it in production unfortunately, so my observations won’t be enough to see that if it’s not always reproducible.
Comment by Andrej Shadura, on 2016-08-19 13:47
Meanwhile, I think it would be good to ditch our own custom Git SmartHTTP protocol implementation and just use one provided by Dulwich.
Comment by Mads Kiilerich, on 2016-08-19 13:56
Fine with me. It would be weird if it solved this problem ... but who knows it it would realign the stars ...
Comment by domruf, on 2016-08-19 14:24
+1 for using (more) dulwich
Comment by domruf, on 2016-08-19 14:25
@almkuzmin You say "...it's easy reproducible". Can you tell us how?
Comment by almkuzmin, on 2016-08-22 06:13
@domruf I started creating 300 files 50 kb long each. Then pushed them with no problem.
for i in `seq 1 300`; do dd if=/dev/urandom of=file.$i bs=50000 count=1; done git add . git commit -am test git push
Next step was increasing file size up to 500Kb, and it made web server crash.
Comment by almkuzmin, on 2016-08-22 06:20
@kiilerix I'm not familiar with running wsgi applications. Can you please provide start script for use with nginx/apache?
Comment by codingtony, on 2016-08-22 12:43
@almkuzmin I've encountered the bug not always when there are big pushes. However when I restart Kallithea, I re-push and it goes through without error. We have git and hg repos. We have more hg repo than git ones and the problem can occur on both (hg or git) pushes.
Comment by domruf, on 2016-08-22 19:04
@almkuzmin does this happen every time on your machine? I just tried it several times (300 files @ 500kb each) and it only happend once. I even tried it with 1MB for each file.
Comment by almkuzmin, on 2016-08-23 05:40
@domruf yeah. I repeated
dd and tried to push: "IOError: [Errno 9] Bad file descriptor"
Comment by almkuzmin, on 2016-08-23 05:55
Well. Just an observation.
While receiving commit, Kallithea (waitress?) uses /tmp, which is mounted as tmpfs in my case. At some point some file(s) apparently get removed while they are still opened by the server process. I can see no disk usage, but there's a lot of occupied space (shown by
df) that is released after service restart.
Comment by almkuzmin, on 2016-08-23 07:38
I luckily managed to find a document on setting up WSGI. Now I'm in front of a new obstacle: the server returns 411 on my push requests (error: RPC failed; HTTP 411 curl 22 The requested URL returned error: 411 Length Required). As far as I understand I have to open new issue.
Comment by almkuzmin, on 2016-08-23 10:56
Comment by Mads Kiilerich, on 2016-08-23 20:44
Please consider contributing a documentation patch ( http://kallithea.readthedocs.io/en/latest/contributing.html ) if you can find a place where local git client configuration could fit in the Kallithea documentation. Or perhaps first discuss what you could have in mind. I guess the git documentation would be a better place.
Anyway, it still sounds like a waitress bug if a client interrupting a POST (before Kallithea gets invoked) can make it crash.
Comment by almkuzmin, on 2016-08-24 06:11
@kiilerix I think it perfectly fits into "Troubleshooting" section.
Q: Git push fails with HTTP 411 code (Length Required)?
A: Set variable
http.postBuffer to a reasonably high value (e.g. 500M):
git config --global http.postBuffer 524288000
Comment by Mads Kiilerich, on 2016-08-24 12:21
Right. I totally agree.
But that will only cover a part of the problem - not the waitress crashing?
Comment by almkuzmin, on 2016-08-24 14:17
I still have no solution for that issue. Except switching to apache/nginx (:
Comment by Thomas De Schampheleire, on 2018-06-15 19:51
See also issue
#219 and mailing list thread https://lists.sfconservancy.org/pipermail/kallithea-general/2018q2/002667.html
Problem seems to be related to waitress but not only waitress.
Following workarounds exist:
- either stay on the stable branch, but use a different web server than waitress, e.g. uwsgi or apache.
- or switch to the default branch, in which case waitress seems to work fine, as do other web servers.