Kallithea issues archive

Issue #229: Bad File Descriptor

Reported by: codingtony
State: resolved
Created on: 2016-07-07 06:32
Updated on: 2018-06-15 19:51


Happens from time to time and it shuts down kallithea

Using kallithea 0.3.2 on Ubuntu 16.04 LTS

2016-07-07 02:27:01.500 INFO  [kallithea.RequestWrapper] IP: Request to /XXXX/YYYYYYY/ time: 0.213s
Traceback (most recent call last):
  File "/usr/local/bin/paster", line 9, in <module>
    load_entry_point('PasteScript', 'console_scripts', 'paster')()
  File "/Kallithea-0.3.2/.eggs/PasteScript-2.0.2-py2.7.egg/paste/script/command.py", line 102, in run
    invoke(command, command_name, options, args[1:])
  File "/Kallithea-0.3.2/.eggs/PasteScript-2.0.2-py2.7.egg/paste/script/command.py", line 141, in invoke
    exit_code = runner.run(args)
  File "/Kallithea-0.3.2/.eggs/PasteScript-2.0.2-py2.7.egg/paste/script/command.py", line 236, in run
    result = self.command()
  File "/Kallithea-0.3.2/.eggs/PasteScript-2.0.2-py2.7.egg/paste/script/serve.py", line 319, in command
  File "/Kallithea-0.3.2/.eggs/PasteScript-2.0.2-py2.7.egg/paste/script/serve.py", line 303, in serve
  File "/Kallithea-0.3.2/.eggs/PasteDeploy-1.5.2-py2.7.egg/paste/deploy/loadwsgi.py", line 189, in server_wrapper
  File "/Kallithea-0.3.2/.eggs/PasteDeploy-1.5.2-py2.7.egg/paste/deploy/util.py", line 55, in fix_call
    val = callable(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/__init__.py", line 21, in serve_paste
    serve(app, **kw)
  File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/__init__.py", line 18, in serve
  File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/server.py", line 154, in run
  File "/usr/lib/python2.7/asyncore.py", line 216, in loop
    poll_fun(timeout, map)
  File "/usr/lib/python2.7/asyncore.py", line 145, in poll
    r, w, e = select.select(r, w, e, timeout)
select.error: (9, 'Bad file descriptor')



Comment by Thomas De Schampheleire, on 2016-07-07 09:01

Could you try with more recent versions of waitress? For example 0.8.9, 0.8.10, 0.9.0. (http://docs.pylonsproject.org/projects/waitress/en/latest/#change-history) Assuming you're using pip, you could do: pip install --upgrade waitress==0.8.9 (and similar for the other versions)

I wonder if it could be a bug in waitress that is meanwhile fixed.

If it's the case, we need to update our setup.py dependencies.

Comment by codingtony, on 2016-07-08 04:41

The issue occurs perhaps once a week. I'll update to 0.9.0 and see.

Comment by almkuzmin, on 2016-08-19 10:20

Dear Thomas,

Upgrading to waitress 0.9.0 didn't help.

2016-08-19 10:16:03.816 INFO  [kallithea.lib.middleware.simplegit] push action on Git repo "rancid/net-cvs/xxx" by "rancid" from 10.x.x.x
2016-08-19 10:16:06.512 INFO  [kallithea.RequestWrapper] IP: 10.x.x.x Request to /rancid/net-cvs/xxx/git-receive-pack time: 2.989s
2016-08-19 10:16:06.685 ERROR [waitress] Exception when servicing <waitress.channel.HTTPChannel connected at 0x7f9fe5ab1790>
Traceback (most recent call last):
  File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/task.py", line 78, in handler_thread
  File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/channel.py", line 369, in service
  File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/parser.py", line 249, in close
  File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/buffers.py", line 298, in close
  File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/buffers.py", line 108, in close
IOError: [Errno 9] Bad file descriptor

Comment by almkuzmin, on 2016-08-19 10:25

neither did help upgrading to 1.0a2

Comment by Mads Kiilerich, on 2016-08-19 12:30

There is no Kallithea code in the stack trace. It could seem like you are hitting a waitress bug.

Comment by almkuzmin, on 2016-08-19 12:53

Does that mean there will be no fix/workaround?

Comment by almkuzmin, on 2016-08-19 13:06

As far as I can see this happens on huge updates. And it's easy reproducible. What I'm afraid most of is that repo could get broken by this unsuccessful update.

Comment by Mads Kiilerich, on 2016-08-19 13:07

I hope there will. I just doubt it is something that can be fixed in Kallithea - not because we won't but because that isn't where the problem is. But it is possible that some Kallithea user can reproduce the problem and track down what is going on and get it fixed.

@andrew_shadura - you are using git and paster serve ... with waitress? Have you seen this problem?

I guess the simplest workaround is to just not use waitress - see http://kallithea.readthedocs.io/en/latest/overview.html#web-server for the big picture.

Comment by codingtony, on 2016-08-19 13:07

And went it happens, kallithea shutdowns...

Comment by Mads Kiilerich, on 2016-08-19 13:09

Clarification: it is not really "Kallithea shuts down". It is the web server you use for serving Kallithea that crashes.

Comment by Andrej Shadura, on 2016-08-19 13:45

@kiilerix, I’m not exactly using Kallithea with Git, I’m mostly working on improving Git support but I’m not using it in production unfortunately, so my observations won’t be enough to see that if it’s not always reproducible.

Comment by Andrej Shadura, on 2016-08-19 13:47

Meanwhile, I think it would be good to ditch our own custom Git SmartHTTP protocol implementation and just use one provided by Dulwich.

Comment by Mads Kiilerich, on 2016-08-19 13:56

Fine with me. It would be weird if it solved this problem ... but who knows it it would realign the stars ...

Comment by domruf, on 2016-08-19 14:24

+1 for using (more) dulwich

Comment by domruf, on 2016-08-19 14:25

@almkuzmin You say "...it's easy reproducible". Can you tell us how?

Comment by almkuzmin, on 2016-08-22 06:13

@domruf I started creating 300 files 50 kb long each. Then pushed them with no problem.

  for i in `seq 1 300`; do dd if=/dev/urandom of=file.$i bs=50000 count=1; done
  git add .
  git commit -am test
  git push

Next step was increasing file size up to 500Kb, and it made web server crash.

Comment by almkuzmin, on 2016-08-22 06:20

@kiilerix I'm not familiar with running wsgi applications. Can you please provide start script for use with nginx/apache?

Comment by codingtony, on 2016-08-22 12:43

@almkuzmin I've encountered the bug not always when there are big pushes. However when I restart Kallithea, I re-push and it goes through without error. We have git and hg repos. We have more hg repo than git ones and the problem can occur on both (hg or git) pushes.

Comment by domruf, on 2016-08-22 19:04

@almkuzmin does this happen every time on your machine? I just tried it several times (300 files @ 500kb each) and it only happend once. I even tried it with 1MB for each file.

Comment by almkuzmin, on 2016-08-23 05:40

@domruf yeah. I repeated dd and tried to push: "IOError: [Errno 9] Bad file descriptor"

Comment by almkuzmin, on 2016-08-23 05:55

Well. Just an observation. While receiving commit, Kallithea (waitress?) uses /tmp, which is mounted as tmpfs in my case. At some point some file(s) apparently get removed while they are still opened by the server process. I can see no disk usage, but there's a lot of occupied space (shown by df) that is released after service restart.

Comment by almkuzmin, on 2016-08-23 07:38

I luckily managed to find a document on setting up WSGI. Now I'm in front of a new obstacle: the server returns 411 on my push requests (error: RPC failed; HTTP 411 curl 22 The requested URL returned error: 411 Length Required). As far as I understand I have to open new issue.

Comment by almkuzmin, on 2016-08-23 10:56

I'm sorry I'm flooding this chat (: I solved my issue using the method from this post

git config --system http.postBuffer 524288000

Comment by Mads Kiilerich, on 2016-08-23 20:44

Please consider contributing a documentation patch ( http://kallithea.readthedocs.io/en/latest/contributing.html ) if you can find a place where local git client configuration could fit in the Kallithea documentation. Or perhaps first discuss what you could have in mind. I guess the git documentation would be a better place.

Anyway, it still sounds like a waitress bug if a client interrupting a POST (before Kallithea gets invoked) can make it crash.

Comment by almkuzmin, on 2016-08-24 06:11

@kiilerix I think it perfectly fits into "Troubleshooting" section. Q: Git push fails with HTTP 411 code (Length Required)? A: Set variable http.postBuffer to a reasonably high value (e.g. 500M): git config --global http.postBuffer 524288000

Comment by Mads Kiilerich, on 2016-08-24 12:21

Right. I totally agree.

But that will only cover a part of the problem - not the waitress crashing?

Comment by almkuzmin, on 2016-08-24 14:17

I still have no solution for that issue. Except switching to apache/nginx (:

Comment by Thomas De Schampheleire, on 2018-06-15 19:51

See also issue #219 and mailing list thread https://lists.sfconservancy.org/pipermail/kallithea-general/2018q2/002667.html

Problem seems to be related to waitress but not only waitress.

Following workarounds exist:

  • either stay on the stable branch, but use a different web server than waitress, e.g. uwsgi or apache.
  • or switch to the default branch, in which case waitress seems to work fine, as do other web servers.