Kallithea issues archive

Could you try with more recent versions of waitress? For example 0.8.9, 0.8.10, 0.9.0. (http://docs.pylonsproject.org/projects/waitress/en/latest/#change-history) Assuming you're using pip, you could do: pip install --upgrade waitress==0.8.9 (and similar for the other versions)

I wonder if it could be a bug in waitress that is meanwhile fixed.

If it's the case, we need to update our setup.py dependencies.

The issue occurs perhaps once a week. I'll update to 0.9.0 and see.

Dear Thomas,

Upgrading to waitress 0.9.0 didn't help.

2016-08-19 10:16:03.816 INFO  [kallithea.lib.middleware.simplegit] push action on Git repo "rancid/net-cvs/xxx" by "rancid" from 10.x.x.x
2016-08-19 10:16:06.512 INFO  [kallithea.RequestWrapper] IP: 10.x.x.x Request to /rancid/net-cvs/xxx/git-receive-pack time: 2.989s
2016-08-19 10:16:06.685 ERROR [waitress] Exception when servicing <waitress.channel.HTTPChannel connected 127.0.0.1:40082 at 0x7f9fe5ab1790>
Traceback (most recent call last):
  File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/task.py", line 78, in handler_thread
    task.service()
  File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/channel.py", line 369, in service
    request.close()
  File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/parser.py", line 249, in close
    body_rcv.getbuf().close()
  File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/buffers.py", line 298, in close
    buf.close()
  File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/buffers.py", line 108, in close
    self.file.close()
IOError: [Errno 9] Bad file descriptor

neither did help upgrading to 1.0a2

There is no Kallithea code in the stack trace. It could seem like you are hitting a waitress bug.

Does that mean there will be no fix/workaround?

As far as I can see this happens on huge updates. And it's easy reproducible. What I'm afraid most of is that repo could get broken by this unsuccessful update.

I hope there will. I just doubt it is something that can be fixed in Kallithea - not because we won't but because that isn't where the problem is. But it is possible that some Kallithea user can reproduce the problem and track down what is going on and get it fixed.

@andrew_shadura - you are using git and paster serve ... with waitress? Have you seen this problem?

I guess the simplest workaround is to just not use waitress - see http://kallithea.readthedocs.io/en/latest/overview.html#web-server for the big picture.

And went it happens, kallithea shutdowns...

Clarification: it is not really "Kallithea shuts down". It is the web server you use for serving Kallithea that crashes.

@kiilerix, I’m not exactly using Kallithea with Git, I’m mostly working on improving Git support but I’m not using it in production unfortunately, so my observations won’t be enough to see that if it’s not always reproducible.

Meanwhile, I think it would be good to ditch our own custom Git SmartHTTP protocol implementation and just use one provided by Dulwich.

Fine with me. It would be weird if it solved this problem ... but who knows it it would realign the stars ...

+1 for using (more) dulwich

@almkuzmin You say "...it's easy reproducible". Can you tell us how?

@domruf I started creating 300 files 50 kb long each. Then pushed them with no problem.

  for i in `seq 1 300`; do dd if=/dev/urandom of=file.$i bs=50000 count=1; done
  git add .
  git commit -am test
  git push

Next step was increasing file size up to 500Kb, and it made web server crash.

@kiilerix I'm not familiar with running wsgi applications. Can you please provide start script for use with nginx/apache?

@almkuzmin I've encountered the bug not always when there are big pushes. However when I restart Kallithea, I re-push and it goes through without error. We have git and hg repos. We have more hg repo than git ones and the problem can occur on both (hg or git) pushes.

@almkuzmin does this happen every time on your machine? I just tried it several times (300 files @ 500kb each) and it only happend once. I even tried it with 1MB for each file.

@domruf yeah. I repeated dd and tried to push: "IOError: [Errno 9] Bad file descriptor"

Well. Just an observation. While receiving commit, Kallithea (waitress?) uses /tmp, which is mounted as tmpfs in my case. At some point some file(s) apparently get removed while they are still opened by the server process. I can see no disk usage, but there's a lot of occupied space (shown by df) that is released after service restart.

I luckily managed to find a document on setting up WSGI. Now I'm in front of a new obstacle: the server returns 411 on my push requests (error: RPC failed; HTTP 411 curl 22 The requested URL returned error: 411 Length Required). As far as I understand I have to open new issue.

I'm sorry I'm flooding this chat (: I solved my issue using the method from this post

git config --system http.postBuffer 524288000

Please consider contributing a documentation patch ( http://kallithea.readthedocs.io/en/latest/contributing.html ) if you can find a place where local git client configuration could fit in the Kallithea documentation. Or perhaps first discuss what you could have in mind. I guess the git documentation would be a better place.

Anyway, it still sounds like a waitress bug if a client interrupting a POST (before Kallithea gets invoked) can make it crash.

@kiilerix I think it perfectly fits into "Troubleshooting" section. Q: Git push fails with HTTP 411 code (Length Required)? A: Set variable http.postBuffer to a reasonably high value (e.g. 500M): git config --global http.postBuffer 524288000

Right. I totally agree.

But that will only cover a part of the problem - not the waitress crashing?

I still have no solution for that issue. Except switching to apache/nginx (:

See also issue #219 and mailing list thread https://lists.sfconservancy.org/pipermail/kallithea-general/2018q2/002667.html

Problem seems to be related to waitress but not only waitress.

Following workarounds exist:

• either stay on the stable branch, but use a different web server than waitress, e.g. uwsgi or apache.
• or switch to the default branch, in which case waitress seems to work fine, as do other web servers.