Issue #219: waitress exception when serving file

Reported by:	Igor Tretyak
State:	resolved
Created on:	2016-05-24 11:14
Updated on:	2018-06-15 19:47

Description

Hello,

Kallithea runs in docker container.

pip modules versions

Babel==1.3
Beaker==1.6.4
FormEncode==1.2.6
Kallithea==0.3
Mako==1.0.0
Markdown==2.2.1
MarkupSafe==0.23
MySQL-python==1.2.5
Paste==2.0.2
PasteDeploy==1.5.2
PasteScript==2.0.2
Pygments==2.1
Pylons==1.0.2
Routes==1.13
SQLAlchemy==0.7.10
Tempita==0.5.3.dev0
URLObject==2.3.4
WebError==0.11
WebHelpers==1.3
WebOb==1.1.1
WebTest==1.4.3
Whoosh==2.5.7
amqplib==1.0.2
anyjson==0.3.3
argparse==1.2.1
celery==2.2.10
chardet==2.0.1
colorama==0.2.5
decorator==4.0.6
docutils==0.11
dulwich==0.9.9
funcsigs==0.4
html5lib==0.999
kombu==1.5.1
mercurial==3.6.3
mock==1.3.0
nose==1.3.7
pbr==1.8.1
psycopg2==2.6.1
py-bcrypt==0.4
pyparsing==1.5.7
python-dateutil==1.5
python-ldap==2.4.10
pytz==2015.7
repoze.lru==0.6
requests==2.2.1
simplejson==3.8.1
six==1.10.0
urllib3==1.7.1
waitress==0.8.8
wheel==0.24.0
wsgiref==0.1.2

# docker version
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:25:01 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:25:01 UTC 2015
 OS/Arch:      linux/amd64

No OS errors, regarding FS or disk, was found.

faced with exeption:

2016-05-24 09:01:30.601 ERROR [waitress] uncaptured python exception, closing channel <waitress.server.TcpWSGIServer listening 0.0.0.0:5000 at 0x7f2c0050f610> (<class 'socket.error'>:[Errno 9] Bad file descriptor [/usr/lib/python2.7/asyncore.py|read|83] [/usr/lib/python2.7/asyncore.py|handle_read_event|443] [/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/server.py|handle_accept|147] [/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/channel.py|__init__|80] [/usr/lib/python2.7/asyncore.py|__init__|250] [/usr/lib/python2.7/socket.py|meth|224])
2016-05-24 09:01:30.601 WARNI [waitress] unhandled close event
2016-05-24 09:01:30.608 ERROR [waitress] Exception when serving /Library-Android/skia/libwebp.git/git-upload-pack
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/channel.py", line 337, in service
    task.service()
  File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/task.py", line 173, in service
    self.execute()
  File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/task.py", line 424, in execute
    self.write(chunk)
  File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/task.py", line 307, in write
    channel.write_soon(towrite)
  File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/channel.py", line 319, in write_soon
    self.outbufs[-1].append(data)
  File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/buffers.py", line 253, in append
    self._set_large_buffer()
  File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/buffers.py", line 236, in _set_large_buffer
    self.buf = TempfileBasedBuffer(self.buf)
  File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/buffers.py", line 116, in __init__
    FileBasedBuffer.__init__(self, self.newfile(), from_buffer)
  File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/buffers.py", line 120, in newfile
    return TemporaryFile('w+b')
  File "/usr/lib/python2.7/tempfile.py", line 498, in TemporaryFile
    _os.close(fd)
OSError: [Errno 9] Bad file descriptor

Kallithea sometimes fails on pulling file from same folder. I can't reproduce it manually by executing 'git clone'.

Attachments

Comments

Comment by Igor Tretyak, on 2016-05-24 11:17

Comment by Igor Tretyak, on 2016-05-24 11:30

Comment by Mads Kiilerich, on 2016-05-24 16:53

Why not use the latest stable Kallithea version?

The stacktrace shows all waitress - no Kallithea there ...

What does 'serving file' mean?

I guess you see it with Git only? Can you explain in details how to reproduce?

Comment by Igor Tretyak, on 2016-05-25 07:57

I haven't seen any waitress or related updates in stable Kallithea. Anyway, tonight I updated Kallithea, Docker and all other system components. We'll see whether the bug gone away.

Comment by Thomas De Schampheleire, on 2018-06-13 20:17

@fragpit Did this problem get resolved? What exactly did you change? (there is another report on the mailing list about this).

Comment by Igor Tretyak, on 2018-06-13 20:44

I switched to uwsgi and it was gone.

Comment by Thomas De Schampheleire, on 2018-06-15 19:45

Another 'Bad file descriptor' problem (happening on git push of a large file, e.g. 1MB or larger) was reported on the mailing list. I reproduced the issue on stable branch (0.3.5) but the problem is gone on the default branch. In both cases, waitress and sqlite were used. See https://lists.sfconservancy.org/pipermail/kallithea-general/2018q2/002667.html

There is also issue ~~#229~~ which looks related.

In all three cases it seems to be something with waitress. But because backstepping the version of waitress on the default branch to the same version as used on stable (waitress 0.8.8) does not reintroduce the problem, there thus must be some other relevant factor.

In any case, there are following workarounds: - on stable, use a different web server than waitress - or switch to default branch (with any web server)

Hence, I will close this issue.

Comment by Thomas De Schampheleire, on 2018-06-15 19:47

Kallithea issues archive