Issue #235: Intermittent "MySQL server has gone away"
| Reported by: | André |
| State: | new |
| Created on: | 2016-07-25 14:34 |
| Updated on: | 2016-07-26 21:52 |
Description
Kallithea 0.3.1
This happens from time to time. To load on the machine (2-core) is minimal.
(repository name replaced by XXXX)
2016-07-25 15:02:04.280 INFO [kallithea.RequestWrapper] IP: 192.168.55.160 Request to /XXXX/info/refs time: 0.002s
2016-07-25 15:02:04.281 ERROR [waitress] Exception when serving /repos/XXXX/info/refs
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/waitress/channel.py", line 337, in service
task.service()
File "/usr/local/lib/python2.7/dist-packages/waitress/task.py", line 173, in service
self.execute()
File "/usr/local/lib/python2.7/dist-packages/waitress/task.py", line 392, in execute
app_iter = self.channel.server.application(env, start_response)
File "/usr/local/lib/python2.7/dist-packages/paste/deploy/config.py", line 291, in __call__
return self.app(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/paste/gzipper.py", line 34, in __call__
response.gzip_start_response)
File "/usr/local/lib/python2.7/dist-packages/paste/cascade.py", line 130, in __call__
return self.apps[-1](environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/paste/registry.py", line 379, in __call__
app_iter = self.application(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/kallithea/lib/middleware/wrapper.py", line 43, in __call__
return self.application(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/kallithea/lib/middleware/https_fixup.py", line 49, in __call__
return self.application(environ, custom_start_response)
File "/usr/local/lib/python2.7/dist-packages/kallithea/lib/base.py", line 312, in __call__
return self._handle_request(environ, start_response)
File "/usr/local/lib/python2.7/dist-packages/kallithea/lib/middleware/simplegit.py", line 69, in _handle_request
if not self._check_ssl(environ):
File "/usr/local/lib/python2.7/dist-packages/kallithea/lib/base.py", line 257, in _check_ssl
if str2bool(Ui.get_by_key('push_ssl').ui_value):
File "/usr/local/lib/python2.7/dist-packages/kallithea/model/db.py", line 355, in get_by_key
return cls.query().filter(cls.ui_key == key).scalar()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2215, in scalar
ret = self.one()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2184, in one
ret = list(self)
File "/usr/local/lib/python2.7/dist-packages/kallithea/lib/caching_query.py", line 83, in __iter__
return Query.__iter__(self)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2227, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1449, in execute
params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
compiled_sql, distilled_params
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 331, in do_execute
cursor.execute(statement, parameters)
File "/usr/lib/pymodules/python2.7/MySQLdb/cursors.py", line 166, in execute
self.errorhandler(self, exc, value)
File "/usr/lib/pymodules/python2.7/MySQLdb/connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (OperationalError) (2006, 'MySQL server has gone away') 'SELECT ui.ui_id AS ui_ui_id, ui.ui_section AS ui_ui_section, ui.ui_key AS ui_ui_key, ui.ui_value AS ui_ui_value, ui.ui_active AS ui_ui_active \nFROM ui \nWHERE ui.ui_key = %s' ('push_ssl',)
Attachments
Comments
Comment by Mads Kiilerich, on 2016-07-25 14:41
This seems unlikely to be caused by Kallithea itself, but it is possible Kallithea could document a setup workaround or somehow could convince sqlalchemy to handle it better. I guess that depends on what the root cause is.
Comment by André, on 2016-07-25 16:06
Ah, true. Sorry, didn't think that it could be drilled down to a sub-component Kallithea uses, i.e. sqlalchemy. I'll look into it further, thanks.
Comment by André, on 2016-07-25 16:25
I'll admittedly never used SQLalchemy myself, but reading here and there about this kind of things, it does seems to be partly up to the application, e.g. this: http://docs.sqlalchemy.org/en/latest/core/pooling.html#disconnect-handling-pessimistic
Comment by Mads Kiilerich, on 2016-07-25 16:37
Perhaps. But why do you need it when nobody else have asked for it?
Comment by Andrej Shadura, on 2016-07-25 16:43
I think @tatsujin raised a valid point. We should handle disconnects properly.
Comment by André, on 2016-07-25 16:52
This could be triggered by the fact that the server has very little load at the moment because of vacations. So the sqlalchemy connections times out (more often).
Comment by Søren Løvborg, on 2016-07-25 17:30
This is a common MySQL problem, see https://dev.mysql.com/doc/refman/5.7/en/gone-away.html
In particular, increasing the server's wait_timeout seems like a reasonable suggestion.
Comment by Marcin Kuzminski, on 2016-07-26 21:52
When using Mysql you MUST set pool_recycle value that is =< than your wait_timeout set in mysql server (usually 3600s).
We had been struggling with such reports in the past in RhodeCode.
Here's an example config we using on RhodeCode installation and we know it fixes the problem, i think it should apply to Kallithea as well:
sqlalchemy.db1.url = mysql://root:secret@localhost/rhodecode # print the sql statements to output sqlalchemy.db1.echo = false # recycle the connections after this ammount of seconds sqlalchemy.db1.pool_recycle = 3600 sqlalchemy.db1.convert_unicode = true
It also must be set for any other db connection like DB based beaker sessions etc.