Issue #235: Intermittent "MySQL server has gone away"
|Created on:||2016-07-25 14:34|
|Updated on:||2016-07-26 21:52|
This happens from time to time. To load on the machine (2-core) is minimal.
(repository name replaced by XXXX)
2016-07-25 15:02:04.280 INFO [kallithea.RequestWrapper] IP: 192.168.55.160 Request to /XXXX/info/refs time: 0.002s 2016-07-25 15:02:04.281 ERROR [waitress] Exception when serving /repos/XXXX/info/refs Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/waitress/channel.py", line 337, in service task.service() File "/usr/local/lib/python2.7/dist-packages/waitress/task.py", line 173, in service self.execute() File "/usr/local/lib/python2.7/dist-packages/waitress/task.py", line 392, in execute app_iter = self.channel.server.application(env, start_response) File "/usr/local/lib/python2.7/dist-packages/paste/deploy/config.py", line 291, in __call__ return self.app(environ, start_response) File "/usr/local/lib/python2.7/dist-packages/paste/gzipper.py", line 34, in __call__ response.gzip_start_response) File "/usr/local/lib/python2.7/dist-packages/paste/cascade.py", line 130, in __call__ return self.apps[-1](environ, start_response) File "/usr/local/lib/python2.7/dist-packages/paste/registry.py", line 379, in __call__ app_iter = self.application(environ, start_response) File "/usr/local/lib/python2.7/dist-packages/kallithea/lib/middleware/wrapper.py", line 43, in __call__ return self.application(environ, start_response) File "/usr/local/lib/python2.7/dist-packages/kallithea/lib/middleware/https_fixup.py", line 49, in __call__ return self.application(environ, custom_start_response) File "/usr/local/lib/python2.7/dist-packages/kallithea/lib/base.py", line 312, in __call__ return self._handle_request(environ, start_response) File "/usr/local/lib/python2.7/dist-packages/kallithea/lib/middleware/simplegit.py", line 69, in _handle_request if not self._check_ssl(environ): File "/usr/local/lib/python2.7/dist-packages/kallithea/lib/base.py", line 257, in _check_ssl if str2bool(Ui.get_by_key('push_ssl').ui_value): File "/usr/local/lib/python2.7/dist-packages/kallithea/model/db.py", line 355, in get_by_key return cls.query().filter(cls.ui_key == key).scalar() File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2215, in scalar ret = self.one() File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2184, in one ret = list(self) File "/usr/local/lib/python2.7/dist-packages/kallithea/lib/caching_query.py", line 83, in __iter__ return Query.__iter__(self) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2227, in __iter__ return self._execute_and_instances(context) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1449, in execute params) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement compiled_sql, distilled_params File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context context) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context context) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 331, in do_execute cursor.execute(statement, parameters) File "/usr/lib/pymodules/python2.7/MySQLdb/cursors.py", line 166, in execute self.errorhandler(self, exc, value) File "/usr/lib/pymodules/python2.7/MySQLdb/connections.py", line 35, in defaulterrorhandler raise errorclass, errorvalue OperationalError: (OperationalError) (2006, 'MySQL server has gone away') 'SELECT ui.ui_id AS ui_ui_id, ui.ui_section AS ui_ui_section, ui.ui_key AS ui_ui_key, ui.ui_value AS ui_ui_value, ui.ui_active AS ui_ui_active \nFROM ui \nWHERE ui.ui_key = %s' ('push_ssl',)
Comment by Mads Kiilerich, on 2016-07-25 14:41
This seems unlikely to be caused by Kallithea itself, but it is possible Kallithea could document a setup workaround or somehow could convince sqlalchemy to handle it better. I guess that depends on what the root cause is.
Comment by André, on 2016-07-25 16:06
Ah, true. Sorry, didn't think that it could be drilled down to a sub-component Kallithea uses, i.e. sqlalchemy. I'll look into it further, thanks.
Comment by André, on 2016-07-25 16:25
I'll admittedly never used SQLalchemy myself, but reading here and there about this kind of things, it does seems to be partly up to the application, e.g. this: http://docs.sqlalchemy.org/en/latest/core/pooling.html#disconnect-handling-pessimistic
Comment by Mads Kiilerich, on 2016-07-25 16:37
Perhaps. But why do you need it when nobody else have asked for it?
Comment by Andrej Shadura, on 2016-07-25 16:43
I think @tatsujin raised a valid point. We should handle disconnects properly.
Comment by André, on 2016-07-25 16:52
This could be triggered by the fact that the server has very little load at the moment because of vacations. So the sqlalchemy connections times out (more often).
Comment by Søren Løvborg, on 2016-07-25 17:30
This is a common MySQL problem, see https://dev.mysql.com/doc/refman/5.7/en/gone-away.html
In particular, increasing the server's
wait_timeout seems like a reasonable suggestion.
Comment by Marcin Kuzminski, on 2016-07-26 21:52
When using Mysql you MUST set
pool_recycle value that is =< than your
wait_timeout set in mysql server (usually 3600s).
We had been struggling with such reports in the past in RhodeCode.
Here's an example config we using on RhodeCode installation and we know it fixes the problem, i think it should apply to Kallithea as well:
sqlalchemy.db1.url = mysql://root:secret@localhost/rhodecode # print the sql statements to output sqlalchemy.db1.echo = false # recycle the connections after this ammount of seconds sqlalchemy.db1.pool_recycle = 3600 sqlalchemy.db1.convert_unicode = true
It also must be set for any other db connection like DB based beaker sessions etc.