Kallithea issues archive

Issue #235: Intermittent "MySQL server has gone away"

Reported by: André
State: new
Created on: 2016-07-25 14:34
Updated on: 2016-07-26 21:52


Kallithea 0.3.1

This happens from time to time. To load on the machine (2-core) is minimal.

(repository name replaced by XXXX)

2016-07-25 15:02:04.280 INFO  [kallithea.RequestWrapper] IP: Request to /XXXX/info/refs time: 0.002s
2016-07-25 15:02:04.281 ERROR [waitress] Exception when serving /repos/XXXX/info/refs
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/waitress/channel.py", line 337, in service
  File "/usr/local/lib/python2.7/dist-packages/waitress/task.py", line 173, in service
  File "/usr/local/lib/python2.7/dist-packages/waitress/task.py", line 392, in execute
    app_iter = self.channel.server.application(env, start_response)
  File "/usr/local/lib/python2.7/dist-packages/paste/deploy/config.py", line 291, in __call__
    return self.app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/paste/gzipper.py", line 34, in __call__
  File "/usr/local/lib/python2.7/dist-packages/paste/cascade.py", line 130, in __call__
    return self.apps[-1](environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/paste/registry.py", line 379, in __call__
    app_iter = self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/kallithea/lib/middleware/wrapper.py", line 43, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/kallithea/lib/middleware/https_fixup.py", line 49, in __call__
    return self.application(environ, custom_start_response)
  File "/usr/local/lib/python2.7/dist-packages/kallithea/lib/base.py", line 312, in __call__
    return self._handle_request(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/kallithea/lib/middleware/simplegit.py", line 69, in _handle_request
    if not self._check_ssl(environ):
  File "/usr/local/lib/python2.7/dist-packages/kallithea/lib/base.py", line 257, in _check_ssl
    if str2bool(Ui.get_by_key('push_ssl').ui_value):
  File "/usr/local/lib/python2.7/dist-packages/kallithea/model/db.py", line 355, in get_by_key
    return cls.query().filter(cls.ui_key == key).scalar()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2215, in scalar
    ret = self.one()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2184, in one
    ret = list(self)
  File "/usr/local/lib/python2.7/dist-packages/kallithea/lib/caching_query.py", line 83, in __iter__
    return Query.__iter__(self)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2227, in __iter__
    return self._execute_and_instances(context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1449, in execute
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 331, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/lib/pymodules/python2.7/MySQLdb/cursors.py", line 166, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/pymodules/python2.7/MySQLdb/connections.py", line 35, in defaulterrorhandler
    raise errorclass, errorvalue
OperationalError: (OperationalError) (2006, 'MySQL server has gone away') 'SELECT ui.ui_id AS ui_ui_id, ui.ui_section AS ui_ui_section, ui.ui_key AS ui_ui_key, ui.ui_value AS ui_ui_value, ui.ui_active AS ui_ui_active \nFROM ui \nWHERE ui.ui_key = %s' ('push_ssl',)



Comment by Mads Kiilerich, on 2016-07-25 14:41

This seems unlikely to be caused by Kallithea itself, but it is possible Kallithea could document a setup workaround or somehow could convince sqlalchemy to handle it better. I guess that depends on what the root cause is.

Comment by André, on 2016-07-25 16:06

Ah, true. Sorry, didn't think that it could be drilled down to a sub-component Kallithea uses, i.e. sqlalchemy. I'll look into it further, thanks.

Comment by André, on 2016-07-25 16:25

I'll admittedly never used SQLalchemy myself, but reading here and there about this kind of things, it does seems to be partly up to the application, e.g. this: http://docs.sqlalchemy.org/en/latest/core/pooling.html#disconnect-handling-pessimistic

Comment by Mads Kiilerich, on 2016-07-25 16:37

Perhaps. But why do you need it when nobody else have asked for it?

Comment by Andrej Shadura, on 2016-07-25 16:43

I think @tatsujin raised a valid point. We should handle disconnects properly.

Comment by André, on 2016-07-25 16:52

This could be triggered by the fact that the server has very little load at the moment because of vacations. So the sqlalchemy connections times out (more often).

Comment by Søren Løvborg, on 2016-07-25 17:30

This is a common MySQL problem, see https://dev.mysql.com/doc/refman/5.7/en/gone-away.html

In particular, increasing the server's wait_timeout seems like a reasonable suggestion.

Comment by Marcin Kuzminski, on 2016-07-26 21:52

When using Mysql you MUST set pool_recycle value that is =< than your wait_timeout set in mysql server (usually 3600s). We had been struggling with such reports in the past in RhodeCode.

Here's an example config we using on RhodeCode installation and we know it fixes the problem, i think it should apply to Kallithea as well:

sqlalchemy.db1.url = mysql://root:secret@localhost/rhodecode
# print the sql statements to output
sqlalchemy.db1.echo = false
# recycle the connections after this ammount of seconds
sqlalchemy.db1.pool_recycle = 3600
sqlalchemy.db1.convert_unicode = true

It also must be set for any other db connection like DB based beaker sessions etc.