Issue #179: Crashes and error pages when kallithea is under load?
| Reported by: | zwarmapapa |
| State: | new |
| Created on: | 2015-12-17 15:44 |
| Updated on: | 2018-05-13 20:00 |
Description
I keep getting crashes, error pages and error report mails when I'm doing several things at once with kallithea, like for example git push 2 projects each with 80+ files changed, or refresh a lot of pages in the browser.
I use SQLite, which is probably the problem? I'm using SQLite because it makes it easier to make full backups.
The webpages just show error 500 internal server error when this happens, then it sends me one or several mails. Here are two of those mails:
URL: http://internal.myserver.com:5000/Applications
Module weberror.errormiddleware:162 in __call__
>> app_iter = self.application(environ, sr_checker)
Module kallithea.lib.middleware.sessionmiddleware:62 in __call__
>> return self.wrap_app(environ, session_start_response)
Module routes.middleware:131 in __call__
>> response = self.app(environ, start_response)
Module pylons.wsgiapp:107 in __call__
>> response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:312 in dispatch
>> return controller(environ, start_response)
Module kallithea.lib.base:442 in __call__
>> session.get('authuser'),
Module kallithea.lib.base:400 in _determine_auth_user
>> return AuthUser.from_cookie(session_authuser)
Module kallithea.lib.auth:640 in from_cookie
>> is_external_auth=cookie.get('is_external_auth', False),
Module kallithea.lib.auth:504 in __init__
>> is_user_loaded = self._fill_data(dbuser)
Module kallithea.lib.auth:527 in _fill_data
>> for k, v in dbuser.get_dict().iteritems():
Module kallithea.model.db:94 in get_dict
>> _json_attr = _json_attr()
Module kallithea.model.db:686 in __json__
>> data.update(self.get_api_data())
Module kallithea.model.db:664 in get_api_data
>> emails=user.emails,
Module kallithea.model.db:460 in emails
>> other = UserEmailMap.query().filter(UserEmailMap.user==self).all()
Module sqlalchemy.orm.query:2115 in all
Module kallithea.lib.caching_query:83 in __iter__
>> return Query.__iter__(self)
Module sqlalchemy.orm.query:2227 in __iter__
Module sqlalchemy.orm.query:2242 in _execute_and_instances
Module sqlalchemy.engine.base:1449 in execute
Module sqlalchemy.engine.base:1584 in _execute_clauseelement
Module sqlalchemy.engine.base:1698 in _execute_context
Module sqlalchemy.engine.base:1691 in _execute_context
Module sqlalchemy.engine.default:331 in do_execute
OperationalError: (OperationalError) no such table: user_email_map u'SELECT user_email_map.email AS user_email_map_email, user_email_map.email_id AS user_email_map_email_id, user_email_map.user_id AS user_email_map_user_id \nFROM user_email_map \nWHERE ? = user_email_map.user_id' (2,)
CGI Variables
CONTENT_TYPE '; charset="utf-8"'
HTTP_ACCEPT 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
HTTP_ACCEPT_ENCODING 'gzip, deflate'
HTTP_ACCEPT_LANGUAGE 'nl,en-US;q=0.7,en;q=0.3'
HTTP_CONNECTION 'keep-alive'
HTTP_HOST 'internal.myserver.com:5000'
HTTP_REFERER 'http://internal.myserver.com:5000/'
HTTP_USER_AGENT 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0'
PATH_INFO '/Applications'
REMOTE_ADDR '84.86.0.4'
REQUEST_METHOD 'GET'
SERVER_NAME 'internal.myserver.com'
SERVER_PORT '5000'
SERVER_PROTOCOL 'HTTP/1.1'
SERVER_SOFTWARE 'waitress'
WSGI Variables
application <kallithea.lib.middleware.sessionmiddleware.SecureSessionMiddleware object at 0x43f73d0>
beaker.get_session <bound method SecureSessionMiddleware._get_session of <kallithea.lib.middleware.sessionmiddleware.SecureSessionMiddleware object at 0x43f73d0>>
beaker.session {'_authentication_token': 'secret', 'authuser': {'is_external_auth': False, 'user_id': 2}, '_accessed_time': 1450363928.652568, '_creation_time': 1450363361.915775}
paste.registry <paste.registry.Registry object at 0x4c6b190>
paste.throw_errors True
pylons.controller <kallithea.controllers.admin.repo_groups.RepoGroupsController object at 0x4d21290>
pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons <pylons.util.PylonsContext object at 0x566c1d0>
pylons.routes_dict {'action': u'show_by_name', 'controller': u'admin/repo_groups', 'group_name': u'Applications'}
routes.route <routes.route.Route object at 0x3fe76d0>
routes.url <routes.util.URLGenerator object at 0x51b0610>
webob._parsed_query_vars (GET([]), '')
webob.adhoc_attrs {'language': 'en-us'}
wsgi process 'Multithreaded'
wsgi.file_wrapper <class 'waitress.buffers.ReadOnlyFileBasedBuffer'>
wsgiorg.routing_args (<routes.util.URLGenerator object at 0x51b0610>, {'action': u'show_by_name', 'controller': u'admin/repo_groups', 'group_name': u'Applications'})
URL: http://internal.myserver.com:5000/Applications/Projects/Java
Module weberror.errormiddleware:162 in __call__
>> app_iter = self.application(environ, sr_checker)
Module kallithea.lib.middleware.sessionmiddleware:62 in __call__
>> return self.wrap_app(environ, session_start_response)
Module routes.middleware:131 in __call__
>> response = self.app(environ, start_response)
Module pylons.wsgiapp:107 in __call__
>> response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:312 in dispatch
>> return controller(environ, start_response)
Module kallithea.lib.base:442 in __call__
>> session.get('authuser'),
Module kallithea.lib.base:400 in _determine_auth_user
>> return AuthUser.from_cookie(session_authuser)
Module kallithea.lib.auth:640 in from_cookie
>> is_external_auth=cookie.get('is_external_auth', False),
Module kallithea.lib.auth:482 in __init__
>> self.anonymous_user = User.get_default_user(cache=True)
Module kallithea.model.db:648 in get_default_user
>> user = User.get_by_username(User.DEFAULT_USER, cache=cache)
Module kallithea.model.db:561 in get_by_username
>> return q.scalar()
Module sqlalchemy.orm.query:2215 in scalar
Module sqlalchemy.orm.query:2184 in one
Module kallithea.lib.caching_query:80 in __iter__
>> return self.get_value(createfunc=lambda:
Module kallithea.lib.caching_query:99 in get_value
>> ret = cache.get_value(cache_key, createfunc=createfunc)
Module beaker.cache:305 in get
>> return self._get_value(key, **kw).get_value()
Module beaker.container:385 in get_value
>> v = self.createfunc()
Module kallithea.lib.caching_query:81 in <lambda>
>> list(Query.__iter__(self)))
Module sqlalchemy.orm.query:2227 in __iter__
Module sqlalchemy.orm.query:2242 in _execute_and_instances
Module sqlalchemy.engine.base:1449 in execute
Module sqlalchemy.engine.base:1584 in _execute_clauseelement
Module sqlalchemy.engine.base:1698 in _execute_context
Module sqlalchemy.engine.base:1691 in _execute_context
Module sqlalchemy.engine.default:331 in do_execute
DatabaseError: (DatabaseError) file is encrypted or is not a database u'SELECT users.firstname AS users_firstname, users.email AS users_email, users.user_data AS users_user_data, users.user_id AS users_user_id, users.username AS users_username, users.password AS users_password, users.active AS users_active, users.admin AS users_admin, users.lastname AS users_lastname, users.last_login AS users_last_login, users.extern_type AS users_extern_type, users.extern_name AS users_extern_name, users.api_key AS users_api_key, users.inherit_default_permissions AS users_inherit_default_permissions, users.created_on AS users_created_on \nFROM users \nWHERE users.username = ?' ('default',)
CGI Variables
CONTENT_TYPE '; charset="utf-8"'
HTTP_ACCEPT 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
HTTP_ACCEPT_ENCODING 'gzip, deflate'
HTTP_ACCEPT_LANGUAGE 'nl,en-US;q=0.7,en;q=0.3'
HTTP_CONNECTION 'keep-alive'
HTTP_HOST 'internal.myserver.com:5000'
HTTP_REFERER 'http://internal.myserver.com:5000/Applications/Projects'
HTTP_USER_AGENT 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0'
PATH_INFO '/Applications/Projects/Java'
REMOTE_ADDR '84.86.0.4'
REQUEST_METHOD 'GET'
SERVER_NAME 'internal.myserver.com'
SERVER_PORT '5000'
SERVER_PROTOCOL 'HTTP/1.1'
SERVER_SOFTWARE 'waitress'
WSGI Variables
application <kallithea.lib.middleware.sessionmiddleware.SecureSessionMiddleware object at 0x43f73d0>
beaker.get_session <bound method SecureSessionMiddleware._get_session of <kallithea.lib.middleware.sessionmiddleware.SecureSessionMiddleware object at 0x43f73d0>>
beaker.session {'_authentication_token': 'secret', 'authuser': {'is_external_auth': False, 'user_id': 2}, '_accessed_time': 1450363780.894332, '_creation_time': 1450363361.915775}
paste.registry <paste.registry.Registry object at 0x63e0690>
paste.throw_errors True
pylons.controller <kallithea.controllers.admin.repo_groups.RepoGroupsController object at 0x50c2310>
pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'}
pylons.pylons <pylons.util.PylonsContext object at 0x5124510>
pylons.routes_dict {'action': u'show_by_name', 'controller': u'admin/repo_groups', 'group_name': u'Applications/Projects/Java'}
routes.route <routes.route.Route object at 0x3fe76d0>
routes.url <routes.util.URLGenerator object at 0x4d00f50>
webob._parsed_query_vars (GET([]), '')
webob.adhoc_attrs {'language': 'en-us'}
wsgi process 'Multithreaded'
wsgi.file_wrapper <class 'waitress.buffers.ReadOnlyFileBasedBuffer'>
wsgiorg.routing_args (<routes.util.URLGenerator object at 0x4d00f50>, {'action': u'show_by_name', 'controller': u'admin/repo_groups', 'group_name': u'Applications/Projects/Java'})
Just to make it clear: I normally don't get these errors, this only happens when I put kallithea under load.
When one of these errors happen, kallithea keeps throwing these errors for about 5 minutes on every action I do, then everything works fine again.
Any idea what causes this and how it can be fixed?
PS: using kallithea commit 8bc8366 (Fri Nov 27 01:48:09 2015 +0100)
Attachments
Comments
Comment by Long Vu, on 2015-12-18 19:26
I have seen that too, with version 0.3 and also SQLite (using SQLite because we are still evaluating Kallithea).
Comment by Mads Kiilerich, on 2015-12-22 13:31
It seems like it must be something generic with pylons/sqlalcheme/sqllite.
http://stackoverflow.com/questions/14262771/why-am-i-suddenly-getting-operationalerror-no-such-table http://stackoverflow.com/questions/28126140/python-sqlite3-operationalerror-no-such-table suggest it could be caused by using relative path to the db ... but I assume you use the default 'sqlite:///%(here)s/kallithea.db' ?
Besides that: I suggest using a real database. For example with postgresql, it is very easy to make a backup with pg_dump - and without taking the system offline.
Comment by Thomas De Schampheleire, on 2018-05-13 20:00
Are you still seeing any such problem? If not we can close this issue, thanks.