Kallithea issues archive

Issue #378: UnicodeDecodeError after upgrade

Reported by: Akos
State: new
Created on: 2020-06-15 13:25
Updated on: 2020-06-16 10:11

Description

I am upgradeing kallithea 0.3.2 to 0.6.1, and after upgrade when I try to access users and repos than contain unicode names, I got this error in logs:

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 1: character maps to <undefined>

‌

Full logfile when I try to access Users page (I have LDAP users with unicode charactes):

Jun 15 16:23:56 hg gearbox[17152]: Traceback (most recent call last):
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/appwrappers/session.py", line 71, in call
Jun 15 16:23:56 hg gearbox[17152]: response = self.next_handler(controller, environ, context)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/appwrappers/i18n.py", line 71, in call
Jun 15 16:23:56 hg gearbox[17152]: return self.next_handler(controller, environ, context)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/wsgiapp.py", line 243, in _dispatch
Jun 15 16:23:56 hg gearbox[17152]: return controller(environ, context)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/kallithea/lib/base.py", line 536, in call
Jun 15 16:23:56 hg gearbox[17152]: return super(BaseController, self).call(environ, context)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/controllers/dispatcher.py", line 118, in call
Jun 15 16:23:56 hg gearbox[17152]: response = self._perform_call(context)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/controllers/dispatcher.py", line 107, in _perform_call
Jun 15 16:23:56 hg gearbox[17152]: r = self._call(action, params, remainder=remainder, context=context)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/controllers/decoratedcontroller.py", line 129, in _call
Jun 15 16:23:56 hg gearbox[17152]: output = controller_caller(context_config, bound_controller_callable, remainder, params)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/controllers/decoration.py", line 21, in _decorated_controller_caller
Jun 15 16:23:56 hg gearbox[17152]: return application_controller_caller(tg_config, controller, remainder, params)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/configurator/components/dispatch.py", line 114, in _call_controller
Jun 15 16:23:56 hg gearbox[17152]: return controller(*remainder, **params)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/kallithea/controllers/admin/users.py", line 70, in index
Jun 15 16:23:56 hg gearbox[17152]: .order_by(func.lower(User.username))
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3319, in all
Jun 15 16:23:56 hg gearbox[17152]: return list(self)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3481, in iter
Jun 15 16:23:56 hg gearbox[17152]: return self._execute_and_instances(context)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances
Jun 15 16:23:56 hg gearbox[17152]: result = conn.execute(querycontext.statement, self._params)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
Jun 15 16:23:56 hg gearbox[17152]: return meth(self, multiparams, params)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
Jun 15 16:23:56 hg gearbox[17152]: return connection._execute_clauseelement(self, multiparams, params)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement
Jun 15 16:23:56 hg gearbox[17152]: distilled_params,
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1324, in execute_context
Jun 15 16:23:56 hg gearbox[17152]: e, statement, parameters, cursor, context
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1521, in handle_dbapi_exception
Jun 15 16:23:56 hg gearbox[17152]: util.raise(exc_info[1], with_traceback=exc_info[2])
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise
Jun 15 16:23:56 hg gearbox[17152]: raise exception
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
Jun 15 16:23:56 hg gearbox[17152]: cursor, statement, parameters, context
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
Jun 15 16:23:56 hg gearbox[17152]: cursor.execute(statement, parameters)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute
Jun 15 16:23:56 hg gearbox[17152]: res = self._query(query)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 317, in _query
Jun 15 16:23:56 hg gearbox[17152]: self._post_get_result()
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 352, in _post_get_result
Jun 15 16:23:56 hg gearbox[17152]: self._rows = self._fetch_row(0)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 325, in _fetch_row
Jun 15 16:23:56 hg gearbox[17152]: return self._result.fetch_row(size, self._fetch_type)
Jun 15 16:23:56 hg gearbox[17152]: File "/usr/lib/python3.6/encodings/cp1252.py", line 15, in decode
Jun 15 16:23:56 hg gearbox[17152]: return codecs.charmap_decode(input,errors,decoding_table)
Jun 15 16:23:56 hg gearbox[17152]: UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 1: character maps to <undefined>
Jun 15 16:23:56 hg gearbox[17152]: Traceback (most recent call last):
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/appwrappers/session.py", line 71, in call
Jun 15 16:23:56 hg gearbox[17152]: response = self.next_handler(controller, environ, context)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/appwrappers/i18n.py", line 71, in call
Jun 15 16:23:56 hg gearbox[17152]: return self.next_handler(controller, environ, context)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/wsgiapp.py", line 243, in _dispatch
Jun 15 16:23:56 hg gearbox[17152]: return controller(environ, context)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/kallithea/lib/base.py", line 536, in call
Jun 15 16:23:56 hg gearbox[17152]: return super(BaseController, self).call(environ, context)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/controllers/dispatcher.py", line 118, in call
Jun 15 16:23:56 hg gearbox[17152]: response = self._perform_call(context)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/controllers/dispatcher.py", line 107, in _perform_call
Jun 15 16:23:56 hg gearbox[17152]: r = self._call(action, params, remainder=remainder, context=context)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/controllers/decoratedcontroller.py", line 129, in _call
Jun 15 16:23:56 hg gearbox[17152]: output = controller_caller(context_config, bound_controller_callable, remainder, params)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/controllers/decoration.py", line 21, in _decorated_controller_caller
Jun 15 16:23:56 hg gearbox[17152]: return application_controller_caller(tg_config, controller, remainder, params)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/tg/configurator/components/dispatch.py", line 114, in _call_controller
Jun 15 16:23:56 hg gearbox[17152]: return controller(*remainder, **params)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/kallithea/controllers/admin/users.py", line 70, in index
Jun 15 16:23:56 hg gearbox[17152]: .order_by(func.lower(User.username))
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3319, in all
Jun 15 16:23:56 hg gearbox[17152]: return list(self)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3481, in iter
Jun 15 16:23:56 hg gearbox[17152]: return self._execute_and_instances(context)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances
Jun 15 16:23:56 hg gearbox[17152]: result = conn.execute(querycontext.statement, self._params)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
Jun 15 16:23:56 hg gearbox[17152]: return meth(self, multiparams, params)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
Jun 15 16:23:56 hg gearbox[17152]: return connection._execute_clauseelement(self, multiparams, params)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement
Jun 15 16:23:56 hg gearbox[17152]: distilled_params,
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1324, in execute_context
Jun 15 16:23:56 hg gearbox[17152]: e, statement, parameters, cursor, context
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1521, in handle_dbapi_exception
Jun 15 16:23:56 hg gearbox[17152]: util.raise(exc_info[1], with_traceback=exc_info[2])
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise
Jun 15 16:23:56 hg gearbox[17152]: raise exception
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
Jun 15 16:23:56 hg gearbox[17152]: cursor, statement, parameters, context
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
Jun 15 16:23:56 hg gearbox[17152]: cursor.execute(statement, parameters)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute
Jun 15 16:23:56 hg gearbox[17152]: res = self._query(query)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 317, in _query
Jun 15 16:23:56 hg gearbox[17152]: self._post_get_result()
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 352, in _post_get_result
Jun 15 16:23:56 hg gearbox[17152]: self._rows = self._fetch_row(0)
Jun 15 16:23:56 hg gearbox[17152]: File "/srv/kallithea/venv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 325, in _fetch_row
Jun 15 16:23:56 hg gearbox[17152]: return self._result.fetch_row(size, self._fetch_type)
Jun 15 16:23:56 hg gearbox[17152]: File "/usr/lib/python3.6/encodings/cp1252.py", line 15, in decode
Jun 15 16:23:56 hg gearbox[17152]: return codecs.charmap_decode(input,errors,decoding_table)
Jun 15 16:23:56 hg gearbox[17152]: UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 1: character maps to <undefined>

Attachments

Comments

Comment by Mads Kiilerich, on 2020-06-15 14:28

What is the sqlalchemy.url line in your .ini file?

From a mysql prompt, what do you get from SELECT * from information_schema.SCHEMATA; ?

Comment by Mads Kiilerich, on 2020-06-15 23:05

(I don’t like to jump to conclusions too early, but I think/hope ttps://kallithea-scm.org/repos/kallithea-incoming/changeset/5273e61453ebacc21f4930e297551078809496b3 and parent revision solves the problem … and hints that you will have to alter the database.)

Comment by Akos, on 2020-06-16 07:16

sqlalchemy.url = mysql://kallithea:xxx@localhost/kallithea

+--------------+--------------------+----------------------------+------------------------+----------+--------------------+
| CATALOG_NAME | SCHEMA_NAME | DEFAULT_CHARACTER_SET_NAME | DEFAULT_COLLATION_NAME | SQL_PATH | DEFAULT_ENCRYPTION |
+--------------+--------------------+----------------------------+------------------------+----------+--------------------+
| def | mysql | utf8mb4 | utf8mb4_0900_ai_ci | NULL | NO |
| def | information_schema | utf8 | utf8_general_ci | NULL | NO |
| def | performance_schema | utf8 | utf8_general_ci | NULL | NO |
| def | kallithea | utf8 | utf8_general_ci | NULL | NO |
| def | sys | utf8mb4 | utf8mb4_0900_ai_ci | NULL | NO |
+--------------+--------------------+----------------------------+------------------------+----------+--------------------+

Comment by Akos, on 2020-06-16 08:11

I got the bug on ubuntu 18.04 with python 3.6. Now I upgrade the system to ubuntu 20.04 with python 3.8.2, delete the old venv and recreate it, and not it works. I can display the users page.

Install steps (same on both os):

  • python3 -m venv /srv/kallithea/venv
  • . /srv/kallithea/venv/bin/activate
  • pip install --upgrade pip setuptools
  • pip install --upgrade kallithea python-ldap mysqlclient
  • install nodejs
  • kallithea-cli front-end-build
  • gearbox serve -c my.ini

The unicode characters didn’t displayed correctly, but is not a big problem:

Correct version: Zoltán, Gáspár, Beáta…

‌

Comment by Akos, on 2020-06-16 08:19

Now I login with my AD user and kallithea probably made a resync with AD and correct the problematic character.

Before login:

After login:

You can close the bug. Eventually if you want I can retest with ubuntu 18.04 and python 3.6.

‌

Comment by Mads Kiilerich, on 2020-06-16 10:11

sqlalchemy.url = mysql://kallithea:xxx@localhost/kallithea

‌

That is missing the charset introduced with 0.4 in https://kallithea-scm.org/repos/kallithea/changeset/6353b5e87091 . I don’t know exactly what made it necessary - it seems like it worked for you without. I guess it is a part of the bigger story about how charsets can be made to work with locales, but full unicode/utf8 is much more simple and reliable once it works. And MySQL doesn’t default to this simple and reliable way …

Your initial trace shows that the runtime connection layer thus assumed that the bytes from the database was using / should use latin1. Using explicit utf8 charset in the URL would at least have given another stacktrace, without 1252.

Nice that your database already used utf8 as charset and collation - that’s on the right track.

Assuming you now have a full stack of unicode/utf8, I don’t know why you saw these typical “rendering utf8 as if it was latin1”. Possibly, the database content was “wrong” (but worked anyway due to consistent “wrong” encodings). That could perhaps be fixed somehow on the database side.

It is possible that something in the py3 unicode stack changed or fixed a bug between 3.6 and 3.8, but I think it is more likely that it is due to “something else”.

‌