Issue #340: 0.4.0: dispatch.wsgi terminates due to LC_ALL or LANG encoding.

Reported by:	Edmund Wong
State:	resolved
Created on:	2019-04-01 10:10
Updated on:	2019-05-12 20:24

Description

I upgraded 0.3.x to 0.4.0 and came across a LC_ALL/LANG issue.

[Mon Apr 01 09:58:06 2019] [error] 2019-04-01 09:58:06.696 DEBUG [tg.appwrappers.caching] Caching enabled: True -> {'data_dir': '/var/www/kali/srv/data/cache/data', 'cache_regions': {'long_term': {'expire': 36000, 'data_dir': '/var/www/kali/srv/data/cache/data', 'type': 'memory', 'enabled': True, 'lock_dir': '/var/www/kali/srv/data/cache/lock', 'key_length': 256}, 'sql_cache_short': {'expire': 10, 'data_dir': '/var/www/kali/srv/data/cache/data', 'type': 'memory', 'enabled': True, 'lock_dir': '/var/www/kali/srv/data/cache/lock', 'key_length': 256}, 'short_term': {'expire': 60, 'data_dir': '/var/www/kali/srv/data/cache/data', 'type': 'memory', 'enabled': True, 'lock_dir': '/var/www/kali/srv/data/cache/lock', 'key_length': 256}}, 'expire': None, 'log_file': None, 'type': 'memory', 'lock_dir': '/var/www/kali/srv/data/cache/lock'}
[Mon Apr 01 09:58:06 2019] [error] 2019-04-01 09:58:06.696 DEBUG [tg.appwrappers.mingflush] MingSessionFlush enabled: False -> {'autoflush': False}
[Mon Apr 01 09:58:06 2019] [error] 2019-04-01 09:58:06.696 DEBUG [tg.appwrappers.transaction_manager] TransactionManager enabled: False -> {'attempts': 1, 'enabled': False, 'commit_veto': None}
[Mon Apr 01 09:58:06 2019] [error] 2019-04-01 09:58:06.696 DEBUG [tg.appwrappers.errorpage] ErrorPageApplicationWrapper enabled: True -> {'content_types': ['text/html', None], 'handle_exceptions': True, 'path': '/error/document', 'status_codes': [400, 401, 403, 404], 'enabled': True}
[Mon Apr 01 09:58:06 2019] [error] 2019-04-01 09:58:06.697 ERROR [kallithea.config.app_cfg] Cannot encode Unicode paths to file system encoding 'ANSI_X3.4-1968'
[Mon Apr 01 09:58:06 2019] [error] 2019-04-01 09:58:06.697 ERROR [kallithea.config.app_cfg] Note: Environment variable LANG is 'en_US.UTF-8' - perhaps change it to some other value from 'locale -a', like 'C.UTF-8' or 'en_US.UTF-8'
[Mon Apr 01 09:58:06 2019] [error] 2019-04-01 09:58:06.697 ERROR [kallithea.config.app_cfg] Terminating ...

Looking at the code in app_cfg.py, I noticed it's testing out u\xe9'.encode(sys.getfilesystemencoding())

But it ends up failing and Terminating.

I go into python and get the following results:

import sys

print(sys.getfilesystemencoding())

prints out 'UTF-8'.

So I copy part of the code form setup-configuration(app) as follows:

import os
import sys
 try:
     u'\xe9'.encode(sys.getfilesystemencoding())
 except UnicodeEncodeError:
     print("Cannot encode Unicode paths to file system encoding %r", sys.getfilesystemencoding())
     for var in ['LC_CTYPE', 'LC_ALL', 'LANG']:
         if var in os.environ:
             val = os.environ[var]
             print("%s is %r" % (var, val))
             break
         else:
             print("No locale setting found...")
     print("Terminating...")

prints : '\xc3\xa9'

so I add the following before the try: block:

print(sys.getfilesystemencoding())

and it prints :

sys : 'ANSI_X3.4-1968'

So I add:

os.environ["LC_ALL"] = "en_US.UTF-8" os.environ["LANG"] = "en_US.UTF-8"

to the dispatch.wsgi.

But I still get the same ANSI_X3.4-1968 result.

These are the settings I have in my.ini:

i18n.enabled = false lang =

I think I might have screwed up the upgrade... though I'm not sure where as I followed the upgrade instructions.. (though I hazard a guess I missed some points somewhere...)

Attachments

Comments

Comment by Edmund Wong, on 2019-04-01 10:11

Comment by Thomas De Schampheleire, on 2019-04-01 11:03

What is the output of 'locale -a' on the server? It should contain en_US.UTF-8, the value you specify. Alternatively, use another UTF-8 value which is present in the output of locale -a.

Comment by Mads Kiilerich, on 2019-04-01 12:59

Also: Notice that Python does evil things with setting system encoding based on environment at startup. Setting it inside Python in os.environ will probably not work as expected. It should probably be set in the environment before launching Python.

Comment by Edmund Wong, on 2019-04-01 14:24

@patrickdepinguin locale -a apparently lists en_US.utf8 instead of what I had thought would be (en_US.UTF-8).

@Mads Killerich bingo. With the results from 'locale -a' and your suggestion, I added "lang=en_US.utf8" to the WSGIDaemonProcess entry in the apache configuration. Now it's working

Should this be written as a note in the documentation? I'm running Kallithea in a CentOS 6.9 system.

Comment by Mads Kiilerich, on 2019-04-01 15:28

Are you sure en_US.UTF-8 doesn't work? I thought it aliased one spelling to the other. But perhaps not in all places. That would be nice to get verified an extra time, now when you get it working and reliably can test while changing one parameter at a time.

Indeed - we should update documentation to cover it better. We added this big early failure, just so we can get reliable feedback early, instead of random and hard-to-reproduce failures later on at runtime.

Comment by Thomas De Schampheleire, on 2019-04-01 19:01

If I set an invalid 'LANG' like 'foo', I get an error from TurboGears2:

2019-04-01 20:49:24.344 ERROR [gearbox] Failed to load application
2019-04-01 20:49:24.345 ERROR [gearbox] unsupported locale setting
Traceback (most recent call last):
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/main.py", line 172, in _run_subcommand
    return cmd.run(parsed_args)
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/command.py", line 31, in run
    self.take_action(parsed_args)
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/commands/serve.py", line 280, in take_action
    relative_to=base, global_conf=parsed_vars)
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/commands/serve.py", line 311, in loadapp
    return loadapp(app_spec, name=name, relative_to=relative_to, **kw)
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 253, in loadapp
    return loadobj(APP, uri, name=name, **kw)
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 277, in loadobj
    global_conf=global_conf)
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 302, in loadcontext
    global_conf=global_conf)
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 326, in _loadconfig
    return loader.get_context(object_type, name, global_conf)
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 459, in get_context
    section)
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 481, in _context_from_use
    object_type, name=use, global_conf=global_conf)
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 412, in get_context
    global_conf=global_conf)
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 302, in loadcontext
    global_conf=global_conf)
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 334, in _loadegg
    return loader.get_context(object_type, name, global_conf)
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 625, in get_context
    object_type, name=name)
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 651, in find_egg_entry_point
    possible.append((entry.load(), protocol, entry.name))
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2411, in load
    return self.resolve()
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2417, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/config/middleware.py", line 17, in <module>
    from kallithea.config.app_cfg import base_config
  File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/config/app_cfg.py", line 34, in <module>
    from kallithea.lib.middleware.simplegit import SimpleGit
  File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/middleware/simplegit.py", line 42, in <module>
    from kallithea.lib.base import BaseVCSController, check_locking_state
  File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/base.py", line 53, in <module>
    from kallithea.lib import auth_modules
  File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/auth_modules/__init__.py", line 24, in <module>
    from kallithea.lib.auth import PasswordGenerator, AuthUser
  File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/auth.py", line 54, in <module>
    from kallithea.lib.utils import get_repo_slug, get_repo_group_slug, \
  File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/utils.py", line 39, in <module>
    from kallithea.lib.vcs.utils.hgcompat import ui, config
  File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/vcs/utils/hgcompat.py", line 11, in <module>
    from mercurial import localrepo
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/mercurial/localrepo.py", line 26, in <module>
    from . import (
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/mercurial/context.py", line 26, in <module>
    from . import (
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/mercurial/subrepo.py", line 23, in <module>
    from . import (
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/mercurial/cmdutil.py", line 23, in <module>
    from . import (
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/mercurial/crecord.py", line 33, in <module>
    locale.setlocale(locale.LC_ALL, u'')
  File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib64/python2.7/locale.py", line 581, in setlocale
    return _setlocale(category, locale)
Error: unsupported locale setting

This is also true for LANG settings that are in itself valid but are not present on the system (i.e. not listed in locale -a), for example: nl_BE.utf8.

When the locale exists but is not unicode capable, like LANG=C, then the error is:

2019-04-01 20:52:48.329 ERROR [kallithea.config.app_cfg] Cannot encode Unicode paths to file system encoding 'ANSI_X3.4-1968'
2019-04-01 20:52:48.329 ERROR [kallithea.config.app_cfg] Note: Environment variable LANG is 'C' - perhaps change it to some other value from 'locale -a', like 'C.UTF-8' or 'en_US.UTF-8'
2019-04-01 20:52:48.329 ERROR [kallithea.config.app_cfg] Terminating ...

If I set LANG to a valid locale, present on the system, Kallithea starts correctly, regardless of the exact spelling of the utf-8 suffix:

en_US.utf8
en_US.utf-8
en_US.UTF8
en_US.UTF-8

So, I think the main clue that fixed the problem for @ewongcc was adding a valid locale (whatever the spelling) in the apache configuration under 'WSGIDaemonProcess'.

I think that the "Apache with mod_wsgi" section docs/setup.rst should indeed be adapted to give a better hint about this. @kiilerix, if you have experience with Apache, please propose a modification, I didn't use it yet.

Comment by Mads Kiilerich, on 2019-04-01 23:59

I have only used Kallithea in production in an international environment where all filenames were pure ASCII.

I guess docs/overview.rst has to mention the locale challenges and say something about UTF-8 being highly recommend (and nothing else feasible), and that a valid utf8 locale has to be set in the environment before starting the Python interpreter. I don't know what we can say about setting the right valid locale. But the "Apache with mod_wsgi" section of docs/setup.rst should probably use the lang and locale options in the examples.

http://blog.dscpl.com.au/2014/09/setting-lang-and-lcall-when-using.html and https://modwsgi.readthedocs.io/en/develop/configuration-directives/WSGIDaemonProcess.html says everything pretty well. I'm not sure we can say it better. So perhaps just apply that advice in our examples, and link to them?

(In general, searching for "Graham Dumpleton mod_wsgi kallithea" gives some very qualified hits.)

For other ways of hosting the WSGI process (and for other Kallithea invocations), we should probably also mention the need for setting environment variables. (If Kallihea were to take care of that, we would probably have to use wrapper scripts. That would probably not be a good idea.)

Comment by Mads Kiilerich, on 2019-04-30 23:01

I propose something like https://kallithea-scm.org/repos/kallithea-incoming/changeset/470436f288a7aef319a7c0e45ce56a0eb814804f

Comment by Thomas De Schampheleire, on 2019-05-12 20:24

Documentation updated in 9bae11163c31. This should help users with similar issues in the future.

Kallithea issues archive