Kallithea issues archive

Issue #377: Sub-Repositories not correctly managed since upgrade to python 3

Reported by: Philippe Corbel
State: new
Created on: 2020-06-12 14:45
Updated on: 2020-06-15 15:38

Description

I’m using Kallithea for the SCM server in my company; it hosts several tens of different mercurial repositories.

Since the upgrade to 0.6.0 and after 0.6.1 (and the use of python v3 in place of v2), i’ve got several “500 Internal Server Error” on some of my repositories.

It is associated with this kind of stack trace:

TRACEBACK:
Traceback (most recent call last):
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/tg/appwrappers/session.py", line 71, in call
response = self.next_handler(controller, environ, context)
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/tg/appwrappers/i18n.py", line 71, in call
return self.next_handler(controller, environ, context)
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/tg/wsgiapp.py", line 243, in _dispatch
return controller(environ, context)
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/kallithea/lib/base.py", line 536, in call
return super(BaseController, self).call(environ, context)
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/tg/controllers/dispatcher.py", line 118, in call
response = self._perform_call(context)
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/tg/controllers/dispatcher.py", line 107, in _perform_call
r = self._call(action, params, remainder=remainder, context=context)
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/tg/controllers/decoratedcontroller.py", line 129, in _call
output = controller_caller(context_config, bound_controller_callable, remainder, params)
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/tg/controllers/decoration.py", line 21, in _decorated_controller_caller
return application_controller_caller(tg_config, controller, remainder, params)
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/tg/configurator/components/dispatch.py", line 114, in _call_controller
return controller(*remainder, **params)
File "<decorator-gen-40>", line 2, in index

File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/kallithea/lib/auth.py", line 614, in __wrapper
return func(*fargs, **fkwargs)
File "<decorator-gen-39>", line 2, in index

File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/kallithea/lib/auth.py", line 657, in _``wrapper
return func(*fargs, **fkwargs)
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/kallithea/controllers/files.py", line 193, in index
return render('files/files.html')
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/kallithea/lib/base.py", line 63, in render
return render_template({'url': url}, 'mako', template_path)
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/tg/render.py", line 208, in render
kwargs['result'] = render_function(template_name, tg_vars, **kwargs)
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/tg/renderers/mako.py", line 134, in ``call
cache_type=cache_type, cache_expire=cache_expire)
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/tg/render.py", line 274, in cached_template
return render_func()
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/tg/renderers/mako.py", line 131, in render_template
return Markup(template.render_unicode(**template_vars))
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/mako/template.py", line 482, in render_unicode
self, self.callable``, args, data, as_unicode=True
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/mako/runtime.py", line 883, in _render
**``kwargs_for_callable(callable``, data)
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/mako/runtime.py", line 920, in _render_context
_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/mako/runtime.py", line 947, in ``exec_template
callable``(context, *args, **kwargs)
File "_base_root_html", line 211, in render_body

File "_base_base_html", line 42, in render_body

File "files_files_html", line 130, in render_main

File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/mako/runtime.py", line 795, in ``include_file
callable``(ctx, **kwargs)
File "files_files_ypjax_html", line 44, in render_body

File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/mako/runtime.py", line 795, in ``include_file
callable``(ctx, **kwargs)
File "files_files_browser_html", line 77, in render_body

File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/kallithea/lib/vcs/nodes.py", line 502, in ``iter
for node in self.nodes:
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/kallithea/lib/vcs/utils/lazy.py", line 41, in ``get
value = self._func(obj)
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/kallithea/lib/vcs/nodes.py", line 487, in nodes
nodes = self.changeset.get_nodes(self.path)
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/kallithea/lib/vcs/backends/hg/changeset.py", line 333, in get_nodes
alias=als))
File "/opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/kallithea/lib/vcs/nodes.py", line 587, in ``init
self.path = name.rstrip('/')
TypeError: a bytes-like object is required, not 'str'

ENVIRON:
HTTP_ACCEPT: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,``/``;q=0.8'
HTTP_ACCEPT_ENCODING: 'gzip, deflate'
HTTP_ACCEPT_LANGUAGE: 'en-US,en;q=0.5'
HTTP_CONNECTION: 'keep-alive'
HTTP_COOKIE: 'kallithea=31c98bd3f0f4363d286b716e118b069533b8ce0592d0df4a4e7a4681a0a6235cd38045bc'
HTTP_HOST: 'kallithea'
HTTP_REFERER: '``http://kallithea/ESW/EBMR/EBMR-BEO'
HTTP_UPGRADE_INSECURE_REQUESTS: '1'
HTTP_USER_AGENT: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0'
PATH_INFO: '/ESW/EBMR/EBMR-BEO/files/tip/'
QUERY_STRING: ''
REMOTE_ADDR: '152.28.11.219'
REMOTE_HOST: '152.28.11.219'
REMOTE_PORT: '53047'
REQUEST_METHOD: 'GET'
SCRIPT_NAME: ''
SERVER_NAME: 'kallithea'
SERVER_PORT: '5000'
SERVER_PROTOCOL: 'HTTP/1.1'
SERVER_SOFTWARE: 'waitress'

WSGI:
beaker.cache: <beaker.cache.CacheManager object at 0x7f94e9b50588>
beaker.get_session: <bound method SessionApplicationWrapper._get_session of <tg.appwrappers.session.SessionApplicationWrapper object at 0x7f94e9b3f940>>
beaker.session: {'_domain': None, '_path': '/', '_creation_time': 1591709877.4332135, '_accessed_time': 1591972809.381667, 'authuser': {'user_id': 3, 'is_external_auth': False}, '_session_csrf_secret_token': '328770984834955768184576621553265253575'}
paste.registry: <tg.support.registry.Registry object at 0x7f9431d34358>
pylons.routes_dict: {'repo_name': 'ESW/EBMR/EBMR-BEO', 'revision': 'tip', 'f_path': '', 'action': 'index', 'controller': 'files'}
routes.route: <routes.route.Route object at 0x7f949bb502b0>
routes.url: <routes.util.URLGenerator object at 0x7f9431d60978>
tg.locals: <tg.wsgiapp.RequestLocals object at 0x7f944784ff28>
tg.routes_dict: {'repo_name': 'ESW/EBMR/EBMR-BEO', 'revision': 'tip', 'f_path': '', 'action': 'index', 'controller': 'files'}
webob._parsed_query_vars: (GET([]), '')
webob.adhoc_attrs: {'authuser': <AuthUser 3: 'corbel'>, 'ip_addr': '152.28.11.219', 'needs_csrf_check': False, 'routes_local': {'mapper': <kallithea.config.routing.Mapper object at 0x7f949c218860>, 'host': 'kallithea', 'protocol': 'http', 'redirect': <function redirect at 0x7f94ec790510>}}
wsgi.errors: <_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>
wsgi.file_wrapper: <class 'waitress.buffers.ReadOnlyFileBasedBuffer'>
wsgi.input: <_io.BytesIO object at 0x7f94320b0eb8>
wsgi.input_terminated: True
wsgi.multiprocess: False
wsgi.multithread: True
wsgi.run_once: False
wsgi.url_scheme: 'http'
wsgi.version: (1, 0)
wsgiorg.routing_args: (<routes.util.URLGenerator object at 0x7f9431d60978>, {'repo_name': 'ESW/EBMR/EBMR-BEO', 'revision': 'tip', 'f_path': '', 'action': 'index', 'controller': 'files'})

Attachments

Comments

Comment by Philippe Corbel, on 2020-06-12 14:53

I’m no Python developer but after some enquiries on the web, the issue TypeError: a bytes-like object is required, not 'str' is typical of code written for and working with Python2 being used with Python3 and not working anymore.

After several tries, I’ve pinpointed that the issue is with the repositories which have sub-repositories inside and the code concerned is the SubModuleNode class in the file /opt/kallithea_0_6_1/venv/lib/python3.7/site-packages/kallithea/lib/vcs/nodes.py.

The object created when accessing a sub-repo does not “receive” the right kind of type for an argument at its construction.

But, I can’t go any further on the analysis…

The issue can easily be reproduced while trying to get the “files” display of a repository.

It’s the same behaviour for a GIT repo with sub-modules…

Comment by Philippe Corbel, on 2020-06-12 14:59

Comment by Philippe Corbel, on 2020-06-12 15:02

Comment by Philippe Corbel, on 2020-06-12 15:06

Comment by Mads Kiilerich, on 2020-06-12 16:05

To help verify the fix of this, do you have a test case you can share? Perhaps a public repo or a script with commands to reproduce a repo that show the crash?

Comment by Mads Kiilerich, on 2020-06-12 16:12

Please, can you try with https://kallithea-scm.org/repos/kallithea/changeset/f48b12755d83cba1c8e6613e9df2b004ade3721f and see if that works for you?

Looking at the code, it seems unlikely that the Git problem is exactly the same. If it is different, perhaps file a separate issue.

Comment by Philippe Corbel, on 2020-06-15 15:02

Many thanks for the quick reaction.

I applied the patch on my Kallithea setup but it did not change anything: same traceback with the same resulting error!

You must be right about GIT, I will try to check if the issue is also visible with it…

By the way, I can’t give you access to my Kallithea server: it is inside my company’s network with no outgoing connexion.

I will try to set up something

Comment by Mads Kiilerich, on 2020-06-15 15:38

Thanks for testing.

Sorry to say it, but my first guess would be that if you don’t see any change, then the patch hasn’t been applied fully. Perhaps try to make an intentional error in the change to verify that it is applied in the right place.

I verified it fixed a similar subrepo issue for me. If your case is different, then we need more detailed description of how to reproduce.