Kallithea issues archive

Issue #308: Internal server error when showing full diff for file that contains '{' in its name when using Mercurial repository

Reported by: Branko Majic
State: resolved
Created on: 2018-02-10 00:11
Updated on: 2018-05-18 19:38

Description

When trying to view a full diff for a file that contains '{' in its name, an internal server error will be thrown. This happens with Mercurial repository. Reproduction steps within this issue are provided for the default branch (changeset 228dd29e79da as of this writing), but same issue happens with stable branch/version 0.3.3 as well (with different traceback).

Reproduction steps

  1. Set-up Kallithea (minimal development environment described in contributing.rst should suffice. Below instructions assume the user will be called admin for simplicity sake.

  2. Log-in into Kallithea and create a new Mercurial repository called test-diff-filename-with-bracket.

  3. Clone the empty repository:

    cd /tmp/
    hg clone http://admin@localhost:5000/test-diff-filename-with-bracket
    
  4. Add a file to repository that includes curly bracket ({) in its name, and push the changes:

    cd /tmp/test-diff-filename-with-bracket
    echo "This is a funny README file." > 'README{'
    hg add 'README{'
    hg commit -m "Added a funny README file with bracket"
    hg push
    
  5. Log-in into Kallithea and open the repository page.

  6. Click on the top revision in the list of latest changes.

  7. Click on the Show full diff for this file link right next to the README{ filename (icon should resemble a paper sheet with <> symbols in it).

Expected results

  1. In step (7), a full diff is shown for the file.

Actual results

  1. In step (7), an internal server error is thrown with the following traceback:

    2018-02-10 01:05:42.099 DEBUG [backlash] Traceback (most recent call last):
      File "/home/user/.virtualenvs/kallithea/lib/python2.7/site-packages/tg/appwrappers/session.py", line 71, in __call__
        response = self.next_handler(controller, environ, context)
      File "/home/user/.virtualenvs/kallithea/lib/python2.7/site-packages/tg/appwrappers/i18n.py", line 71, in __call__
        return self.next_handler(controller, environ, context)
      File "/home/user/.virtualenvs/kallithea/lib/python2.7/site-packages/tg/wsgiapp.py", line 285, in _dispatch
        return controller(environ, context)
      File "/home/user/projects/kallithea/kallithea/lib/base.py", line 553, in __call__
        return super(BaseController, self).__call__(environ, context)
      File "/home/user/.virtualenvs/kallithea/lib/python2.7/site-packages/tg/controllers/dispatcher.py", line 119, in __call__
        response = self._perform_call(context)
      File "/home/user/.virtualenvs/kallithea/lib/python2.7/site-packages/tg/controllers/dispatcher.py", line 108, in _perform_call
        r = self._call(action, params, remainder=remainder, context=context)
      File "/home/user/.virtualenvs/kallithea/lib/python2.7/site-packages/tg/controllers/decoratedcontroller.py", line 119, in _call
        output = controller_caller(context_config, bound_controller_callable, remainder, params)
      File "/home/user/.virtualenvs/kallithea/lib/python2.7/site-packages/tg/decorators.py", line 44, in _decorated_controller_caller
        return application_controller_caller(tg_config, controller, remainder, params)
      File "/home/user/.virtualenvs/kallithea/lib/python2.7/site-packages/tg/configuration/app_config.py", line 127, in call_controller
        return controller(*remainder, **params)
      File "<decorator-gen-88>", line 2, in diff
    
      File "/home/user/projects/kallithea/kallithea/lib/auth.py", line 810, in __wrapper
        return func(*fargs, **fkwargs)
      File "<decorator-gen-87>", line 2, in diff
    
      File "/home/user/projects/kallithea/kallithea/lib/auth.py", line 860, in __wrapper
        return func(*fargs, **fkwargs)
      File "/home/user/projects/kallithea/kallithea/controllers/files.py", line 682, in diff
        enable_comments=False)
      File "/home/user/projects/kallithea/kallithea/lib/diffs.py", line 202, in wrapped_diff
        context=line_context)
      File "/home/user/projects/kallithea/kallithea/lib/diffs.py", line 252, in get_gitdiff
        ignore_whitespace, context)
      File "/home/user/projects/kallithea/kallithea/lib/diffs.py", line 262, in get_diff
        ignore_whitespace=ignore_whitespace, context=context)
      File "/home/user/projects/kallithea/kallithea/lib/vcs/backends/hg/repository.py", line 260, in get_diff
        file_filter = match(self.path, '', [path])
      File "/home/user/.virtualenvs/kallithea/lib/python2.7/site-packages/mercurial/match.py", line 158, in match
        listsubrepos=listsubrepos, badfn=badfn)
      File "/home/user/.virtualenvs/kallithea/lib/python2.7/site-packages/mercurial/match.py", line 385, in __init__
        root)
      File "/home/user/.virtualenvs/kallithea/lib/python2.7/site-packages/mercurial/match.py", line 838, in _buildmatch
        regex, mf = _buildregexmatch(kindpats, globsuffix)
      File "/home/user/.virtualenvs/kallithea/lib/python2.7/site-packages/mercurial/match.py", line 874, in _buildregexmatch
        raise error.Abort(_("invalid pattern (%s): %s") % (k, p))
    Abort: invalid pattern (glob): README{
    

Additional information

The error occurs in kallithea.lib.vcs.backends.hg.repository.MercurialRepository.get_diff, in call file_filter = match(self.path, '', [path]).

The match function comes from mercurial.match module, and treats the passed-in patterns ([path]) as extended glob patterns by default.

In order to make this call work for paths that include globbing characters, the path must be either escaped, or (more easily), the match function should be told to treat all provided patterns as plain paths by passing-in additional default='path' parameter to it.

Attachments

Comments

Comment by Mads Kiilerich, on 2018-02-10 14:54

Thanks.

I guess it would be easy-ish to reproduce in kallithea/tests/functional/test_compare.py ?

Considering the other case where we also might want handle filenames starting with glob:, it would perhaps be better to prefix the filename with path: before passing to match?

Can you spot/test similar problems in other places?

Comment by Branko Majic, on 2018-02-10 15:09

From what I can tell, the only occurrences are on controllers compare.py, pullrequests.py, and changeset.py. So, theoretically, a fix could be done there instead (in those three places).

I can't see any other piece of code that might pass on a glob (or something else) towards this code path.

I already made a PR that fixes the issue by enforcing path type in Mercurial's get_diff itself. It should be noted that Git repository backend does not support any sort of pattern-matching, so any ability to specify the glob or regex paths would be Mercurial-specific - so not sure if this kind of dual behaviour should be introduced for something that is more of a standard interface method.

Comment by Branko Majic, on 2018-02-10 15:09

Wait a minute... I didn't send out PR for this one, nevermind :)

Comment by Branko Majic, on 2018-02-10 15:12

I just did a quick-hack testing with default='path'. So, fix can be done on controller or Mercurial backend level. I'm inclined towards the backend just so any code would benefit from it, but if you have other ideas, let me know. I'll try to implement a test for this - if it gets implemented on the backend side, I guess regular unittests should be more appropriate.

Comment by Mads Kiilerich, on 2018-02-10 18:00

It seems like a fix at vcs level would be the best option. But it is quite messy anyway, so no big deal.

But I don't see how specifying default='path' can help handle a file named glob:foo - that will override the default and use globbing for foo?

Comment by Branko Majic, on 2018-02-10 21:52

Ah... Right, good point. It completely slipped my mind :)

Looking into the match function signature again, there is also an option exact, which states patterns are actually filenames (include/exclude still apply). Doing some quick testing seems to result in correct behaviour (I tried some glob: and re: filenames with brackets etc.

This might be the better option then?

Comment by Mads Kiilerich, on 2018-02-11 14:11

right - exact=True seems perfect

Comment by Thomas De Schampheleire, on 2018-05-18 19:38

Fixed with b4a5632733d9