Kallithea issues archive

Issue #130: [unicode] Some characters in filenames cause the indexing to fail

Reported by: Samuel Delisle
State: new
Created on: 2015-05-04 17:05
Updated on: 2015-08-06 19:53


While creating the index necessary for full-text search, it looks like filenames with special accented characters (è, à, é, ...?) within a repository cause an exception.

Here's the output of the command: /opt/kallithea/venv/bin/paster make-index /opt/kallithea/data/production.ini

/opt/kallithea/venv/bin/paster make-index /opt/kallithea/data/production.ini
2015-05-04 12:54:42.587 INFO  [kallithea.model] initializing db for sqlite:////opt/kallithea/data/kallithea.db?timeout=60
2015-05-04 12:54:42.747 INFO  [kallithea.model.scm] scanning for repositories in /opt/kallithea/repos
Traceback (most recent call last):
  File "/opt/kallithea/venv/bin/paster", line 9, in <module> load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
  File "/opt/kallithea/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 104, in run invoke(command, command_name, options, args[1:])
  File "/opt/kallithea/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke exit_code = runner.run(args)
  File "/opt/kallithea/venv/local/lib/python2.7/site-packages/kallithea/lib/utils.py", line 753, in run return super(BasePasterCommand, self).run(args[1:])
  File "/opt/kallithea/venv/local/lib/python2.7/site-packages/paste/script/command.py", line 238, in run result = self.command()
  File "/opt/kallithea/venv/local/lib/python2.7/site-packages/kallithea/lib/paster_commands/make_index.py", line 84, in command .run(full_index=self.options.full_index)
  File "/opt/kallithea/venv/local/lib/python2.7/site-packages/kallithea/lib/indexers/daemon.py", line 451, in run self.update_indexes()
  File "/opt/kallithea/venv/local/lib/python2.7/site-packages/kallithea/lib/indexers/daemon.py", line 443, in update_indexes self.update_file_index()
  File "/opt/kallithea/venv/local/lib/python2.7/site-packages/kallithea/lib/indexers/daemon.py", line 390, in update_file_index i, iwc = self.add_doc(writer, path, repo, repo_name)
  File "/opt/kallithea/venv/local/lib/python2.7/site-packages/kallithea/lib/indexers/daemon.py", line 175, in add_doc node = self.get_node(repo, path, index_rev)
  File "/opt/kallithea/venv/local/lib/python2.7/site-packages/kallithea/lib/indexers/daemon.py", line 163, in get_node node = cs.get_node(node_path)
  File "/opt/kallithea/venv/local/lib/python2.7/site-packages/kallithea/lib/vcs/backends/hg/changeset.py", line 347, in get_node % (path, self.short_id))
kallithea.lib.vcs.exceptions.NodeDoesNotExistError: There is no file nor directory at the given path: 'Mod?le.xml' at revision b2a74d2081af</pre>

The actual filename referred to should be "Modèle.xml", not "Mod?le.xml" In the console it's not an actual interrogation point, it's a "♦".

This is using Kallithea 0.2.1 on an Ubuntu server (running through VirtualBox, but that shouldn't change anything?), using Mercurial repositories.



Comment by Samuel Delisle, on 2015-05-04 17:08

Comment by Mads Kiilerich, on 2015-05-04 17:31

Try setting HGENCODING="UTF-8" before running the command.

Comment by Samuel Delisle, on 2015-05-04 19:05

Is that an environment variable? I added HGENCODING="UTF-8" in /etc/environment, it does the same thing... Not sure if I did it right, but after rebooting, echo $HGENCODING gives me "UTF-8" as expected. I didn't find anything in Kallithea's settings related to the encoding.

It's probably relevant to note that the files in question were created on Windows. I notice Mercurial's documentation seem to point out compatibility issues for non-ascii characters: http://mercurial.selenic.com/wiki/EncodingStrategy

Comment by Mads Kiilerich, on 2015-05-04 19:09

Yes, it seems to be an inconsistent repo. It should still work with some caveats.

I guess the main issue here is that the error is considered fatal. It should just continue, possible after issuing a warning.

Comment by Samuel Delisle, on 2015-05-04 19:24

Exactly, I don't mind if it doesn't index that particular file properly, but it should just continue.

Comment by Thomas De Schampheleire, on 2015-08-06 19:53