Kallithea issues archive

Issue #135: Forking of git repositories does not work on windows

Reported by:
State: closed
Created on: 2015-05-22 12:37
Updated on: 2015-06-11 19:51


Error description

Forking a git repository on a windows system hangs in an infinite loop while displaying

Repository "test-fork" is being created, you will be redirected when this process is finished. 

The log reveals that the reason for this error is git failing to clone the repository due to incorrectly escaped repository paths.

2015-05-22 17:54:25.391 ERROR [kallithea.lib.vcs.backends.git.repository] Couldn't run git command (git -c core.quotepath=false clone -q --bare -- 'file:///c:\Kallithea2\Repo\test' 'c:\Kallithea2\Repo\test-fork').
Original error was:Subprocess exited due to an error:
fatal: could not create leading directories of ''c:\Kallithea2\Repo\test-fork''

On windows, git does not accept single quotation marks.

Cause of error

The cause of the error seems to be in lib/vcs/backends/git/repository.py. In this module pipes.quote is used in multilple places to escape repository paths. However the output of pipes.quote is only suitable for unix shells but nor for the windows shell.

Possible solution

A possible solution would be to replace lines 20 - 25 in the file repository.py, which are

    # Python <=2.7
    from pipes import quote
except ImportError:
    # Python 3.3+
    from shlex import quote

by a platform independent version of quote. My suggestion would be:

import sys
mswindows = (sys.platform == "win32")

if mswindows:
    from subprocess import list2cmdline
    def quote(s):
        return list2cmdline([s])
        # Python <=2.7
        from pipes import quote
    except ImportError:
        # Python 3.3+
        from shlex import quote

Kind regards, Matthias

System info:

Kallithea version:Z
Python version:
Git version:
Git path:
Upgrade info endpoint:
    Note: please make sure this server can access this URL.

Python Packages
amqplib         1.0.2
anyjson         0.3.3
Babel           1.3
Beaker          1.6.4
celery          2.2.10
decorator       3.4.2
docutils        0.11
dulwich         0.9.9
FormEncode      1.2.6
Kallithea       0.2.1
kombu           1.5.1
Mako            1.0.0
Markdown        2.2.1
MarkupSafe      0.23
mercurial       3.3.3
mock            1.0.1
nose            1.3.6
Paste           2.0.1
PasteDeploy     1.5.2
PasteScript     1.7.5
pip             6.1.1
pycrypto        2.6.1
Pygments        2.0.2
Pylons          1.0
pyparsing       1.5.7
python-dateutil 1.5
pytz            2015.2
repoze.lru      0.6
Routes          1.13
setuptools      15.0
simplejson      2.5.2
six             1.9.0
SQLAlchemy      0.7.10
Tempita         0.5.2
URLObject       2.3.4
waitress        0.8.8
WebError        0.10.3
WebHelpers      1.3
WebOb           1.0.8
WebTest         1.4.3
Whoosh          2.5.7



Comment on 2015-05-22 12:39

Comment on 2015-05-22 17:33

Comment by Mads Kiilerich, on 2015-05-27 22:22

The proposed patch has been applied to the stable branch.

As discussed on https://bitbucket.org/conservancy/kallithea/pull-request/141/fix-for-issue-135-added-platform/diff:

The need for this patch do however point at a root cause that requires a different and real solution - please see https://bitbucket.org/Unity-Technologies/kallithea/commits/38d5d9686ddf4101b63bc9aa3a77bbc77a665576 . I don't use git and can't give it much testing - please review and test it carefully.

I thus don't consider this issue fully solved.

Comment by Mads Kiilerich, on 2015-06-06 17:13

Comment on 2015-06-06 22:49

I'm on it. Will give you the results tomorrow.

Comment by Mads Kiilerich, on 2015-06-07 00:21

I use it in production now, but we don't use git much and don't use windows.

Comment on 2015-06-07 19:40

I reviewed the changes. Most changes are fine. Only the changes in kallithea.lib.middleware.pygrack are not OK. They break the ability to pull from the repository. This is not detected by the unit tests as pygrack.py is not covered. I had to revert the changes to get git remote operations working again (https://bitbucket.org/mkroll/kallithea/commits/1780906d945270101a5a15701dcb9e40a05b13d3) . I haven't found out yet what exactly was broken. I also found that the test kallithea.tests.vcs.test_git. GitRepositoryTest.test_git_cmd_injection is broken on Windows due to invalid characters in the tested path names (" is not allowed on Windows, fixed in https://bitbucket.org/mkroll/kallithea/commits/4558ca5e6a275b2784054061389c1ea66701a0a7).

I did some manual testing and ran the test suite under Ubuntu 14.04 and WIndows 8.1. The results of the manual tests are:

repository creation ... OK
adding file on frontend ... OK
cloning ... OK
pushing ... OK
pushing to different branch ... OK
forking ... OK
coparing revisions ... OK
comparing forks ... OK
creating pull request ... OK
fetching from fork ... OK
viewing changesets ... OK
viewing history ... OK
downloading ZIP file ... Failure (created zip file is locked by other process -> download works on second try because zip file is cached)

The unit tests under Ubuntu give 9 errors and 8 failure which are all unrelated to git. There is no regression compared to https://kallithea-scm.org/repos/kallithea/changeset/c44885d0e546303772771141418c5c7d91e0180c.

On Windows there are quite a lot of failling tests:

test_create_non_ascii (kallithea.tests.functional.test_admin_repos.TestAdminReposControllerGIT) ... Error
test_delete_non_ascii (kallithea.tests.functional.test_admin_repos.TestAdminReposControllerGIT) ... Error
test_create_non_ascii (kallithea.tests.functional.test_admin_repos.TestAdminReposControllerHG) ... Error
test_delete_non_ascii (kallithea.tests.functional.test_admin_repos.TestAdminReposControllerHG) ... Error
test_compare_forks_on_branch_extra_commits_origin_has_incomming_git (kallithea.tests.functional.test_compare.TestCompareController) ... Error
test_org_repo_new_commits_after_forking_simple_diff_git (kallithea.tests.functional.test_compare.TestCompareController) ... Error
test_org_repo_new_commits_after_forking_simple_diff_git (kallithea.tests.functional.test_compare.TestCompareController) ... Error
test_org_repo_new_commits_after_forking_simple_diff_hg (kallithea.tests.functional.test_compare.TestCompareController) ... Error
test_add_file_into_git_2 (kallithea.tests.functional.test_files.TestFilesController) ... Failure
test_add_file_into_hg_2 (kallithea.tests.functional.test_files.TestFilesController) ... Failure
test_archival (kallithea.tests.functional.test_files.TestFilesController) ... Error
test_delete_file_view_commit_changes_git (kallithea.tests.functional.test_files.TestFilesController) ... Error
test_delete_file_view_commit_changes_hg (kallithea.tests.functional.test_files.TestFilesController) ... Error
test_delete_file_view_git (kallithea.tests.functional.test_files.TestFilesController) ... Error
test_delete_file_view_not_on_branch_git (kallithea.tests.functional.test_files.TestFilesController) ... Error
test_delete_file_view_not_on_branch_hg (kallithea.tests.functional.test_files.TestFilesController) ... Error
test_edit_file_view_commit_changes_git (kallithea.tests.functional.test_files.TestFilesController) ... Error
test_edit_file_view_commit_changes_hg (kallithea.tests.functional.test_files.TestFilesController) ... Error
test_edit_file_view_git (kallithea.tests.functional.test_files.TestFilesController) ... Error
test_edit_file_view_not_on_branch_git (kallithea.tests.functional.test_files.TestFilesController) ... Error
test_edit_file_view_not_on_branch_hg (kallithea.tests.functional.test_files.TestFilesController) ... Error
test_raw_file_ok (kallithea.tests.functional.test_files.TestFilesController) ... Failure
test_diff_markup (kallithea.tests.models.test_diff_parsers.DiffLibTest) ... Failure
test_clone_after_repo_was_locked_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
test_clone_after_repo_was_locked_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
test_clone_and_create_lock_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
test_clone_and_create_lock_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
test_clone_git_dir_as_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
test_clone_hg_repo_as_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
test_clone_hg_repo_by_admin (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
test_clone_non_existing_path_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
test_clone_non_existing_path_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
test_clone_wrong_credentials_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
test_clone_wrong_credentials_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
test_ip_restriction_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
test_ip_restriction_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
test_push_back_to_wrong_url_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Error
test_push_back_to_wrong_url_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
test_push_invalidates_cache_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Error
test_push_invalidates_cache_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
test_push_new_file_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Error
test_push_new_file_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
test_push_on_locked_repo_by_other_user_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Error
test_push_on_locked_repo_by_other_user_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
test_push_unlocks_repository_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
test_push_unlocks_repository_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
test_push_wrong_credentials_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Error
test_push_wrong_credentials_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
test_archive_default_stream (kallithea.tests.vcs.test_archives.HgArchiveTest) ... Failure
test_filenode_path (kallithea.tests.vcs.test_filenodes_unicode_path.GitFileNodeUnicodePathTest) ... Error
test_filenode_path (kallithea.tests.vcs.test_filenodes_unicode_path.HgFileNodeUnicodePathTest) ... Error
test_mimetype (kallithea.tests.vcs.test_nodes.NodeBasicTest) ... Failure
test_get_repo_multialias (kallithea.tests.vcs.test_vcs.VCSTest) ... Error

Compared to https://kallithea-scm.org/repos/kallithea/changeset/c44885d0e546303772771141418c5c7d91e0180c there are the following regressions:

test_archival (kallithea.tests.functional.test_files.TestFilesController) ... Error
test_diff_markup (kallithea.tests.models.test_diff_parsers.DiffLibTest) ... Failure
test_archive_default_stream (kallithea.tests.vcs.test_archives.HgArchiveTest) ... Failure
test_get_repo_multialias (kallithea.tests.vcs.test_vcs.VCSTest) ... Error

It seems to me that none of those is related to to the changes in the git code.

I would say that https://bitbucket.org/mkroll/kallithea/commits/4558ca5e6a275b2784054061389c1ea66701a0a7 shoulfd be fine. However, the reason why the changes to pygrack.py are not working needs to be investigated. Also the large number of failing tets on windows needs further attention.

Comment by Mads Kiilerich, on 2015-06-07 20:33

Thanks for the thorough testing!

Did the manual download of zip fail both on windows and ubuntu?

It is weird you see failures on ubuntu. I don't see any failures. That should be investigated separately.

The failures on Windows are more likely. We try to be cross platform but most of the contributors are on some kind of unix.

Comment by Mads Kiilerich, on 2015-06-07 21:35

I reproduced a git clone/pull failure and fixed it with https://bitbucket.org/Unity-Technologies/kallithea/commits/f99c4d130eed411e9b29b54387e666a1bba385d6 - does that work for you?

Comment on 2015-06-07 21:53

Yes, works both on Windows and Ubuntu.

Btw: Zip downloading works on Ubuntu. Apparently python does not lock the file for other processes o Linux but only on windows.

Comment by Mads Kiilerich, on 2015-06-07 23:24

Just to clarify: This is an unrelated problem, right? zip downloading was broken on windows before too?

Will this fix it:

--- a/kallithea/controllers/files.py
+++ b/kallithea/controllers/files.py
@@ -566,2 +566,3 @@ class FilesController(BaseRepoController
             fd, archive = tempfile.mkstemp()
+            os.close(fd)
             temp_stream = open(archive, 'wb')
@@ -582,4 +583,2 @@ class FilesController(BaseRepoController
-                    if fd:  # fd means we used temporary file
-                        os.close(fd)
                     if not archive_cache_enabled:

Comment on 2015-06-08 04:50

Yes, it is unrelated and was broken before. It just turned up while testing. The patch is working.

Comment by Mads Kiilerich, on 2015-06-09 22:24

I think this has been solved now and can be closed. Please follow up on other issues separately.

Comment on 2015-06-11 19:51