Issue #135: Forking of git repositories does not work on windows
| Reported by: | |
| State: | closed |
| Created on: | 2015-05-22 12:37 |
| Updated on: | 2015-06-11 19:51 |
Description
Error description
Forking a git repository on a windows system hangs in an infinite loop while displaying
Repository "test-fork" is being created, you will be redirected when this process is finished.
The log reveals that the reason for this error is git failing to clone the repository due to incorrectly escaped repository paths.
2015-05-22 17:54:25.391 ERROR [kallithea.lib.vcs.backends.git.repository] Couldn't run git command (git -c core.quotepath=false clone -q --bare -- 'file:///c:\Kallithea2\Repo\test' 'c:\Kallithea2\Repo\test-fork'). Original error was:Subprocess exited due to an error: fatal: could not create leading directories of ''c:\Kallithea2\Repo\test-fork''
On windows, git does not accept single quotation marks.
Cause of error
The cause of the error seems to be in lib/vcs/backends/git/repository.py. In this module pipes.quote is used in multilple places to escape repository paths. However the output of pipes.quote is only suitable for unix shells but nor for the windows shell.
Possible solution
A possible solution would be to replace lines 20 - 25 in the file repository.py, which are
try: # Python <=2.7 from pipes import quote except ImportError: # Python 3.3+ from shlex import quote
by a platform independent version of quote. My suggestion would be:
import sys mswindows = (sys.platform == "win32") if mswindows: from subprocess import list2cmdline def quote(s): return list2cmdline([s]) else: try: # Python <=2.7 from pipes import quote except ImportError: # Python 3.3+ from shlex import quote
Kind regards, Matthias
System info:
Kallithea version:Z
0.2.1
Python version:
2.7.9
Platform:
Windows-8-6.2.9200
Git version:
1.9.5
Git path:
git
Upgrade info endpoint:
Note: please make sure this server can access this URL.
Python Packages
amqplib 1.0.2
anyjson 0.3.3
Babel 1.3
Beaker 1.6.4
celery 2.2.10
decorator 3.4.2
docutils 0.11
dulwich 0.9.9
FormEncode 1.2.6
Kallithea 0.2.1
kombu 1.5.1
Mako 1.0.0
Markdown 2.2.1
MarkupSafe 0.23
mercurial 3.3.3
mock 1.0.1
nose 1.3.6
Paste 2.0.1
PasteDeploy 1.5.2
PasteScript 1.7.5
pip 6.1.1
pycrypto 2.6.1
Pygments 2.0.2
Pylons 1.0
pyparsing 1.5.7
python-dateutil 1.5
pytz 2015.2
repoze.lru 0.6
Routes 1.13
setuptools 15.0
simplejson 2.5.2
six 1.9.0
SQLAlchemy 0.7.10
Tempita 0.5.2
URLObject 2.3.4
waitress 0.8.8
WebError 0.10.3
WebHelpers 1.3
WebOb 1.0.8
WebTest 1.4.3
Whoosh 2.5.7
Attachments
Comments
Comment on 2015-05-22 12:39
Comment on 2015-05-22 17:33
Comment by Mads Kiilerich, on 2015-05-27 22:22
The proposed patch has been applied to the stable branch.
As discussed on https://bitbucket.org/conservancy/kallithea/pull-request/141/fix-for-issue-135-added-platform/diff:
The need for this patch do however point at a root cause that requires a different and real solution - please see https://bitbucket.org/Unity-Technologies/kallithea/commits/38d5d9686ddf4101b63bc9aa3a77bbc77a665576 . I don't use git and can't give it much testing - please review and test it carefully.
I thus don't consider this issue fully solved.
Comment by Mads Kiilerich, on 2015-06-06 17:13
We really need your testing of the latest edition found on https://bitbucket.org/Unity-Technologies/kallithea/commits/ecdd9a8ec5552d12ed81409d702d8561b0823b0a
Comment on 2015-06-06 22:49
I'm on it. Will give you the results tomorrow.
Comment by Mads Kiilerich, on 2015-06-07 00:21
I use it in production now, but we don't use git much and don't use windows.
Comment on 2015-06-07 19:40
I reviewed the changes. Most changes are fine. Only the changes in kallithea.lib.middleware.pygrack are not OK. They break the ability to pull from the repository. This is not detected by the unit tests as pygrack.py is not covered. I had to revert the changes to get git remote operations working again (https://bitbucket.org/mkroll/kallithea/commits/1780906d945270101a5a15701dcb9e40a05b13d3) . I haven't found out yet what exactly was broken. I also found that the test kallithea.tests.vcs.test_git. GitRepositoryTest.test_git_cmd_injection is broken on Windows due to invalid characters in the tested path names (" is not allowed on Windows, fixed in https://bitbucket.org/mkroll/kallithea/commits/4558ca5e6a275b2784054061389c1ea66701a0a7).
I did some manual testing and ran the test suite under Ubuntu 14.04 and WIndows 8.1. The results of the manual tests are:
repository creation ... OK adding file on frontend ... OK cloning ... OK pushing ... OK pushing to different branch ... OK forking ... OK coparing revisions ... OK comparing forks ... OK creating pull request ... OK fetching from fork ... OK viewing changesets ... OK viewing history ... OK downloading ZIP file ... Failure (created zip file is locked by other process -> download works on second try because zip file is cached)
The unit tests under Ubuntu give 9 errors and 8 failure which are all unrelated to git. There is no regression compared to https://kallithea-scm.org/repos/kallithea/changeset/c44885d0e546303772771141418c5c7d91e0180c.
On Windows there are quite a lot of failling tests:
test_create_non_ascii (kallithea.tests.functional.test_admin_repos.TestAdminReposControllerGIT) ... Error test_delete_non_ascii (kallithea.tests.functional.test_admin_repos.TestAdminReposControllerGIT) ... Error test_create_non_ascii (kallithea.tests.functional.test_admin_repos.TestAdminReposControllerHG) ... Error test_delete_non_ascii (kallithea.tests.functional.test_admin_repos.TestAdminReposControllerHG) ... Error test_compare_forks_on_branch_extra_commits_origin_has_incomming_git (kallithea.tests.functional.test_compare.TestCompareController) ... Error test_org_repo_new_commits_after_forking_simple_diff_git (kallithea.tests.functional.test_compare.TestCompareController) ... Error test_org_repo_new_commits_after_forking_simple_diff_git (kallithea.tests.functional.test_compare.TestCompareController) ... Error test_org_repo_new_commits_after_forking_simple_diff_hg (kallithea.tests.functional.test_compare.TestCompareController) ... Error test_add_file_into_git_2 (kallithea.tests.functional.test_files.TestFilesController) ... Failure test_add_file_into_hg_2 (kallithea.tests.functional.test_files.TestFilesController) ... Failure test_archival (kallithea.tests.functional.test_files.TestFilesController) ... Error test_delete_file_view_commit_changes_git (kallithea.tests.functional.test_files.TestFilesController) ... Error test_delete_file_view_commit_changes_hg (kallithea.tests.functional.test_files.TestFilesController) ... Error test_delete_file_view_git (kallithea.tests.functional.test_files.TestFilesController) ... Error test_delete_file_view_not_on_branch_git (kallithea.tests.functional.test_files.TestFilesController) ... Error test_delete_file_view_not_on_branch_hg (kallithea.tests.functional.test_files.TestFilesController) ... Error test_edit_file_view_commit_changes_git (kallithea.tests.functional.test_files.TestFilesController) ... Error test_edit_file_view_commit_changes_hg (kallithea.tests.functional.test_files.TestFilesController) ... Error test_edit_file_view_git (kallithea.tests.functional.test_files.TestFilesController) ... Error test_edit_file_view_not_on_branch_git (kallithea.tests.functional.test_files.TestFilesController) ... Error test_edit_file_view_not_on_branch_hg (kallithea.tests.functional.test_files.TestFilesController) ... Error test_raw_file_ok (kallithea.tests.functional.test_files.TestFilesController) ... Failure test_diff_markup (kallithea.tests.models.test_diff_parsers.DiffLibTest) ... Failure test_clone_after_repo_was_locked_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure test_clone_after_repo_was_locked_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure test_clone_and_create_lock_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure test_clone_and_create_lock_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure test_clone_git_dir_as_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure test_clone_hg_repo_as_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure test_clone_hg_repo_by_admin (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure test_clone_non_existing_path_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure test_clone_non_existing_path_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure test_clone_wrong_credentials_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure test_clone_wrong_credentials_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure test_ip_restriction_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure test_ip_restriction_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure test_push_back_to_wrong_url_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Error test_push_back_to_wrong_url_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure test_push_invalidates_cache_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Error test_push_invalidates_cache_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure test_push_new_file_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Error test_push_new_file_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure test_push_on_locked_repo_by_other_user_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Error test_push_on_locked_repo_by_other_user_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure test_push_unlocks_repository_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure test_push_unlocks_repository_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure test_push_wrong_credentials_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Error test_push_wrong_credentials_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure test_archive_default_stream (kallithea.tests.vcs.test_archives.HgArchiveTest) ... Failure test_filenode_path (kallithea.tests.vcs.test_filenodes_unicode_path.GitFileNodeUnicodePathTest) ... Error test_filenode_path (kallithea.tests.vcs.test_filenodes_unicode_path.HgFileNodeUnicodePathTest) ... Error test_mimetype (kallithea.tests.vcs.test_nodes.NodeBasicTest) ... Failure test_get_repo_multialias (kallithea.tests.vcs.test_vcs.VCSTest) ... Error
Compared to https://kallithea-scm.org/repos/kallithea/changeset/c44885d0e546303772771141418c5c7d91e0180c there are the following regressions:
test_archival (kallithea.tests.functional.test_files.TestFilesController) ... Error test_diff_markup (kallithea.tests.models.test_diff_parsers.DiffLibTest) ... Failure test_archive_default_stream (kallithea.tests.vcs.test_archives.HgArchiveTest) ... Failure test_get_repo_multialias (kallithea.tests.vcs.test_vcs.VCSTest) ... Error
It seems to me that none of those is related to to the changes in the git code.
I would say that https://bitbucket.org/mkroll/kallithea/commits/4558ca5e6a275b2784054061389c1ea66701a0a7 shoulfd be fine. However, the reason why the changes to pygrack.py are not working needs to be investigated. Also the large number of failing tets on windows needs further attention.
Comment by Mads Kiilerich, on 2015-06-07 20:33
Thanks for the thorough testing!
Did the manual download of zip fail both on windows and ubuntu?
It is weird you see failures on ubuntu. I don't see any failures. That should be investigated separately.
The failures on Windows are more likely. We try to be cross platform but most of the contributors are on some kind of unix.
Comment by Mads Kiilerich, on 2015-06-07 21:35
I reproduced a git clone/pull failure and fixed it with https://bitbucket.org/Unity-Technologies/kallithea/commits/f99c4d130eed411e9b29b54387e666a1bba385d6 - does that work for you?
Comment on 2015-06-07 21:53
Yes, works both on Windows and Ubuntu.
Btw: Zip downloading works on Ubuntu. Apparently python does not lock the file for other processes o Linux but only on windows.
Comment by Mads Kiilerich, on 2015-06-07 23:24
Just to clarify: This is an unrelated problem, right? zip downloading was broken on windows before too?
Will this fix it:
--- a/kallithea/controllers/files.py +++ b/kallithea/controllers/files.py @@ -566,2 +566,3 @@ class FilesController(BaseRepoController fd, archive = tempfile.mkstemp() + os.close(fd) temp_stream = open(archive, 'wb') @@ -582,4 +583,2 @@ class FilesController(BaseRepoController stream.close() - if fd: # fd means we used temporary file - os.close(fd) if not archive_cache_enabled:
Comment on 2015-06-08 04:50
Yes, it is unrelated and was broken before. It just turned up while testing. The patch is working.
Comment by Mads Kiilerich, on 2015-06-09 22:24
I think this has been solved now and can be closed. Please follow up on other issues separately.
Comment on 2015-06-11 19:51