Kallithea issues archive

Issue #33: git clone fails with HTTP-Fehler 417 Expectation failed

Reported by: Sven R. Kunze
State: resolved
Created on: 2014-09-09 11:07
Updated on: 2014-09-24 17:52

Description

This error occurs sometimes (but not always) when doing a pip install -e git+https://....

Attachments

Comments

Comment by Mads Kiilerich, on 2014-09-09 11:26

What is "...." and how do that relate to Kallithea?

Comment by Sven R. Kunze, on 2014-09-15 10:33

https:// is a URL refering to our kallithea instance.

We actually updated from rhodecode to kallithea because of this error. Unfortunately, it persists.

Comment by Sven R. Kunze, on 2014-09-15 10:59

Might be related to https://rhodecode.com/help/discussions/problems/7628-git-shallow-clone-not-working

Btw. with rhodecode, we had other similar occasional errors with sometimes random-looking error-codes like: 0, None, 403. None of them makes acutally sense as when trying 5 minutes later, it works gracefully.

Comment by domruf, on 2014-09-15 11:29

Do you have any log entries that shed some light on this problem?

The problem only happens on git repositories? The problem never happens when you clone the repository regularly?

Comment by Sven R. Kunze, on 2014-09-15 13:43

I will see if I can get my hands on some logfiles. I remember us looking through them when using rhodecode, but the error was not comprehensible.

We indedd serve git repositories only. It happens occasionally and there is no apparent pattern. After talking to my team members, they also incurred it when doing "git clone https://" instead of "pip install -e git+https://....".

Comment by Mads Kiilerich, on 2014-09-15 19:23

Comment by Sven R. Kunze, on 2014-09-16 08:10

Btw. pip install -e does not only git clone but also git fetch/git pull. We run into that situation as well as when using pip install -e to create and update of a local git repository.

Comment by Mads Kiilerich, on 2014-09-16 22:16

For the --depth clone issue, please try

--- a/kallithea/lib/vcs/subprocessio.py
+++ b/kallithea/lib/vcs/subprocessio.py
@@ -375,14 +375,20 @@ class SubprocessIOChunker(object):
             except Exception:
                 pass
             bg_out.stop()
+            out = ''.join(bg_out)
             bg_err.stop()
-            err = '%s' % ''.join(bg_err)
-            if err:
+            err = ''.join(bg_err)
+            if (err.strip() == 'fatal: The remote end hung up unexpectedly' and
+                out.startswith('0034shallow ')):
+                # hack inspired by https://github.com/schacon/grack/pull/7
+                bg_out = iter([out])
+                _p = None
+            elif err:
                 raise EnvironmentError(
                     "Subprocess exited due to an error:\n" + err)
-            raise EnvironmentError(
-                "Subprocess exited with non 0 ret code:%s" % _returncode)
-
+            else:
+                raise EnvironmentError(
+                    "Subprocess exited with non 0 ret code:%s" % _returncode)
         self.process = _p
         self.output = bg_out
         self.error = bg_err
@@ -392,7 +398,7 @@ class SubprocessIOChunker(object):
         return self

     def next(self):
-        if self.process.poll():
+        if self.process and self.process.poll():
             err = '%s' % ''.join(self.error)
             raise EnvironmentError("Subprocess exited due to an error:\n" + err)
         return self.output.next()

Comment by Sven R. Kunze, on 2014-09-17 08:11

Is there a testcase to provoke the 417? We would not want to jeopardize our production system.

Comment by Mads Kiilerich, on 2014-09-17 08:18

I don't know. You reported the problem ;-)

You also suggested it could be related to --depth clones ... which this patch seems to fix.

Comment by Sven R. Kunze, on 2014-09-17 09:51

Well, I meant something like: py.test -k test_417. I also think that would be a good addition in order to indicate such problems in the future.

If you are confident that this patch works, we look forward to the next version of kallithea as it will fix other problems as well.

Comment by Mads Kiilerich, on 2014-09-17 09:55

This is a collaboration project. We would appreciate contribution of test coverage for this.

I'm only confident your problem has been solved when you tell me.

Comment by Sven R. Kunze, on 2014-09-17 10:38

I understand that. However, we have no idea how to provoke that error. It occurs completely randomly. This might be related as well: https://github.com/bower/bower/issues/744

Right now, we had this:

error: RPC failed; result=22, HTTP code = 500
fatal: The remote end hung up unexpectedly

apache-log shows:

[Wed Sep 17 11:51:47 2014] [error] [client IP] Premature end of script headers: wsgi.py
[Wed Sep 17 11:51:47 2014] [error] [client IP] no acceptable variant: /usr/share/apache2/error/HTTP_INTERNAL_SERVER_ERROR.html.var

Are there some other logs I can sift through?

I will give your patch a try.

Comment by Mads Kiilerich, on 2014-09-17 11:26

I would suggest running it in the foreground with paster and a single client hammering it. That will usually give better debug output.

Comment by Sven R. Kunze, on 2014-09-22 12:36

Maybe, this is related to our #32 .

I have a hammer.py script running for several hours now after applying that fix and no error so far. I will have this script run for several days in order to make sure it works reliably.

Do you think the patch above is still necessary when the patch of #32 works as well?

Comment by Mads Kiilerich, on 2014-09-22 13:55

I could reproduce something that looked like the failure rhodecode issue you linked to when trying to clone with git --depth ... and the patch above seemed to fix it ... and it is unrelated to the fix for #32.

I do however not know anything about git (or care that much about it) so I would appreciate if someone with git knowledge could review and/or give it some serious testing.

It is possible that your initial problem was a duplicate of #32 but you gave a hint towards another and real problem.

Comment by Sven R. Kunze, on 2014-09-23 08:16

I am sorry, but the hammer script could not reproduce the error.

Comment by Sven R. Kunze, on 2014-09-24 11:11

Another random error:

error: RPC failed; result=22, HTTP code = 403
fatal: The remote end hung up unexpectedly

Apache-Log still reveals nothing.

Comment by Mads Kiilerich, on 2014-09-24 17:52

Fix pushed.