Issue #242: [wait-for-feedback] Random http status 403

Reported by:	Thomas Güttler
State:	closed
Created on:	2016-08-17 09:29
Updated on:	2018-06-09 20:36

Description

We get random http status 403 messages.

It happens roughly in 1 of 50 git clone commands.

Here is the output of our client (pip (Python install tool) started from jenkins):

Obtaining foowork from git+https://source.example.com:40443/repos/foowork@release/2016.371#egg=foowork (from -r /tmp/tmp1jaIlG/requirements.txt (line 12))
Cloning https://source.example.com:40443/repos/foowork (to release/2016.371) to ./src/foowork
Complete output from command git clone -q https://source.example.com:40443/repos/foowork /home/foowork_eins_di2468/src/foowork:

---------------------------------------- error: RPC failed; result=22, HTTP code = 403
fatal: The remote end hung up unexpectedly
Command "git clone -q https://source.example.com:40443/repos/foowork /home/foowork_eins_di2468/src/foowork" failed with error code 128 in None

Has anybody a hint why this fails sometimes?

How can I debug this?

Version: Kallithea==0.3.2

Attachments

Comments

Comment by Thomas Güttler, on 2016-08-17 09:29

Comment by Andrej Shadura, on 2016-08-17 09:33

Try looking into your server-side logs. You may or may not need to add debug = true and tune log levels if your logs aren’t verbose enough.

Comment by Thomas Güttler, on 2016-08-17 10:33

Comment by Thomas Güttler, on 2016-08-18 09:07

Tonight one CI failed again. Here are is the logging output:

I renamed our internal repos to "repowhichfailed" and to "otherrepo".

There are several request coming in in the same second.

2016-08-17 23:47:58.589 DEBUG [kallithea.lib.middleware.simplegit] Repository path is /home/kallithea/repos/otherrepo
2016-08-17 23:47:58.589 DEBUG [kallithea.lib.middleware.simplegit] Checking locking on repository
2016-08-17 23:47:58.592 DEBUG [kallithea.lib.middleware.simplegit] pathinfo: /repowhichfailed/git-upload-pack detected as Git True
2016-08-17 23:47:58.599 DEBUG [kallithea.lib.middleware.simplegit] Extracted repo name is repowhichfailed
2016-08-17 23:47:58.608 DEBUG [kallithea.lib.middleware.simplegit] Anonymous access is disabled, running authentication
2016-08-17 23:47:58.609 DEBUG [kallithea.lib.middleware.simplegit] Not enough credentials to access this repository as anonymous user
2016-08-17 23:47:58.609 DEBUG [kallithea.lib.middleware.simplegit] Running PRE-AUTH for container based authentication
2016-08-17 23:47:58.840 DEBUG [kallithea.lib.auth] checking VCS protocol permissions set([u'repository.read']) for user:readonly repository:repowhichfailed
2016-08-17 23:47:58.841 DEBUG [kallithea.lib.auth_modules] Authentication against [u'kallithea.lib.auth_modules.auth_internal', u'kallithea.lib.auth_modules.auth_ldap'] plugins
2016-08-17 23:47:58.841 DEBUG [kallithea.lib.auth] Permission to repo: repowhichfailed denied for user: readonly @ PermissionMiddleware

The strange thing: "Permission to repo: repowhichfailed denied for user: readonly @ PermissionMiddleware"

Some seconds before a different CI run was able to access the repo:

[Wed Aug 17 23:46:26 2016] [error] 2016-08-17 23:46:26.131 DEBUG [kallithea.lib.auth] checking VCS protocol permissions set([u'repository.read']) for user:readonly repository:repowhichfailed
[Wed Aug 17 23:46:26 2016] [error] 2016-08-17 23:46:26.132 DEBUG [kallithea.lib.auth] Permission to repo: repowhichfailed granted for user: readonly @ PermissionMiddleware

Do you have a clue ?

I guess it is related to multiprocessing, since there are several CI runs accessing the repos at once.

Comment by Mads Kiilerich, on 2016-08-18 12:21

It could perhaps also be a network / ldap glitch.

Comment by Thomas De Schampheleire, on 2018-05-18 20:22

Did you finally find this issue?

Comment by Thomas De Schampheleire, on 2018-06-09 20:36

Closing due to no further feedback from submitter.

Kallithea issues archive