Kallithea issues archive

Issue #316: TypeError("Unicode-objects must be encoded before hashing")

Reported by: Wise Kaa
State: closed
Created on: 2018-05-07 15:41
Updated on: 2020-06-18 20:23

Description

What is it ?

2018-05-07 18:34:08.877 INFO  [kallithea.lib.base] IP: 46.18.200.242 User: <AuthUser('id:1[default] auth:True')> accessed /_admin/register
Error - <type 'exceptions.TypeError'>: Unicode-objects must be encoded before hashing
URL: http://git.kngk.org/_admin/register
File '/usr/local/lib/python2.7/site-packages/weberror/errormiddleware.py', line 171 in __call__
  app_iter = self.application(environ, sr_checker)
File '/usr/local/lib/python2.7/site-packages/kallithea/lib/middleware/sessionmiddleware.py', line 62 in __call__
  return self.wrap_app(environ, session_start_response)
File '/usr/local/lib/python2.7/site-packages/routes/middleware.py', line 131 in __call__
  response = self.app(environ, start_response)
File '/usr/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 103 in __call__
  response = self.dispatch(controller, environ, start_response)
File '/usr/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 313 in dispatch
  return controller(environ, start_response)
File '/usr/local/lib/python2.7/site-packages/kallithea/lib/base.py', line 446 in __call__
  return WSGIController.__call__(self, environ, start_response)
File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 214 in __call__
  response = self._dispatch_call()
File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 164 in _dispatch_call
  response = self._inspect_call(func)
File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
  result = self._perform_call(func, args)
File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 57 in _perform_call
  return func(**args)
File '<decorator-gen-2>', line 2 in register
File '/usr/local/lib/python2.7/site-packages/kallithea/lib/auth.py', line 857 in __wrapper
  return func(*fargs, **fkwargs)
File '/usr/local/lib/python2.7/site-packages/kallithea/controllers/login.py', line 151 in register
  UserModel().create_registration(form_result)
File '/usr/local/lib/python2.7/site-packages/kallithea/model/user.py', line 186 in create_registration
  new_user = self.create(form_data)
File '/usr/local/lib/python2.7/site-packages/kallithea/model/user.py', line 92 in create
  v = get_crypt_password(v)
File '/usr/local/lib/python2.7/site-packages/kallithea/lib/auth.py', line 136 in get_crypt_password
  return KallitheaCrypto.hash_string(password)
File '/usr/local/lib/python2.7/site-packages/kallithea/lib/auth.py', line 110 in hash_string
  return bcrypt.hashpw(str_, bcrypt.gensalt(10))
File '/usr/local/lib/python2.7/site-packages/bcrypt/__init__.py', line 62 in hashpw
  raise TypeError("Unicode-objects must be encoded before hashing")
TypeError: Unicode-objects must be encoded before hashing

Kallithea 0.3.4

Attachments

Comments

Comment by Wise Kaa, on 2018-05-07 15:42

Comment by Wise Kaa, on 2018-05-07 15:43

Comment by Wise Kaa, on 2018-05-07 15:45

Comment by Mads Kiilerich, on 2018-05-07 15:57

That sounds like https://kallithea-scm.org/repos/kallithea/changeset/fefd7279e798e279ed71acf3d538319412d2fb44?at=stable in 0.3.4 doesn't work.

Can you reproduce the problem? Which unicode username/password?

Comment by Wise Kaa, on 2018-05-07 16:21

Yes. It is on register page. I don't Unicode strings, just ascii string. Username: stas Password: QazWsx321

Comment by Thomas De Schampheleire, on 2018-05-07 19:35

Can you try following patch?

diff --git a/kallithea/lib/auth.py b/kallithea/lib/auth.py
--- a/kallithea/lib/auth.py
+++ b/kallithea/lib/auth.py
@@ -103,6 +103,12 @@ class KallitheaCrypto(object):

         :param password: password to hash
         """
+        try:
+            password = str(password)
+        except UnicodeEncodeError:
+            log.warning('rejecting non-ascii password')
+            return False
+        log.info(">>>>>>>>>>>>>>>>>> hash_string(%s)", repr(str_))
         if is_windows:
             return hashlib.sha256(str_).hexdigest()
         elif is_unix:
@@ -127,6 +133,7 @@ class KallitheaCrypto(object):
         except UnicodeEncodeError:
             log.warning('rejecting non-ascii password')
             return False
+        log.info(">>>>>>>>>>>>>>>>>> hash_check(%s)", repr(password))
         if is_windows:
             return hashlib.sha256(password).hexdigest() == hashed
         elif is_unix:
@@ -138,6 +145,7 @@ class KallitheaCrypto(object):


 def get_crypt_password(password):
+    log.info(">>>>>>>>>>>>>>>>>> get_crypt_password(%s)", repr(password))
     return KallitheaCrypto.hash_string(password)

It seems that the method used for calculating the password hash on registration is different than the one fixed in the commit Mads referenced, used to verify a password on login.

A simple test with the credentials you mentioned: 'stas' / 'QazWsx321' does not cause this problem on my side, though.

Please send the output of the extra '>>>>>>>>' log statements from your test. We should be able to see exactly what Kallithea sees as password.

What version of bcrypt is in use? (Use 'pip freeze | grep bcrypt'). On my side, I have py-bcrypt 0.3, but I tested with 0.4 as well.

How did you install Kallithea?

Comment by Wise Kaa, on 2018-05-08 05:56

Bcrypt: bcrypt==3.1.4 py-bcrypt==0.4

My machine is FreeBSD 10.4-STABLE.

2018-05-08 08:54:49.112 INFO  [kallithea.lib.base] IP: 46.18.200.242 User: <AuthUser('id:1[default] auth:True')> accessed /_admin/register
2018-05-08 08:54:49.205 INFO  [kallithea.lib.auth] >>>>>>>>>>>>>>>>>> get_crypt_password(u'QweAsd321')
Error - <type 'exceptions.UnboundLocalError'>: local variable 'password' referenced before assignment
URL: http://git.kngk.org/_admin/register
File '/usr/local/lib/python2.7/site-packages/weberror/errormiddleware.py', line 171 in __call__
  app_iter = self.application(environ, sr_checker)
File '/usr/local/lib/python2.7/site-packages/kallithea/lib/middleware/sessionmiddleware.py', line 62 in __call__
  return self.wrap_app(environ, session_start_response)
File '/usr/local/lib/python2.7/site-packages/routes/middleware.py', line 131 in __call__
  response = self.app(environ, start_response)
File '/usr/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 103 in __call__
  response = self.dispatch(controller, environ, start_response)
File '/usr/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 313 in dispatch
  return controller(environ, start_response)
File '/usr/local/lib/python2.7/site-packages/kallithea/lib/base.py', line 446 in __call__
  return WSGIController.__call__(self, environ, start_response)
File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 214 in __call__
  response = self._dispatch_call()
File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 164 in _dispatch_call
  response = self._inspect_call(func)
File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
  result = self._perform_call(func, args)
File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 57 in _perform_call
  return func(**args)
File '<decorator-gen-2>', line 2 in register
File '/usr/local/lib/python2.7/site-packages/kallithea/lib/auth.py', line 864 in __wrapper
  return func(*fargs, **fkwargs)
File '/usr/local/lib/python2.7/site-packages/kallithea/controllers/login.py', line 151 in register
  UserModel().create_registration(form_result)
File '/usr/local/lib/python2.7/site-packages/kallithea/model/user.py', line 186 in create_registration
  new_user = self.create(form_data)
File '/usr/local/lib/python2.7/site-packages/kallithea/model/user.py', line 92 in create
  v = get_crypt_password(v)
File '/usr/local/lib/python2.7/site-packages/kallithea/lib/auth.py', line 143 in get_crypt_password
  return KallitheaCrypto.hash_string(password)
File '/usr/local/lib/python2.7/site-packages/kallithea/lib/auth.py', line 107 in hash_string
  password = str(password)
UnboundLocalError: local variable 'password' referenced before assignment

Comment by Wise Kaa, on 2018-05-08 06:04

I changed password to str_

2018-05-08 09:03:16.038 INFO  [kallithea.lib.base] IP: 46.18.200.242 User: <AuthUser('id:1[default] auth:True')> accessed /_admin/register
2018-05-08 09:03:16.141 INFO  [kallithea.lib.auth] >>>>>>>>>>>>>>>>>> get_crypt_password(u'QweAsd321')
2018-05-08 09:03:16.141 INFO  [kallithea.lib.auth] >>>>>>>>>>>>>>>>>> hash_string(u'QweAsd321')
Error - <type 'exceptions.TypeError'>: Unicode-objects must be encoded before hashing
URL: http://git.kngk.org/_admin/register
File '/usr/local/lib/python2.7/site-packages/weberror/errormiddleware.py', line 171 in __call__
  app_iter = self.application(environ, sr_checker)
File '/usr/local/lib/python2.7/site-packages/kallithea/lib/middleware/sessionmiddleware.py', line 62 in __call__
  return self.wrap_app(environ, session_start_response)
File '/usr/local/lib/python2.7/site-packages/routes/middleware.py', line 131 in __call__
  response = self.app(environ, start_response)
File '/usr/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 103 in __call__
  response = self.dispatch(controller, environ, start_response)
File '/usr/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 313 in dispatch
  return controller(environ, start_response)
File '/usr/local/lib/python2.7/site-packages/kallithea/lib/base.py', line 446 in __call__
  return WSGIController.__call__(self, environ, start_response)
File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 214 in __call__
  response = self._dispatch_call()
File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 164 in _dispatch_call
  response = self._inspect_call(func)
File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
  result = self._perform_call(func, args)
File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 57 in _perform_call
  return func(**args)
File '<decorator-gen-2>', line 2 in register
File '/usr/local/lib/python2.7/site-packages/kallithea/lib/auth.py', line 864 in __wrapper
  return func(*fargs, **fkwargs)
File '/usr/local/lib/python2.7/site-packages/kallithea/controllers/login.py', line 151 in register
  UserModel().create_registration(form_result)
File '/usr/local/lib/python2.7/site-packages/kallithea/model/user.py', line 186 in create_registration
  new_user = self.create(form_data)
File '/usr/local/lib/python2.7/site-packages/kallithea/model/user.py', line 92 in create
  v = get_crypt_password(v)
File '/usr/local/lib/python2.7/site-packages/kallithea/lib/auth.py', line 143 in get_crypt_password
  return KallitheaCrypto.hash_string(password)
File '/usr/local/lib/python2.7/site-packages/kallithea/lib/auth.py', line 116 in hash_string
  return bcrypt.hashpw(str_, bcrypt.gensalt(10))
File '/usr/local/lib/python2.7/site-packages/bcrypt/__init__.py', line 62 in hashpw
  raise TypeError("Unicode-objects must be encoded before hashing")
TypeError: Unicode-objects must be encoded before hashing

Comment by Wise Kaa, on 2018-05-08 08:13

I was correct method:

 diff auth.py auth.py.orig
106,111d105
<         try:
<             password = str(str_)
<         except UnicodeEncodeError:
<             log.warning('rejecting non-ascii password')
<             return False
<         log.info(">>>>>>>>>>>>>>>>>> hash_string(%s)", repr(str_))
113c107
<             return hashlib.sha256(password).hexdigest()
---
>             return hashlib.sha256(str_).hexdigest()
116c110
<             return bcrypt.hashpw(password, bcrypt.gensalt(10))
---
>             return bcrypt.hashpw(str_, bcrypt.gensalt(10))
136d129
<         log.info(">>>>>>>>>>>>>>>>>> hash_check(%s)", repr(password))
148d140
<     log.info(">>>>>>>>>>>>>>>>>> get_crypt_password(%s)", repr(password))

Now it is OK.

Comment by Wise Kaa, on 2018-05-08 08:42 

 diff auth.py auth.py.orig
106,111d105
<         try:
<             password = str(str_)
<         except UnicodeEncodeError:
<             log.warning('rejecting non-ascii password')
<             return False
<         log.info(">>>>>>>>>>>>>>>>>> hash_string(%s)", repr(str_))
113c107
<             return hashlib.sha256(password).hexdigest()
---
>             return hashlib.sha256(str_).hexdigest()
116c110
<             return bcrypt.hashpw(password, bcrypt.gensalt(10))
---
>             return bcrypt.hashpw(str_, bcrypt.gensalt(10))
136d129
<         log.info(">>>>>>>>>>>>>>>>>> hash_check(%s)", repr(password))
148d140
<     log.info(">>>>>>>>>>>>>>>>>> get_crypt_password(%s)", repr(password))

Comment by Wise Kaa, on 2018-05-08 09:27

Corrected:

diff auth.py auth.py.orig
106,111d1054-STABLE (WEB) #3 r328435: Fri Jan 26 16:30:17 MSK 2018
<         try:]# mc
<             password = str(str_)
<         except UnicodeEncodeError:
<             log.warning('rejecting non-ascii password')
<             return False
<         log.info(">>>>>>>>>>>>>>>>>> hash_string(%s)", repr(str_))
113c107
<             return hashlib.sha256(password).hexdigest()
---
>             return hashlib.sha256(str_).hexdigest()
116c110
<             return bcrypt.hashpw(password, bcrypt.gensalt(10))
---
>             return bcrypt.hashpw(str_, bcrypt.gensalt(10))
136,141d129
<         try:
<             hashed = str(hashed)
<         except UnicodeEncodeError:
<             log.warning('rejecting non-ascii hashed password')
<             return False
<         log.info(">>>>>>>>>>>>>>>>>> hash_check(%s)", repr(password))
153d140
<     log.info(">>>>>>>>>>>>>>>>>> get_crypt_password(%s)", repr(password))

Comment by Thomas De Schampheleire, on 2018-05-08 10:29

In unified diff, the total change you did seems to be:

diff --git a/kallithea/lib/auth.py b/kallithea/lib/auth.py
--- a/kallithea/lib/auth.py
+++ b/kallithea/lib/auth.py
@@ -103,11 +103,17 @@ class KallitheaCrypto(object):

         :param password: password to hash
         """
+        try:
+            password = str(str_)
+        except UnicodeEncodeError:
+            log.warning('rejecting non-ascii password')
+            return False
+        log.info(">>>>>>>>>>>>>>>>>> hash_string(%s)", repr(str_))
         if is_windows:
-            return hashlib.sha256(str_).hexdigest()
+            return hashlib.sha256(password).hexdigest()
         elif is_unix:
             import bcrypt
-            return bcrypt.hashpw(str_, bcrypt.gensalt(10))
+            return bcrypt.hashpw(password, bcrypt.gensalt(10))
         else:
             raise Exception('Unknown or unsupported platform %s' \
                             % __platform__)
@@ -127,6 +133,12 @@ class KallitheaCrypto(object):
         except UnicodeEncodeError:
             log.warning('rejecting non-ascii password')
             return False
+        try:
+            hashed = str(hashed)
+        except UnicodeEncodeError:
+            log.warning('rejecting non-ascii hashed password')
+            return False
+        log.info(">>>>>>>>>>>>>>>>>> hash_check(%s)", repr(password))
         if is_windows:
             return hashlib.sha256(password).hexdigest() == hashed
         elif is_unix:
@@ -138,6 +150,7 @@ class KallitheaCrypto(object):


 def get_crypt_password(password):
+    log.info(">>>>>>>>>>>>>>>>>> get_crypt_password(%s)", repr(password))
     return KallitheaCrypto.hash_string(password)

Where the real change is to also verify the second parameter of hash_check.

It puzzles me a bit of what was really failing in the original situation. For registration, only hash_check seems to be used, at least that is in your traceback, and you showed the same problem with my first (corrected) patch.

Without changes (back to the original situation), do you see the same problem with any username password combination, e.g. 'foobar':'barfoo' ?

Comment by Wise Kaa, on 2018-05-08 11:52

Yes, i will foobar:barfoo, but this is not work. Now it is OK with your/my patches.

Comment by Mads Kiilerich, on 2018-05-08 13:59

Why was the issue closed? It hasn't been fixed upstream?

Also, can you confirm that you are running in an environment where LANG=C?

Comment by Thomas De Schampheleire, on 2018-05-13 19:22

Reopening issue until it has been fixed in the upstream code.

Comment by Thomas De Schampheleire, on 2018-05-13 19:24

@wisekaa03 Could you check the locale settings in the terminal in which you start Kallithea? E.g. by running:

env | grep LANG
env | grep LC_

Comment by Richard H., on 2018-12-16 01:29

Hi there!

I just stumbled upon this problem. And I seem to have found the culprit - by incident.

Dec 16 01:43:06 pkg: py27-py-bcrypt-0.3 deinstalled

Dec 16 01:43:16 pkg: py27-bcrypt-3.1.4_1 installed

I just updated pkg and it changed my versions! So it seems there is a mismatch between ports version and pkg version of py27-bcrypt. I restored the package from ports and all is well again. When I installed kallithea, it chose this version after all.

Using FreeBSD 10.4-RELEASE too by the way.

Comment by Mads Kiilerich, on 2018-12-16 23:24

Oh. Right. These two projects both provide a 'bcrypt' module. But they should also conflict. So weird that despite setup.py requiring bcrypt >= 3.1.0, at runtime it ended up using py-bcrypt 0.3 .

Since the "right" bcrypt has __version__ and the other doesn't, we could perhaps add an

assert getattr(bcrypt, "__version__", '').split(".", 2)[:2] == ["3", "1"]

or

assert getattr(bcrypt, "__version__", '').startswith("3.1.")

Comment by Richard H., on 2018-12-20 11:00

It seems the error just happens, when the update happens without restarting the service. When trying to restart it, the dependency check fails. Well, FreeBSD is quite behind as well, just wanted to share my insight so if anybody else stumbles, the information will be there :)

Comment by Thomas De Schampheleire, on 2018-12-25 18:57

@chainria So it seems you are not using a virtualenv, but rather are relying on system packages, is that correct?

@kiilerix An assert like you proposed could be done. But we'd need to reference it from the setup.py dependency list to keep them in sync. More generally though, this problem could happen with every package: someone could update a package to a version outside of the supported range.

Comment by Richard H., on 2018-12-27 07:45

@patrickdepinguin Yes, that's right. The port did not install using a virtualenv and since it worked, I never gave it much thought for now. The dependencies weren't heavy either. I might switch to a virtualenv later, but as of right now I am quite new to Python.