Kallithea News

Posted on 31 Mar 2019

We are very happy to announce Kallithea 0.4.0. This is a major release, including a lot of development since 0.3.0 was released 3 years ago. The release includes several big changes, both in back-end and front-end. These changes are now finished, leaving Kallithea in a better and more consistent state than ever. We consider this release ready for production use, even though it isn't feature complete yet and we would like to make more changes before calling it "1.0".

This also marks the end of the 0.3.x set of releases: any new bug fixes or security fixes will no longer be applied to 0.3.x but only to 0.4.x. We encourage everyone to upgrade to 0.4.0 so they are on a supported track.

The main changes in this release are:

the front-end was converted to use Bootstrap and less files. The intention of these changes for now has been to refactor and clean up the code base. As a bonus, rendering on small screens (mobile devices) has improved a lot. But, even though the interface certainly looks cleaner, the overall visual look has not changed that much.
the back-end web framework was changed from Pylons to TurboGears2. As a consequence, paster serve is replaced by gearbox serve.
the test framework was changed from nose to pytest.
administrative commands previously launched via paster have been integrated in a new script kallithea-cli, except for the serve command (now handled via gearbox).
the internal db_migrate scripts have been removed and database migration is now handled via alembic.

In addition to these big changes, there have been improvements and bug fixes in almost all areas.

Performance was improved on several fronts, both in database access as in the user interface.

A number of new API calls were added: get_changeset, get_changesets, get_pullrequest and comment_pullrequest.

Authentication via email address is now supported, in addition to username authentication.

You can now create hyperlinks out of issue references (e.g. statements like bug #123) in a much more powerful way, with full regular expression back reference support.

The documentation has been restructured and improved.

The npm program is now used to download sources and tools for the web front-end. Thanks to this, the Kallithea release itself ships much less 3rd-party code.

Notification emails have been completely restyled and their content reworked. They now also display properly in Microsoft Outlook. The 'notifications' feature in the UI itself has been removed.

Pull request and changeset comments, pull request descriptions and commit messages are now rendered as monospaced plain text, allowing source snippets, markup snippets and diagrams to be rendered correctly without special syntax. Support for ReStructuredText (RST) in comments has been dropped, but rendering of repository README files in ReStructuredText format is still supported.

The minimally supported Mercurial version has increased to 4.1.1, supporting up to 4.9. Obsolescence markers and phases are now displayed in changelogs.

See 'Changes' below for more details.

Upgrading from 0.3.x

Please refer to the upgrade instructions in the documentation for detailed steps: https://kallithea.readthedocs.io/en/stable/upgrade.html. Below are just some specific notes.

The upgrade process involves an upgrade of the database schema (via alembic). Although this should go smooth, do not forget to make a database backup.

As mentioned in the upgrade instructions, generate a new configuration file and apply the necessary changes to fit your installation. There have been many changes in the configuration file; starting from the old one would be a source of errors or confusion.

The front-end code needs to be built after installation, via kallithea-cli front-end-build. For this, the npm program is expected to be installed, as it is used to install several front-end dependencies.

If you were using paster serve my.ini to start Kallithea, then now use gearbox serve -c my.ini. The gearbox command is supplied by TurboGears2.

There have been changes to the Git hooks. Don't forget to reinstall them if you have Git repositories, as mentioned in the upgrade instructions. This is not necessary for Mercurial repositories.

Changes

Below are the most relevant changes between 0.3.7 and 0.4.0. Note that it is not a complete list: there have been more than 1500 commits. Many changes are purely internal refactoring. On the other hand, some refactorings / rewritings might have fixed a lot of problems or have bigger implications that can be enumerated.

Please refer to the source repository if you are interested in full details.

API

fail when given unknown arguments
add API call get_changeset to read metadata and review status for a changeset
add API call get_changesets to get a list of changesets in a repository, with some filtering options.
add API call get_pullrequest
add API call comment_pullrequest
create_repo/fork_repo: only allow repo creation in existing groups
get_repo: add optional parameter with_revision_names to list branches, tags and bookmarks
get_repo: add optional parameter with_pullrequests to list pull requests in the repository
pull: allow pulling from a custom remote

Authentication

authenticate using either username or email address
when a auth plugin can't be imported try the next one instead of breaking completely
raise log level of 'permission denied' from DEBUG to INFO (Issue #243)
better handling of 'password reset' for external authentication: don't suggest the user the password was reset when it isn't.
better support for IPv6 by stripping RFC4007 zone identifiers from IPv6 addresses before doing access control
drop active_from_extern from internal auth API
users_and_groups_data should not be available for anonymous/default user
recaptcha: Update to Google recaptcha API v2 (Issue #313)
LDAP: add option to set custom CA certificates directory (OPT_X_TLS_CACERTDIR)
LDAP: prevent query language injection of usernames
LDAP: refactor parameter handling - make it clear that port is optional
LDAP: better handling of LDAP and authentication errors - don't log a stack trace ... but also don't reveal anything to the user
LDAP: change default to LDAPS on port 636 - insecure authentication is kind of pointless
crowd: add HTTPS ability (Issue #315)
add support for "Bearer" auth scheme (API key variant)
let container authentication get email, first and last name from custom headers

Back-end

replace Pylons with TurboGears2

Configuration file (ini)

allow canonical_url to contain more than just a hostname
cache: make instance_id = * the default and deprecate it
clarify that we only recommend and support single threaded operation
drop disabled conditional sections in shipped ini files - use kallithea-cli config-create again when needed
drop setting initial_repo_scan: repo scanning is done initially in kallithea-cli db-create and can be requested explicitly in the UI
error email settings are comma-separated under backlash
i18n: use TG2 compatible settings names i18n.lang and i18n.enabled
increase number of gunicorn worker processes in the default configuration
issues: backout special whitespace handling
issues: support generic regex replacements in issue_url and issue_prefix
refactor beaker configuration
remove 'pdebug' setting (profiling)
set 'debug' only in one place
set default log level to WARN
set default uwsgi socket listen queue size to 128
specify utf8 for sample MySQL connection strings
tweak template http_server conditionals - don't leave an empty section when using UWSGI

CLI commands

cache-keys: drop 'gearbox cache-keys --show/--cleanup' - they are rarely used, and use cases are covered by commands for updating metadata
convert gearbox celeryd into kallithea-cli celery-run
convert gearbox cleanup-repos into kallithea-cli repo-purge-deleted
convert gearbox install-iis into kallithea-cli iis-install
convert gearbox ishell into kallithea-cli ishell
convert gearbox make-config into kallithea-cli config-create
convert gearbox make-index into kallithea-cli index-create
convert gearbox make-rcext into kallithea-cli extensions-create
convert gearbox repo-scan into kallithea-cli repo-scan
convert gearbox setup-db into kallithea-cli db-create
convert gearbox update-repoinfo into kallithea-cli repo-update-metadata
config-create: allow configuration of any ini value
config-create: drop support for comma separated custom values - just specify multiple values
config-create: make the UWSGI host/port configurable
front-end-build: new command in kallithea-cli
repo-purge-deleted: improve reporting to user
repo-scan: don't report missing repos as removed if not using --remove-missing
setup-db: perform an initial repository scan as stated by the docs (Issue #302)

Celery (task scheduler)

upgrade to Celery 3
clarify that Celery 4 doesn't work

Database

add some indexes to improve performance
always match user emails case insensitively
check database version before starting Kallithea
cleanup: remove redundant database loads
drop length limits on TEXT columns
drop RhodeCode compatibility (database rebranding etc.)
introduce 'alembic' for database migration
match case-insensitively using func.lower, not ilike
upgrade to SQLAlchemy 1.1, fixing invalid PostgreSQL SQL in User.is_default_user

Documentation

add documentation about internationalization from a user perspective
add information about extra dependencies we now need
add information on importing repositories
auth: improve API key documentation for users
document canonical_url - it was introduced long time ago but seems useful
document how to use Alembic for database migrations
document Kallithea customization options
document X-Kallithea-Notification-Type header
mention use of static_files, its setup and implications
recommend --upgrade for all pip installations in a virtualenv
reminder to add c:\path\to\git\bin to PATH
remove references to pythonhosted.org (Issue #293)
reorganize index page
tweak email documentation - add gmail example
update email documentation for TurboGears2
upgrade: clarify database upgrade instructions
upgrade: make upgrade instructions from version control more explicit
upgrade: misc. clarification on upgrade from 0.3.x to 0.4
vcs_setup: add note about Git HTTP POST buffer

Front-end / style

convert to Bootstrap and Less
add viewport meta element so small devices react to the responsive design
enable visual feedback for Tab-based keyboard menu navigation
Introduce 'front-end' directory with source files for building the front-end
mark site as IE10+ compatible to bypass Compatibility mode
Move .less files to the front-end folder
use monospace for changeset messages
use npm less to generate style.css from style.less
use some ARIA accessibility markup
Use codemirror from node_modules and stop bundling it
Use DataTables from node_modules and stop bundling it
Use jQuery and Flot from node_modules and stop bundling them
Use select2 from node_modules and stop bundling it
Use the existing bootstrap.js from node_modules and stop bundling Bootstrap

Notifications / Email

remove UI notification feature
pullrequests: linkify changesets in html notification email to reviewers
pullrequests: add URL changesets in txt notification email to reviewers
new email templates with html styling
improve display of HTML emails in Microsoft Outlook
restrict amount of visible hyperlinks
restructure notification mail content
add relevant title to subject of emails
add X-Kallithea-Notification-Type header
fix fallback sending to multiple admins

Test suite

switch to pytest; remove nose support

Unicode support

config: abort early if the environment doesn't allow Python to pass Unicode strings to the file system layer
don't crash on sending mails with unicode comments without appropriate environment configuration (Issue #275)
hg: set encoding to utf-8 by default to always show unicode characters correctly
repositories: fix crash when forking repositories with unicode names
repositories: fix unicode error when scanning and finding repositories with unicode names

User interface functionality

Admin

apply LOWER() on journal filtering term for suffix/infix matching
auth: make sure list of auth modules is consistent
better handling of rescan/delete and invalidate at the same time (Issue #252)
hooks: only flash 'Updated hooks' if there are changes
hooks: prevent creation of existing custom hook
hooks: prevent editing of builtin hooks (Issue #226)
hooks: restore delete functionality as intended
settings: rescope 'Analytics HTML' option to generic HTML/JS/CSS customization
settings: fix saving of visual settings of page size
settings: improve and fix handling of clone URI templates (Issue #336)

Comments

reload after some kinds of general comments - the ones that change the whole page state
display comment previews while submitting
linkify revision hashes and issue tracker references
change comment formatting to plain text instead of RST
drop preview - no need for it without RST
limit comment deletion to a 5 minutes grace period
warn/highlight the fact that this comment is outside the context

Diffs

correct handling of links to old filename in renames
fix crash when displaying diff on a single file
fix display of images for deleted files
fix display of links to before / after files - it was not obvious what the hashes / file names did
introduce 'show inline comments' control for pull-requests too
make pull request diffs collapsable as changeset diffs are
only highlight of difference between del and add line for one-liners
show correct operation for file diffs instead of '???'
tweak mergely side-by-side diff view

Files

always show the requested version - not last changeset that touched the file
improve select2 performance with many branches
set page title explicitly when navigating history
support annotation on files larger than cut_off_limit
use revision in Edit and Delete links
use same parent/child navigation as on changeset page

Journal

make "repository:" filtering condition work as expected (Issue #261)
make "username:" filtering condition work as expected
remove right column on journal page - my & watched repos are already available under "my"

Pull requests

better handling of Mercurial pullrequests with missing revisions - don't crash
better handling of revision range pullrequests with missing revisions - don't crash
don't show empty "additional changesets" (Issue #280)
fix "additional changes" js error - make sure all cs tables have a first column to align with (Issue #274)
fix broken delete button on overview list
handle missing revisions - especially after Git GC (Issue #176)
make issue references clickable in pull request titles and lists
make it possible to delete pull requests instead of commenting
prevent creation of invalid pull requests, empty or unrelated or criss cross
reword the grumpy "Rejected" to the more neutral and factual "Not approved"
show warning when updating PR and the reviewer list has changed since the form load
talk about "iterations" instead of the ambiguous "updates"
try to improve usability of PR update functionality
when updating a PR, only add and remove the reviewers that actually were added/removed

Search

better messaging when the index hasn't been built yet
make "repository:" condition work as expected
make "repository:" condition work case-insensitively as expected
prevent pathname related conditions from removing "stop words"
prevent username related conditions from removing "stop words"

Other user interface changes

autocomplete: highlight the matched search string
autocomplete: use select2 for autocomplete fields
base: fix tabbing to navbar links - Chrome requires the <a> to have a href
changelog: add evolve information to the graph and mark unstable changesets in red based on that information
changelog: add phases tags in changelog
changeset: always allow status changes
changeset: don't crash with unhandled EmptyRepositoryError when visiting (non-existing) changeset in an empty repo
changeset: fix broken revision links in title
home: make sure users and group autocomplete is case insensitive
js: remove mousetrap - the current keyboard bindings are quite unknown and irrelevant
login: clarify "Remember be" checkbox - it controls if session should expire with browser session or at timeout
login: have self-signup confirmation message respect site_name
prevent diff line numbers from being pasted to text
repositories: allow to pull changes from forked repo
repositories: make sure repositories not only differ in casing
repository groups: fix display of breadcrumb for nested repo groups
repository groups: fix display of links to parent repo groups
repository groups: show breadcrumb

Version control systems support

Git

add references for Git pull request heads
catch MemoryErrors when calling Git diff
clarify that non-bare git repos not are supported (Issue #254)
db: ensure git hooks work when the repositories base path is a symlink
fix push to empty repo (Issue #323)
graph: detect git branches and colorize them properly without rainbow effect (Issue #188)
hooks: cleanup and documentation of the installed Git hooks
hooks: if available, use sys.executable as executable for git hooks
improve performance working with git changesets
include an LF at the end of the service advertisement (Issue #230)
introduce 'branches' attribute on changesets, making it possible for Git to show multiple branches for a changeset
model: fix incorrect reporting of pull from remote git repo
submodules: fix links to nodes that are submodule links
submodules: fix URL for submodules - make it link to the external URL
submodules: let links to git submodules be real links that can open in new tabs - not just in-page navigation
submodules: remove support for submodules without a URL

Mercurial

Fix showing of obsolescence markers in Mercurial 4.6+ (Issue #330)
Increase minimum Mercurial version to 4.1.1
Support Mercurial up to 4.9

Thanks

We would like to thank everyone that contributed to the Kallithea repository since release 0.3 (the numbers are the amount of commits)...

823 Mads Kiilerich
375 Dominik Ruf
312 Thomas De Schampheleire
143 Søren Løvborg
 56 timeless@gmail.com
 45 Andrej Shadura
 13 Konstantin Veretennicov
 12 Lars Kruse
 11 Branko Majic
 10 FUJIWARA Katsunori
  8 Takumi IINO
  6 Manuel Jacob
  5 Étienne Gilli
  4 Robert Rauch
  4 Allan Nordhøy
  4 Anton Schur
  4 Jan Heylen
  3 Asterios Dimitriou
  3 Eivind Tagseth
  3 Kateryna Musina
  2 Patrick Vane
  2 Pheng Heong Tan
  2 Karl Goetz
  2 Daniel Hobley
  2 Viktar Vauchkevich
  2 Angel Ezquerra
  2 Jiří Suchan
  1 Sam Jaques
  1 Robert Martinez
  1 ssantos
  1 Ching-Chen Mao
  1 YFdyh000
  1 Marko Semet
  1 Jesús Sánchez
  1 Alessandro Molina
  1 Danni Randeris
  1 Chris Rule
  1 Ante Ilic
  1 Holger Schramm
  1 Edmund Wong
  1 Oscar Curero

... as well as everyone contributing in other ways, e.g. by reporting issues, discussing via mail or IRC, etc.

Kallithea 0.4.0 released