Kallithea 0.4.0rc1 released

We are happy to announce a release candidate for Kallithea 0.4.0, which will be the next major release after more than 3 years of development. The release notes below are the draft release notes for the final release.

Please try out this release candidate and report any problems you may find. Also, please share your feedback about the release notes proposed below. If there is information missing, confusing, etc. then let us know so we can improve it for the final release.

We hope to create the final release by the end of March.

Release 0.4.0 can be considered a major release, as it encompasses more than 3 years of development since 0.3.0. The release was held back by the gradual introduction of several big changes both in the back-end as in the front-end, causing the code base to be in a state of flux for an extended period of time:

  • the user interface (front-end) was converted to use Bootstrap and less files
  • the web framework (back-end) was changed from Pylons to TurboGears2. As a consequence, paster serve is replaced by gearbox serve.
  • the test framework was changed from nose to pytest.
  • administrative commands previously launched via paster have been integrated in a new script kallithea-cli, except for the serve command (now handled via gearbox).
  • the internal db_migrate scripts have been removed and database migration is now handled via alembic.

In addition to these big changes, there have been improvements and bug fixes in almost all areas.

Performance was improved on several fronts, both in database access as in the user interface.

A number of new API calls were added: get_changeset, get_changesets, get_pullrequest and comment_pullrequest.

Authentication via email address is now supported, in addition to username authentication.

You can now create hyperlinks out of issue references (e.g. statements like issue #123) in a much more powerful way, with full regular expression back reference support.

The documentation has been restructured and improved.

Most JavaScript libraries are no longer shipped with Kallithea itself, but are downloaded via npm as part of the installation procedure.

Notification emails have been completely restyled and their content reworked. They now also display properly in Microsoft Outlook. The 'notifications' feature in the UI itself has been removed.

Support for ReStructuredText (RST) in pull request and changeset comments has been dropped. The comments are now rendered as monospace plain text, allowing diagrams to be rendered correctly without special syntax.

The minimally supported Mercurial version has increased to 4.1.1, supporting up to 4.8. Obsolescence markers and phases are now displayed in changelogs.

See 'Changes' below for more details.

Upgrading from 0.3.x

Please refer to the upgrade instructions in the documentation for detailed steps: https://kallithea.readthedocs.io/en/default/upgrade.html. Below are just some specific notes.

The upgrade process involves an upgrade of the database schema (via alembic). Although this should go smooth, do not forget to make a database backup.

As mentioned in the upgrade instructions, generate a new configuration file and apply the necessary changes to fit your installation. There have been many changes in the configuration file; starting from the old one would be a source of errors or confusion.

The front-end code needs to be built after installation, via kallithea-cli front-end-build. For this, the npm program is expected to be installed, as it is used to install several front-end dependencies.

If you were using paster serve my.ini to start Kallithea, then now use gearbox serve -c my.ini. The gearbox command is supplied by TurboGears2.

Changes

Below are the most relevant changes between 0.3.7 and 0.4.0. Note that it is not a complete list: there have been more than 1500 commits and many of them are purely internal refactoring. Please refer to the repository if you are interested in full details.

API

  • fail when given unknown arguments
  • add API call get_changeset to read metadata and review status for a changeset
  • add API call get_changesets to get a list of changesets in a repository, with some filtering options.
  • add API call get_pullrequest
  • add API call comment_pullrequest
  • create_repo/fork_repo: only allow repo creation in existing groups
  • get_repo: add optional parameter with_revision_names to list branches, tags and bookmarks
  • get_repo: add optional parameter with_pullrequests to list pull requests in the repository
  • pull: allow pulling from a custom remote

Authentication

  • authenticate using either username or email address
  • when a auth plugin can't be imported try the next one instead of breaking completely
  • fix crash on invalid bcrypt password
  • raise log level of 'permission denied' from DEBUG to INFO (Issue #243)
  • better handling of 'password reset' for external authentication: don't suggest the user the password was reset when it isn't.
  • strip RFC4007 zone identifiers from IPv6 addresses before doing access control
  • drop active_from_extern from internal auth API
  • users_and_groups_data should not be available for anonymous/default user
  • recaptcha: Update to Google recaptcha API v2 (Issue #313)
  • LDAP: add option to set custom CA certificates directory (OPT_X_TLS_CACERTDIR)
  • LDAP: prevent query language injection of usernames
  • LDAP: refactor parameter handling - make it clear that port is optional
  • LDAP: better handling of LDAP and authentication errors - don't log a stack trace ... but also don't reveal anything to the user
  • LDAP: change default to LDAPS on port 636 - insecure authentication is kind of pointless
  • crowd: add HTTPS ability (Issue #315)
  • add support for "Bearer" auth scheme (API key variant)
  • let container authentication get email, first and last name from custom headers

Back-end

  • replace Pylons with TurboGears2
  • add TurboGears2 DebugBar support

Configuration file (ini)

  • allow canonical_url to contain more than just a hostname
  • cache: make instance_id = * the default and deprecate it
  • clarify that we only recommend and support single threaded operation
  • drop disabled conditional sections in shipped ini files - use kallithea-cli config-create again when needed
  • drop setting initial_repo_scan: repo scanning is done initially in kallithea-cli db-create and can be requested explicitly in the UI
  • error email settings are comma-separated under backlash
  • i18n: use TG2 compatible settings names i18n.lang and i18n.enabled
  • increase number of gunicorn worker processes in the default configuration
  • issues: backout special whitespace handling
  • issues: support generic regex replacements in issue_url and issue_prefix
  • refactor beaker configuration
  • remove 'pdebug' setting (profiling)
  • set 'debug' only in one place
  • set default log level to WARN
  • set default uwsgi socket listen queue size to 128
  • specify utf8 for sample MySQL connection strings
  • tweak template http_server conditionals - don't leave an empty section when using UWSGI

CLI commands

  • cache-keys: drop 'gearbox cache-keys --show/--cleanup' - they are rarely used, and use cases are covered by commands for updating metadata
  • convert 'gearbox celeryd' into 'kallithea-cli celery-run'
  • convert 'gearbox cleanup-repos' into 'kallithea-cli repo-purge-deleted'
  • convert 'gearbox install-iis' into 'kallithea-cli iis-install'
  • convert 'gearbox ishell' into 'kallithea-cli ishell'
  • convert 'gearbox make-config' into 'kallithea-cli config-create'
  • convert 'gearbox make-index' into 'kallithea-cli index-create'
  • convert 'gearbox make-rcext' in 'kallithea-cli extensions-create'
  • convert 'gearbox repo-scan' into 'kallithea-cli repo-scan'
  • convert 'gearbox setup-db' into 'kallithea-cli db-create'
  • convert 'gearbox update-repoinfo' into 'kallithea-cli repo-update-metadata'
  • config-create: allow configuration of any ini value
  • config-create: drop support for comma separated custom values - just specify multiple values
  • config-create: make the UWSGI host/port configurable
  • db-create: fix failure when db is created for the first time (Issue #272)
  • front-end-build: new command in kallithea-cli
  • iis-install: fix failure - replace paster ensure_file with explicit file writing
  • repo-purge-deleted: improve reporting to user
  • repo-scan: don't report missing repos as removed if not using --remove-missing
  • setup-db: perform an initial repository scan as stated by the docs (Issue #302)

Celery (task scheduler)

  • upgrade to Celery 3
  • clarify that Celery 4 doesn't work

Database

  • add some indexes to improve performance
  • always match user emails case insensitively
  • check database version before starting kallithea
  • cleanup: remove redundant database loads
  • drop length limits on TEXT columns
  • drop RhodeCode compatibility (database rebranding etc.)
  • introduce 'alembic' for database migration
  • match case-insensitively using func.lower, not ilike
  • upgrade to SQLAlchemy 1.1, fixing invalid PostgreSQL SQL in User.is_default_user

Documentation

  • add documentation about internationalization from a user perspective
  • add information about extra dependencies we now need
  • add information on importing repositories
  • auth: improve API key documentation for users
  • document canonical_url - it was introduced long time ago but seems useful
  • document how to use Alembic for database migrations
  • document Kallithea customization options
  • document X-Kallithea-Notification-Type header
  • mention use of static_files, its setup and implications
  • recommend --upgrade for all pip installations in a virtualenv
  • reminder to add c:\path\to\git\bin to PATH
  • remove references to pythonhosted.org (Issue #293)
  • reorganize index page
  • tweak email documentation - add gmail example
  • update email documentation for TurboGears2
  • upgrade: clarify database upgrade instructions
  • upgrade: make upgrade instructions from version control more explicit
  • upgrade: misc. clarification on upgrade from 0.3.x to 0.4
  • vcs_setup: add note about Git HTTP POST buffer

Front-end / style

  • convert to Bootstrap and Less
  • add viewport meta element so small devices react to the responsive design
  • enable visual feedback for Tab-based keyboard menu navigation
  • Introduce 'front-end' directory with source files for building the front-end
  • mark site as IE10+ compatible to bypass Compatibility mode
  • Move .less files to the front-end folder
  • Use codemirror from node_modules and stop bundling it
  • Use DataTables from node_modules and stop bundling it
  • Use jQuery and Flot from node_modules and stop bundling them
  • use monospace for changeset messages
  • use npm less to generate kallithea stylesheet style.css from style.less
  • Use select2 from node_modules and stop bundling it
  • use some ARIA accessibility markup
  • Use the existing bootstrap.js from node_modules and stop bundling Bootstrap

Notifications / Email

  • remove UI notification feature
  • pullrequests: linkify changesets in html notification email to reviewers
  • pullrequests: add URL changesets in txt notification email to reviewers
  • new email templates with html styling
  • improve display of HTML emails in Microsoft Outlook
  • restrict amount of visible hyperlinks
  • restructure notification mail content
  • add relevant title to subject of emails
  • add X-Kallithea-Notification-Type header
  • fix fallback sending to multiple admins

Test suite

  • switch to pytest; remove nose support

Unicode support

  • config: abort early if the environment doesn't allow Python to pass Unicode strings to the file system layer
  • don't crash on sending mails with unicode comments without appropriate environment configuration (Issue #275)
  • hg: set encoding to utf-8 by default to always show unicode characters correctly
  • repositories: fix crash when forking repositories with unicode names
  • repositories: fix unicode error when scanning and finding repositories with unicode names

User interface functionality

Admin

  • apply LOWER() on journal filtering term for suffix/infix matching
  • auth: make sure list of auth modules is consistent
  • better handling of rescan/delete and invalidate at the same time (Issue #252)
  • hooks: only flash 'Updated hooks' if there are changes
  • hooks: prevent creation of existing custom hook
  • hooks: prevent editing of builtin hooks (Issue #226)
  • hooks: restore delete functionality as intended
  • settings: rescope 'Analytics HTML' option to generic HTML/JS/CSS customization
  • settings: fix saving of visual settings of page size

Comments

  • reload after some kinds of general comments - the ones that change the whole page state
  • display comment previews while submitting
  • linkify revision hashes and issue tracker references
  • change comment formatting to plain text instead of RST
  • drop preview - no need for it without RST
  • limit comment deletion to a 5 minutes grace period
  • warn/highlight the fact that this comment is outside the context

Diffs

  • correct handling of links to old filename in renames
  • fix crash when displaying diff on a single file
  • fix display of images for deleted files
  • fix display of links to before / after files - it was not obvious what the hashes / file names did
  • introduce 'show inline comments' control for pull-requests too
  • make pull request diffs collapsable as changeset diffs are
  • only highlight of difference between del and add line for one-liners
  • show correct operation for file diffs instead of '???'
  • tweak mergely side-by-side diff view

Files

  • always show the requested version - not last changeset that touched the file
  • improve select2 performance with many branches
  • set page title explicitly when navigating history
  • support annotation on files larger than cut_off_limit
  • use revision in Edit and Delete links
  • use same parent/child navigation as on changeset page

Journal

  • make "repository:" filtering condition work as expected (Issue #261)
  • make "username:" filtering condition work as expected
  • remove right column on journal page - my & watched repos are already available under "my"

Pull requests

  • better handling of Mercurial pullrequests with missing revisions - don't crash
  • better handling of revision range pullrequests with missing revisions - don't crash
  • don't show empty "additional changesets" (Issue #280)
  • fix "additional changes" js error - make sure all cs tables have a first column to align with (Issue #274)
  • fix broken delete button on overview list
  • handle missing revisions - especially after Git GC (Issue #176)
  • make issue references clickable in pull request titles and lists
  • make it possible to delete pull requests instead of commenting
  • prevent creation of invalid pull requests, empty or unrelated or criss cross
  • reword the grumpy "Rejected" to the more neutral and factual "Not approved"
  • show warning when updating PR and the reviewer list has changed since the form load
  • talk about "iterations" instead of the ambiguous "updates"
  • try to improve usability of PR update functionality
  • when updating a PR, only add and remove the reviewers that actually were added/removed

Search

  • better messaging when the index hasn't been built yet
  • make "repository:" condition work as expected
  • make "repository:" condition work case-insensitively as expected
  • prevent pathname related conditions from removing "stop words"
  • prevent username related conditions from removing "stop words"

Other user interface changes

  • autocomplete: highlight the matched search string
  • autocomplete: use select2 for autocomplete fields
  • base: fix tabbing to navbar links - Chrome requires the <a> to have a href
  • changelog: add evolve information to the graph and mark unstable changesets in red based on that information
  • changelog: add phases tags in changelog
  • changeset: always allow status changes
  • changeset: don't crash with unhandled EmptyRepositoryError when visiting (non-existing) changeset in an empty repo
  • changeset: fix broken revision links in title
  • home: make sure users and group autocomplete is case insensitive
  • js: remove mousetrap - the current keyboard bindings are quite unknown and irrelevant
  • login: clarify "Remember be" checkbox - it controls if session should expire with browser session or at timeout
  • login: have self-signup confirmation message respect site_name
  • prevent diff line numbers from being pasted to text
  • repositories: allow to pull changes from forked repo
  • repositories: make sure repositories not only differ in casing
  • repository groups: fix display of breadcrumb for nested repo groups
  • repository groups: fix display of links to parent repo groups
  • repository groups: show breadcrumb

Version control systems support

Git

  • add references for Git pull request heads
  • catch MemoryErrors when calling Git diff
  • clarify that non-bare git repos not are supported (Issue #254)
  • db: ensure git hooks work when the repositories base path is a symlink
  • fix push to empty repo (Issue #323)
  • graph: detect git branches and colorize them properly without rainbow effect (Issue #188)
  • hooks: cleanup and documentation of the installed Git hooks
  • hooks: if available, use sys.executable as executable for git hooks
  • improve performance working with git changesets
  • include an LF at the end of the service advertisement (Issue #230)
  • introduce 'branches' attribute on changesets, making it possible for Git to show multiple branches for a changeset
  • model: fix incorrect reporting of pull from remote git repo
  • submodules: fix links to nodes that are submodule links
  • submodules: fix URL for submodules - make it link to the external URL
  • submodules: let links to git submodules be real links that can open in new tabs - not just in-page navigation
  • submodules: remove support for submodules without a URL

Mercurial

  • Fix showing of obsolescence markers in Mercurial 4.6+ (Issue #330)
  • Increase minimum Mercurial version to 4.1.1
  • Support Mercurial up to 4.8

Thanks

We would like to thank everyone that contributed to the Kallithea repository (the numbers are the amount of commits)...

 811 Mads Kiilerich
 375 Dominik Ruf
 289 Thomas De Schampheleire
 143 Søren Løvborg
  56 timeless@gmail.com
  45 Andrej Shadura
  13 Konstantin Veretennicov
  12 Lars Kruse
  11 Branko Majic
  10 FUJIWARA Katsunori
   8 Takumi IINO
   6 Manuel Jacob
   4 Robert Rauch
   4 Anton Schur
   4 Jan Heylen
   3 Étienne Gilli
   3 Asterios Dimitriou
   3 Eivind Tagseth
   3 Kateryna Musina
   2 Patrick Vane
   2 Pheng Heong Tan
   2 Karl Goetz
   2 Daniel Hobley
   2 Viktar Vauchkevich
   2 Angel Ezquerra
   2 Jiří Suchan
   1 Sam Jaques
   1 Robert Martinez
   1 ssantos
   1 Ching-Chen Mao
   1 YFdyh000
   1 Marko Semet
   1 Allan Nordhøy
   1 Jesús Sánchez
   1 Alessandro Molina
   1 Danni Randeris
   1 Chris Rule
   1 Ante Ilic
   1 Holger Schramm
   1 Edmund Wong
   1 Oscar Curero

... as well as everyone contributing in other ways, e.g. by reporting issues, discussing via mail or IRC, etc.