`checkdata` : High-level integrity tests¶

The lino.modlib.checkdata plugin adds support for defining application-level data integrity tests. For the application developer it provides a method to define data checkers for their models. For the site maintainer it adds the checkdata django-admin command. For the end user it adds a set of automatic actions.

Before reading this document, we recommend to read the end-user documentation.

This page is a tested document and the following instructions are used for initialization:

>>> from lino import startup
>>> startup('lino_book.projects.roger.settings.demo')
>>> from lino.api.doctest import *
>>> from django.core.management import call_command
>>> from atelier.sheller import Sheller
>>> shell = Sheller(settings.SITE.project_dir.parent)

The application developer can decide to add a MessagesByOwner table to the detail layout of any model. This enables the end users to focus on the data problems related to a given database object.

Data checkers ¶

In the web interface you can select Explorer ‣ System ‣ Data checkers to see a table of all available checkers.

>>> rt.show(checkdata.Checkers, language="en")
... 
=================================== ========================================================
 value                               text
----------------------------------- --------------------------------------------------------
 beid.SSINChecker                    Check for invalid SSINs
 cal.ConflictingEventsChecker        Check for conflicting calendar entries
 cal.EventGuestChecker               Entries without participants
 cal.LongEntryChecker                Too long-lasting calendar entries
 cal.ObsoleteEventTypeChecker        Obsolete generated calendar entries
 countries.PlaceChecker              Check data of geographical places
 courses.MemberChecker               Check membership payments
 finan.FinancialVoucherItemChecker   Check for invalid account/partner combination
 ledger.VoucherChecker               Check integrity of ledger vouchers
 linod.SystemTaskChecker                   Check for missing system tasks
 memo.PreviewableChecker             Check for previewables needing update
 phones.ContactDetailsOwnerChecker   Check for mismatches between contact details and owner
 printing.CachedPrintableChecker     Check for missing target files
 sepa.BankAccountChecker             Check for partner mismatches in bank accounts
 system.BleachChecker                Find unbleached html content
 uploads.UploadChecker               Check metadata of upload files
 uploads.UploadsFolderChecker        Find orphaned files in uploads folder
 vat.VatColumnsChecker               Check VAT columns configuration
 vat.VatIdChecker                    Validate VAT id from online registry
=================================== ========================================================

The lino_xl.lib.countries.PlaceChecker class is a simple example of how to write a data checker:

from lino.api import _
from lino.modlib.checkdata.choicelists import Checker

class PlaceChecker(Checker):
    model = 'countries.Place'
    verbose_name = _("Check data of geographical places.")

    def get_checkdata_problems(self, obj, fix=False):
        if obj.name.isdigit():
            yield (False, _("Name contains only digits."))

PlaceChecker.activate()

More examples of data checkers we recommend to explore:

lino_xl.lib.countries.PlaceChecker
lino_xl.lib.beid.mixins.BeIdCardHolderChecker
lino_xl.lib.addresses.AddressOwnerChecker
lino_xl.lib.vat.VatIdChecker
lino.mixins.dupable.DupableChecker
lino.modlib.uploads.UploadChecker
lino.modlib.uploads.UploadsFolderChecker
lino_welfare.modlib.pcsw.models.SSINChecker
lino_welfare.modlib.pcsw.models.ClientCoachingsChecker
lino_welfare.modlib.isip.mixins.OverlappingContractsChecker
lino_welfare.modlib.dupable_clients.models.SimilarClientsChecker

Showing all data problem messages ¶

In the web interface you can select Explorer ‣ System ‣ Data problem messages to see all problems.

The demo database deliberately contains some data problems.

>>> rt.login("robin").show(checkdata.AllMessages, language="en")
... 
============= ====================================================================== ============================================================== =======================================================================
 Responsible   Database object                                                        Message text                                                   Checker
------------- ---------------------------------------------------------------------- -------------------------------------------------------------- -----------------------------------------------------------------------
 Robin Rood                                                                           File uploads/2015/05/foo.pdf has no upload entry.              uploads.UploadsFolderChecker (Find orphaned files in uploads folder)
 Robin Rood    `Recurring event #4 Assumption of Mary </#/api/cal/Events/25>`__       Event conflicts with Activity #1 001  1.                       cal.ConflictingEventsChecker (Check for conflicting calendar entries)
 Robin Rood    `Recurring event #11 Ascension of Jesus </#/api/cal/Events/77>`__      Event conflicts with Mittagessen (14.05.2015 11:10).           cal.ConflictingEventsChecker (Check for conflicting calendar entries)
 Robin Rood    `Recurring event #12 Pentecost </#/api/cal/Events/85>`__               Event conflicts with 4 other events.                           cal.ConflictingEventsChecker (Check for conflicting calendar entries)
 Rolf Rompen   `Mittagessen (14.05.2015 11:10) </#/api/cal/Events/1120>`__            Event conflicts with Recurring event #11 Ascension of Jesus.   cal.ConflictingEventsChecker (Check for conflicting calendar entries)
 Robin Rood    `First meeting (25.05.2015 13:30) </#/api/cal/Events/1136>`__          Event conflicts with Recurring event #12 Pentecost.            cal.ConflictingEventsChecker (Check for conflicting calendar entries)
 Robin Rood    `Absent for private reasons (25.05.2015) </#/api/cal/Events/1166>`__   Event conflicts with Recurring event #12 Pentecost.            cal.ConflictingEventsChecker (Check for conflicting calendar entries)
 Robin Rood    `Karl Kaivers (ME) </#/api/courses/Pupils/141>`__                      Member until 2015-12-31, but no payment.                       courses.MemberChecker (Check membership payments)
 Robin Rood    `Laura Laschet (ME) </#/api/courses/Pupils/143>`__                     Member until 2015-12-31, but no payment.                       courses.MemberChecker (Check membership payments)
 Robin Rood    `Josefine Leffin (MEL) </#/api/courses/Pupils/145>`__                  Member until 2015-12-31, but no payment.                       courses.MemberChecker (Check membership payments)
 Robin Rood    `Marie-Louise Meier (ME) </#/api/courses/Pupils/149>`__                Member until 2015-12-31, but no payment.                       courses.MemberChecker (Check membership payments)
 Robin Rood    `Alfons Radermacher (ME) </#/api/courses/Pupils/153>`__                Member until 2015-12-31, but no payment.                       courses.MemberChecker (Check membership payments)
 Robin Rood    `Christian Radermacher (MEL) </#/api/courses/Pupils/155>`__            Member until 2015-12-31, but no payment.                       courses.MemberChecker (Check membership payments)
 Robin Rood    `Edgard Radermacher (ME) </#/api/courses/Pupils/157>`__                Member until 2015-12-31, but no payment.                       courses.MemberChecker (Check membership payments)
 Robin Rood    `Guido Radermacher (ME) </#/api/courses/Pupils/159>`__                 Member until 2015-12-31, but no payment.                       courses.MemberChecker (Check membership payments)
 Robin Rood    `Hedi Radermacher (ME) </#/api/courses/Pupils/161>`__                  Member until 2015-12-31, but no payment.                       courses.MemberChecker (Check membership payments)
 Robin Rood    `Jean Radermacher (ME) </#/api/courses/Pupils/163>`__                  Member until 2015-12-31, but no payment.                       courses.MemberChecker (Check membership payments)
 Robin Rood    `Erna Ärgerlich (ME) </#/api/courses/Pupils/169>`__                    Member until 2015-12-31, but no payment.                       courses.MemberChecker (Check membership payments)
 Robin Rood    `Jean Dupont (ME) </#/api/courses/Pupils/171>`__                       Member until 2015-12-31, but no payment.                       courses.MemberChecker (Check membership payments)
 Robin Rood    `Marie-Louise Vandenmeulenbos (MEC) </#/api/courses/Pupils/174>`__     Member until 2015-12-31, but no payment.                       courses.MemberChecker (Check membership payments)
 Robin Rood    `Bernd Brecht (ME) </#/api/courses/Pupils/177>`__                      Member until 2015-12-31, but no payment.                       courses.MemberChecker (Check membership payments)
 Robin Rood    `Jérôme Jeanémart (ME) </#/api/courses/Pupils/181>`__                  Member until 2015-12-31, but no payment.                       courses.MemberChecker (Check membership payments)
 Robin Rood    `Source document PRC_29_2015.pdf </#/api/uploads/Uploads/1>`__         Upload entry uploads/2015/05/PRC_29_2015.pdf has no file       uploads.UploadChecker (Check metadata of upload files)
============= ====================================================================== ============================================================== =======================================================================

Filtering data problem messages ¶

The user can set the table parameters e.g. to see only messages of a given type ("checker"). The following snippet simulates the situation of selecting the ConflictingEventsChecker.

>>> chk = checkdata.Checkers.get_by_value('cal.ConflictingEventsChecker')
>>> rt.show(checkdata.MessagesByChecker, chk)
... 
============= ====================================================================== ==============================================================
 Responsible   Database object                                                        Message text
------------- ---------------------------------------------------------------------- --------------------------------------------------------------
 Robin Rood    `Recurring event #4 Assumption of Mary </#/api/cal/Events/25>`__       Event conflicts with Activity #1 001  1.
 Robin Rood    `Recurring event #11 Ascension of Jesus </#/api/cal/Events/77>`__      Event conflicts with Mittagessen (14.05.2015 11:10).
 Robin Rood    `Recurring event #12 Pentecost </#/api/cal/Events/85>`__               Event conflicts with 4 other events.
 Rolf Rompen   `Mittagessen (14.05.2015 11:10) </#/api/cal/Events/1120>`__            Event conflicts with Recurring event #11 Ascension of Jesus.
 Robin Rood    `First meeting (25.05.2015 13:30) </#/api/cal/Events/1136>`__          Event conflicts with Recurring event #12 Pentecost.
 Robin Rood    `Absent for private reasons (25.05.2015) </#/api/cal/Events/1166>`__   Event conflicts with Recurring event #12 Pentecost.
============= ====================================================================== ==============================================================

Running the checkdata command ¶

>>> set_log_level(logging.WARNING)
>>> call_command('checkdata')

You can see the list of all available checkers also from the command line using:

$ python manage.py checkdata --list

>>> call_command('checkdata', list=True)
... 
=================================== ========================================================
 value                               text
----------------------------------- --------------------------------------------------------
 beid.SSINChecker                    Check for invalid SSINs
 cal.ConflictingEventsChecker        Check for conflicting calendar entries
 cal.EventGuestChecker               Entries without participants
 cal.LongEntryChecker                Too long-lasting calendar entries
 cal.ObsoleteEventTypeChecker        Obsolete generated calendar entries
 countries.PlaceChecker              Check data of geographical places
 courses.MemberChecker               Check membership payments
 finan.FinancialVoucherItemChecker   Check for invalid account/partner combination
 ledger.VoucherChecker               Check integrity of ledger vouchers
 linod.SystemTaskChecker                   Check for missing system tasks
 memo.PreviewableChecker             Check for previewables needing update
 phones.ContactDetailsOwnerChecker   Check for mismatches between contact details and owner
 printing.CachedPrintableChecker     Check for missing target files
 sepa.BankAccountChecker             Check for partner mismatches in bank accounts
 system.BleachChecker                Find unbleached html content
 uploads.UploadChecker               Check metadata of upload files
 uploads.UploadsFolderChecker        Find orphaned files in uploads folder
 vat.VatColumnsChecker               Check VAT columns configuration
 vat.VatIdChecker                    Validate VAT id from online registry
=================================== ========================================================

>>> set_log_level(logging.INFO)
>>> call_command('checkdata', 'cal.')
Run 4 data checkers on 1171 Calendar entries...
- Recurring event #4 Assumption of Mary : Event conflicts with Activity #1 001  1.
- Recurring event #11 Ascension of Jesus : Event conflicts with Mittagessen (14.05.2015 11:10).
- Recurring event #12 Pentecost : Event conflicts with 4 other events.
- Mittagessen (14.05.2015 11:10) : Event conflicts with Recurring event #11 Ascension of Jesus.
- First meeting (25.05.2015 13:30) : Event conflicts with Recurring event #12 Pentecost.
- Absent for private reasons (25.05.2015) : Event conflicts with Recurring event #12 Pentecost.
Found 6 and fixed 0 problems in Calendar entries.
1 check have been run. Found 6 and fixed 0 problems.

>>> call_command('checkdata', 'foo')
Traceback (most recent call last):
...
Exception: No checker matches ('foo',)

The --prune option instructs checkdata to remove all existing error messages before running the tests. This makes the operation quicker on sites with many existing data problem messages. Don't use this in combination with a filter because --prune removes all messages, not only those that you ask to rebuild.

>>> set_log_level(logging.WARNING)
>>> shell("python manage.py checkdata --prune")
... 
Prune 23 existing messages...
Run 1 data checkers on 72 Persons...
Run 6 data checkers on 1171 Calendar entries...
- Recurring event #4 Assumption of Mary : Event conflicts with Activity #1 001  1.
- Recurring event #11 Ascension of Jesus : Event conflicts with Mittagessen (14.05.2015 11:10).
- Recurring event #12 Pentecost : Event conflicts with 4 other events.
- Mittagessen (14.05.2015 11:10) : Event conflicts with Recurring event #11 Ascension of Jesus.
- First meeting (25.05.2015 13:30) : Event conflicts with Recurring event #12 Pentecost.
- Absent for private reasons (25.05.2015) : Event conflicts with Recurring event #12 Pentecost.
Found 6 and fixed 0 problems in Calendar entries.
...
34 checks have been run. Found 23 and fixed 0 problems.

NB the above example uses atelier.sheller instead of call_command. Both methods are functionally equivalent.

Language of checkdata messages ¶

Every detected checkdata problem is stored in the database in the language of the responsible user. A possible pitfall with this is the following example.

The checkdata message "Similar clients" appeared in English and not in the language of the responsible user. That was because the checker did this:

msg = _("Similar clients: {clients}").format(
    clients=', '.join([str(i) for i in lst]))
yield (False, msg)

The correct way is like this:

msg = format_lazy(_("Similar clients: {clients}"),
    clients=', '.join([str(i) for i in lst]))
yield (False, msg)

See Introduction to internationalization (i18n) for details.

Database models ¶

class lino.modlib.checkdata.Problem¶

Django model used to store a data problem message.

checker¶: The Checker that reported this problem.

message¶: The message text. This is a concatenation of all messages that were yielded by the checker.

user¶

The user responsible for fixing this problem.

This field is being filled by the get_responsible_user method of the checker.

class lino.modlib.checkdata.Problems¶: The base table for data problem messages.

class lino.modlib.checkdata.MyMessages¶: Shows the data problem messages assigned to me.

class lino.modlib.checkdata.Checkers¶

The list of data checkers known by this application.

This was the first use case of a ChoiceList with a detail_layout.

class lino.modlib.checkdata.Checker¶

Base class for all data checkers.

model¶

The model to be checked. If this is a string, Lino will resolve it at startup.

If this is an abstract model, get_checkable_models() will yield all models that inherit from it.

If this is None, the checker is unbound, i.e. the problem messages will not be bound to a particular database object. This is used to detect missing database objects. For example vat.VatColumnsChecker is unbound.

Instead of setting model you might want to define your own get_checkable_models() method. For example, ledger.VoucherChecker does this because it wants to get all MTI children (not only top-level models). See Lino core utilities for more explanations.

classmethod check_instance(cls, *args, **kwargs)¶: Run get_checkdata_problems() on this checker for the given database object.

get_checkable_models(self)¶

Return a list of the models to check.

The default implementation returns all top-level models that inherit from model (or [None] when model is None).

classmethod activate(cls)¶

Creates an instance of this class and adds it as a choice to the Checkers choicelist.

The application developer must call this on their subclass in order to "register" or "activate" the checker.

update_problems(self, obj=None, delete=True, fix=False)¶

Update the problem messages of this checker for the specified object.

obj is None on unbound checkers.

When delete is False, the caller is responsible for deleting any existing objects.

get_checkdata_problems(self, obj, fix=False)¶: Return or yield a series of (fixable, message) tuples, each describing a data problem. fixable is a boolean saying whether this problem can be automatically fixed. And if fix is True, this method is also responsible for fixing it.

get_responsible_user(self, obj)¶

The site user to be considered responsible for problems detected by this checker on the given database object obj. This will be stored in user.

The given obj is an instance of model, unless for unbound checkers (i.e. whose model is None).

The default implementation returns the main checkdata responsible defined for this site (see responsible_user).

Automatic actions ¶

This plugin automatically installs two actions on every model that has at least one active data checker:

class lino.core.model.Model

fix_problems¶

Update data problem messages and repair those which are automatically fixable.

Implementing class: lino.modlib.checkdata.FixProblemsByController.

check_data¶

Update data problem messages for this database object, also removing messages that no longer exist. This action does not change anything else in the database.

Implementing class: lino.modlib.checkdata.UpdateMessagesByController.

Internal utilities ¶

lino.modlib.checkdata.get_checkable_models(*args)¶

Return an OrderedDict mapping each model which has at least one checker to a list of these checkers.

The dict is ordered to avoid that checkers run in a random order.

checkdata : High-level integrity tests¶

`checkdata` : High-level integrity tests¶