linod: asynchronous Lino daemon

The lino.modlib.linod plugin adds functionality for background work such as running system tasks, sending notifications, and logging to the lino.log from multiple processes. It uses the ASGI interface to run an asynchronous Lino daemon process.

What it does

The Lino daemon process is responsible for running the scheduled background jobs defined by your application. This includes for example:

  • Send out emails for notifications

  • Run daily maintenance tasks such as checkdata or checksummaries.

Usage

Applications can register a job using the dd.api.schedule_often() or dd.api.schedule_daily() decorators. For example (taken from lino.modlib.checkdata):

@dd.schedule_daily()
def checkdata(ar):
    """Run all data checkers."""
    check_data(fix=False)

The code for the example above should be in one of your application's models.py modules.

You can see a status of these jobs by running:

$ cd ~/projects/mysite
$ python manage.py linod

How to activate

A site maintainer can activate this plugin automatically by setting the use_lino site setting to True.

use_linod

Whether to install the lino.modlib.linod plugin and activate asynchronous functionality.

>>> import lino
>>> lino.startup('lino_book.projects.noi1e.settings.demo')
>>> from lino.api.doctest import *

System tasks

class lino.modlib.linod.Procedure

Is basically a Job which contains a function func to run at default interval given by every_unit and every_value.

The default interval can be overridden by JobRule.

func

The function to run as a system task.

Type:

Callable[[BaseRequest], None]

every_unit

The default unit of the interval at which the task func will run.

Type:

str

every_value

The default value of the interval at which the task func will run.

Type:

int

start_datetime

The time at which this task should run first.

Type:

datetime.datetime

run(self, ar)

Calls the function stored in func passing ar as a positional argument.

Parameters:

ar -- an instance of BaseRequest

class lino.modlib.linod.Procedures

The choicelist that contains instances Procedure as choices.

>>> rt.show(linod.Procedures)
============================== ============================== ==============================
 value                          name                           text
------------------------------ ------------------------------ ------------------------------
 clear_too_old_job_history      clear_too_old_job_history      clear_too_old_job_history
 event_notification_scheduler   event_notification_scheduler   event_notification_scheduler
 checksummaries                 checksummaries                 checksummaries
 checkdata                      checkdata                      checkdata
 send_pending_emails_often      send_pending_emails_often      send_pending_emails_often
 send_pending_emails_daily      send_pending_emails_daily      send_pending_emails_daily
 clear_seen_messages            clear_seen_messages            clear_seen_messages
============================== ============================== ==============================
class lino.modlib.linod.LogLevels

A choicelist with the equivalent log levels from the logging module.

class lino.modlib.linod.JobRule

Overrides the recurrent rule of a Procedure.

A subclass of Sequenced and RecurrenceSet.

name

Some name for this job rule.

Type:

str

procedure

The pointer to an instance of Procedure.

cancelled

Tells whether the task should be ignored.

cancelled is set to True if the tasks fails and raises an exception, otherwise can be set cancelled from the ui.

Type:

bool

log_level

Tells what sort of messeages to put into the Job.message field.

Type:

Choice

Value:

LogLevels.notset

run(self, ar, lgr=None) → Job

Performs a routine job.

Parameters:

  • ar -- An instance of BaseRequest

  • lgr -- Logger obtained by calling logging.getLogger.

Returns:

An instance of Job.

class lino.modlib.linod.Job

A historical record of a system task.

start_datetime

Tells at what time exactly this job started.

Type:

datetime.datetime

rule

A pointer to an instance of JobRule which describes the nature of the task.

message

Stores information about the job, mostly logs.

class lino.modlib.linod.JobRules

The default actor for the JobRule model.

class lino.modlib.linod.Jobs

The default actor for the Jobs model.

class lino.modlib.linod.JobsByRule
class lino.modlib.linod.tasks.Tasks

The base class that manages the system tasks.

setup(self)

Gathers all the jobs defined using dd.api.schedule_often() and dd.api.schedule_daily() decorator.

This method is called at startup of the linod worker process. Also does cleaning up of removed tasks and reactivating the cancelled tasks.

>>> from lino.modlib.linod.tasks import Tasks
>>> tasks = Tasks()
>>> tasks.setup()
run(self) → datetime.datetime

Runs the system tasks which are due for the current time.

Returns:

an instance of datetime.datetime; tells the parent scope at what time the next call to run() method should be done.

status(self) → list[str]

Returns the status of all the tasks.

>>> for task_status in tasks.status():
...     print(task_status)
... 
Job rule #1 <Procedures.clear_too_old_job_history every 1 daily> next run at: ...
Job rule #2 <Procedures.event_notification_scheduler every 300 secondly> next run at: ...
Job rule #3 <Procedures.checksummaries every 1 daily> next run at: ...
Job rule #4 <Procedures.checkdata every 1 daily> next run at: ...
Job rule #5 <Procedures.send_pending_emails_often every 10 secondly> next run at: ...
Job rule #6 <Procedures.send_pending_emails_daily every 1 daily> next run at: ...
Job rule #7 <Procedures.clear_seen_messages every 1 daily> next run at: ...
create_rule()

Creates a JobRule for a given Procedure

Parameters:

procedure -- an instance of Procedure

cancel_rule()

Sets Job.cancelled to True for a given rule.

Parameters:

rule -- an instance of JobRule

class lino.modlib.linod.SetupTasks

An action to adjust the database objects to the latest changes.

Should be run only in development environment where no linod worker present. A production run of linod worker does this on each spin up by calling Tasks.setup.

class lino.modlib.linod.RunJob

An action to run a job immediately.

Simulating the daemon in a developer environment

To run the Lino daemon in a development environment run python manage.py runworker `linod_{{project_name}} here if your project name is my_project you will need to run python manage.py runworker linod_my_project. And afterwards to initiate the background tasks and logging you will have to run the python manage.py initiate_linod command in an another terminal.

Usage for developers

The lino.modlib.linod plugin is dependent on a running redis-server in the background and also requires the django-channels, channels-redis Python packages to be installed.

To install redis on a Debian-based Linux distribution run the following command as root:

$ apt update
$ apt install redis

To install the dependent python packages run the following command after activating your Python environment:

$ pip install django-channels channels-redis

Now you simply go to your project directory and invoke the admin command:

$ cd ~/projects/mysite
$ python manage.py runworker linod_mysite

This process will run as long as you don't kill it, e.g. until you hit Ctrl-C.

Open up another terminal window, on which invoke the following command to initiate the background task's cor-routine and logger cor-routine:

$ cd ~/projects/mysite
$ python manage.py initiate_linod