`linod`: The Lino daemon¶

This page documents the linod plugin for developers. We assume that you have read the end-user docs.

This plugin defines the linod admin command, which is responsible for running the background tasks.

The “d” stands for “daemon”, like in sshd, cupsd, systemd and other background processes on Linuxs systems.

When use_channels is True, this plugin uses channels to provide an ASGI application, and the linod command then includes Channels’ runworker command.

This page contains code snippets (lines starting with >>>), which are being tested during our development workflow. The following snippet initializes the demo project used throughout this page.

>>> from lino_book.projects.noi1e.startup import *

Usage ¶

Other plugins can register a background task using the schedule_often() or schedule_daily() decorators. For example (taken from lino.modlib.checkdata):

@dd.schedule_daily()
def checkdata(ar):
    """Run all data checkers."""
    check_data(fix=False)

The code that registers a background task should be in your plugin’s models.py modules.

lino.modlib.linod.schedule_daily(**kwargs)¶: Shortcut to create a background_task() that is suggested to run every day.

lino.modlib.linod.schedule_often(**kwargs)¶: Shortcut to create a background_task() that is suggested to run every 10 seconds.

lino.modlib.linod.background_task(**kwargs)¶

Register the decorated function as a background task.

Keyword arguments are used as default values when Lino creates a SystemTask instance for this procedure.

Except for the special keyword class_name, which defaults to “linod.SystemTask”. It is used by lino_xl.lib.invoicing to register a procedure that will create an invoicing task instead of a normal background task. lino_xl.lib.invoicing.InvoicingTask instead of SystemTask.

Plugin configuration ¶

lino.modlib.linod.use_channels¶: Whether to use channels and daphne to run in asynchronous mode.

lino.modlib.linod.background_sleep_time¶: How many seconds the background task runner should sleep when there is nothing to do.

lino.modlib.linod.daemon_user¶: Name of the site user who executes the background task runner.

The `linod` admin command ¶

>>> from atelier.sheller import Sheller
>>> shell = Sheller(settings.SITE.project_dir)

linod¶

Use this in a developer environment to run a Lino daemon.

How to play with it:

$ pm prep
$ pm linod
Running worker for channels ['linod_settings']
Start task runner using <Logger lino (INFO)>...
Run 1 data checkers on 110 Vouchers...
...

And there it seems to hang, but actually it is just sleeping: checking every 5 seconds whether there is something to do. You can ask it to be make more noise when sleeping if you set LINO_LOGLEVEL to DEBUG before running it:

$ LINO_LOGLEVEL=DEBUG pm linod

>>> shell("django-admin linod --help")
usage: django-admin linod [-h] [--force] [--version] [-v {0,1,2,3}] [--settings SETTINGS]
                          [--pythonpath PYTHONPATH] [--traceback] [--no-color]
                          [--force-color] [--skip-checks]

options:
  -h, --help            show this help message and exit
  --force               Force starts the runworker process even if a log_socket_file exists.
                        Use only in production server.
  --version             Show program's version number and exit.
  -v {0,1,2,3}, --verbosity {0,1,2,3}
                        Verbosity level; 0=minimal output, 1=normal output, 2=verbose output,
                        3=very verbose output
  --settings SETTINGS   The Python path to a settings module, e.g. "myproject.settings.main".
                        If this isn't provided, the DJANGO_SETTINGS_MODULE environment
                        variable will be used.
  --pythonpath PYTHONPATH
                        A directory to add to the Python path, e.g.
                        "/home/djangoprojects/myproject".
  --traceback           Display a full stack trace on CommandError exceptions.
  --no-color            Don't colorize the command output.
  --force-color         Force colorization of the command output.
  --skip-checks         Skip system checks.

Background procedures ¶

>>> rt.show(linod.Procedures)
============================== ============================== ============================== ================== =============================================================
 value                          name                           text                           Task class         Suggested recurrency
------------------------------ ------------------------------ ------------------------------ ------------------ -------------------------------------------------------------
 event_notification_scheduler   event_notification_scheduler   event_notification_scheduler   linod.SystemTask   every=300, every_unit=secondly
 generate_calendar_entries      generate_calendar_entries      generate_calendar_entries      linod.SystemTask   every=1, every_unit=daily
 checksummaries                 checksummaries                 checksummaries                 linod.SystemTask   every=1, every_unit=daily
 checkdata                      checkdata                      checkdata                      linod.SystemTask   every=1, every_unit=daily
 send_weekly_report             send_weekly_report             send_weekly_report             linod.SystemTask   every=1, every_unit=weekly, saturday=True, start_time=04:00
 delete_older_changes           delete_older_changes           delete_older_changes           linod.SystemTask   every=1, every_unit=daily
 send_pending_emails_often      send_pending_emails_often      send_pending_emails_often      linod.SystemTask   every=10, every_unit=secondly
 send_pending_emails_daily      send_pending_emails_daily      send_pending_emails_daily      linod.SystemTask   every=1, every_unit=daily
 clear_seen_messages            clear_seen_messages            clear_seen_messages            linod.SystemTask   every=1, every_unit=daily
 read_inbox                     read_inbox                     read_inbox                     linod.SystemTask   every=300, every_unit=secondly
 run_invoicing_tasks            run_invoicing_tasks            run_invoicing_tasks            invoicing.Task     every=1, every_unit=daily
============================== ============================== ============================== ================== =============================================================

While the procedures are in a choicelist (i.e. end users cannot edit them), the list of system tasks is configurable. The default situation is that every procedure has created one system task:

Logging levels ¶

>>> rt.show(linod.LogLevels)
========== ========== ===============
 value      text       Numeric value
---------- ---------- ---------------
 DEBUG      DEBUG      10
 INFO       INFO       20
 WARNING    WARNING    30
 ERROR      ERROR      40
 CRITICAL   CRITICAL   50
========== ========== ===============

DEBUG means to include detailed debug messages. You should not set this for a longer period on a production site because it bloats the log files.

INFO means to show informative messages.

WARNING is the recommended value for most tasks. Only warnings and error messages are logged.

The levels ERROR and CRITICAL (log only errors and critical messages) exist only for exceptional situations. You should probably not use them.

System tasks ¶

>>> rt.show(linod.SystemTasks)
===== ============================== =============== ========== ======================= ==============================
 No.   Name                           Logging level   Disabled   Status                  Background procedure
----- ------------------------------ --------------- ---------- ----------------------- ------------------------------
 1     event_notification_scheduler   WARNING         No         Scheduled to run asap   event_notification_scheduler
 2     generate_calendar_entries      INFO            No         Scheduled to run asap   generate_calendar_entries
 3     checksummaries                 INFO            No         Scheduled to run asap   checksummaries
 4     checkdata                      INFO            No         Scheduled to run asap   checkdata
 5     send_weekly_report             INFO            No         Scheduled to run asap   send_weekly_report
 6     delete_older_changes           INFO            No         Scheduled to run asap   delete_older_changes
 7     send_pending_emails_often      WARNING         No         Scheduled to run asap   send_pending_emails_often
 8     send_pending_emails_daily      INFO            No         Scheduled to run asap   send_pending_emails_daily
 9     clear_seen_messages            INFO            No         Scheduled to run asap   clear_seen_messages
 10    read_inbox                     WARNING         No         Scheduled to run asap   read_inbox
===== ============================== =============== ========== ======================= ==============================

>>> rt.show(invoicing.Task)
===== ================================== ========== =========== =======================
 No.   Name                               Disabled   When        Status
----- ---------------------------------- ---------- ----------- -----------------------
 1     Make Service reports (SRV)         No         Every day   Scheduled to run asap
 2     Make Sales invoices (SLS)          No         Every day   Scheduled to run asap
 3     Make Subscription invoices (SUB)   No         Every day   Scheduled to run asap
===== ================================== ========== =========== =======================

Dependencies ¶

When use_channels is True, the lino.modlib.linod plugin requires the django-channels and channels-redis Python packages to be installed, as well as a running redis-server.

To install redis on a Debian-based Linux distribution, run the following command as root:

$ apt update
$ apt install redis

To install the required Python packages, run the following command after activating your Python environment:

$ pm install

>>> list(dd.plugins.linod.get_requirements(settings.SITE))
['channels', 'channels_redis', 'daphne']

Usage for developers ¶

To run the Lino daemon in a development environment, run pm linod in your project directory:

$ cd ~/lino/lino_local/mysite
$ pm linod

This process will run as long as you don’t kill it, e.g. until you hit Ctrl-C.

Another way to kill the linod process is using the kill command:

$ kill -s SIGTERM 123456

If you kill linod with another signal than SIGTERM, Lino will not run its shutdown method, which is responsible e.g. for logging the “Done” message.

You may change the logging level by setting LINO_LOGLEVEL:

$ LINO_LOGLEVEL=debug pm linod

Testing instructions for developers ¶

General remarks:

The following demo projects are useful when testing linod:
- cms1 has some background tasks, but use_channels set to False
- noi1r has use_channels set to True.
- chatter : an instant messaging system has no background tasks, but it has use_channels set to True.
use_channels changes the way pm linod works.
For testing logging to a file you can create a log directory, and don’t forget to remove it after your tests because a log directory causes different output for certain commands and would cause the unit test suite to fail if you forget to delete it.
When you start pm runserver before pm linod, runserver will write directly to the lino.log file because there is no socket file. Two processes writing to the same file is likely to cause unpredictable results.
You can set Site.log_each_action_request to True

Example testing session 1

In terminal 1:

go noi1e
mkdir settings/log
LINO_LOGLEVEL=debug pm linod

In terminal 2:

go noi1r
pm runserver

In your browser: sign in as robin, go to Configure ‣ System ‣ System tasks, click “Run now” on one of them. The linod process in terminal 1 should run the task.

In terminal 1, hit Ctrl-C to stop the linod. Then do something in the browser and verify that runserver no longer writes to the lino.log. That’s normal because the runserver process believes that a socket server is running. Now restart the linod process and verify that runserver is again being logged. The socket file did not exist for some time and now it’s a new socket file, but this doesn’t disturb logging.

In terminal 1:

go noi1e
rm -rf settings/log

If you remove the log directory before stopping the linod, you will get the following exception when linod stops:

FileNotFoundError: [Errno 2] No such file or directory: '.../noi1e/settings/log/lino.log'

Example testing session 2

In terminal 1:

go cms1
LINO_LOGLEVEL=debug pm linod

Expected output:

actors.discover() : registering 135 actors
actors.initialize()
Analyzing Tables...
Analyze 22 slave tables...
Discovering choosers for database fields...
Start task runner using <Logger lino (DEBUG)>...
Start next task runner loop.
Too early to start System task #1 (update_publisher_pages)
Too early to start System task #2 (checkdata)
Let task runner sleep for 4.996284 seconds.
...
(etc until you hit Ctrl-C)

Class reference ¶

class lino.modlib.linod.Procedure¶

A callable function designed to run in background at default interval given by every_unit and every_value.

The default interval can be overridden by SystemTask.

func¶

The function to run as a system task.

Type:: Callable[[BaseRequest], None]

every_unit¶

The default unit of the interval at which the task func will run.

Type:: str

every_value¶

The default value of the interval at which the task func will run.

Type:: int

start_datetime¶

The time at which this task should run first.

Type:: datetime.datetime

run(self, ar)¶

Calls the function stored in func passing ar as a positional argument.

Parameters:: ar – an instance of BaseRequest

class lino.modlib.linod.Procedures¶: The choicelist of background procedures available in this application.

class lino.modlib.linod.LogLevels¶

A choicelist of logging levels available in this application.

See Logging levels

class lino.modlib.linod.SystemTask¶

Django model used to represent a background task.

Specifies how to run a given Procedure on this site.

Inherits from Sequenced, RecurrenceSet and Runnable.

start_datetime¶

Tells at what time exactly this job started.

Type:: datetime.datetime

run(self, ar, lgr=None) → Job¶

Performs a routine job.

Calls self.procedure.run.
Cancels the rule in case of a failure.
Creates an instance of Job

Parameters:

ar – An instance of BaseRequest
lgr – Logger obtained by calling logging.getLogger.

Returns:

An instance of Job.

class lino.modlib.linod.SystemTasks¶: The default table for the SystemTask model.

class lino.modlib.linod.Runnable¶

Model mixin used by SystemTask and other models.

The only known other model that subclasses this mixin is lino_xl.lib.invoicing.Task.

It defines two actions:

run_now¶: Explicitly request to tun this task as soon as possible.

cancel_run¶: Cancel the explicit request to tun this task as soon as possible.

It defines the following database fields:

procedure¶

The background procedure to run in this task.

This points to an instance of Procedure.

requested_at¶: The timestamp when a user has explicitly requested to run this task.

last_start_time¶: The timestamp when this task has started running in the task runner.

last_end_time¶: The timestamp when this task has finished running in the task runner.

message¶: Stores information about the job, mostly logs.

disabled¶

Tells whether the task should be ignored.

Lino sets this to True when the tasks fails and raises an exception. But it can also be checked by an end user in the web interface.

log_level¶

The logging level to apply when running this task.

See LogLevels.

linod: The Lino daemon¶

`linod`: The Lino daemon¶