Welcome | Get started | Dive | Contribute | Topics | Reference | Changes | More

linod: The Lino daemon

This page documents the linod plugin for developers. We assume that you have read the end-user docs.

This plugin defines the linod admin command, which is responsible for running the background tasks and the socket-based log server.

The “d” stands for “daemon”, like in sshd, cupsd, systemd and other background processes on a Linux system.

When linod.use_channels is True, this plugin uses channels to provide an ASGI application, and the linod command then includes Channels’ runworker command.

Lines starting with >>> in this document are code snippets that get tested as part of our development workflow.

>>> import lino
>>> lino.startup('lino_book.projects.noi1e.settings.demo')
>>> from lino.api.doctest import *

Usage

Other plugins can register a background task using the dd.api.schedule_often() or dd.api.schedule_daily() decorators. For example (taken from lino.modlib.checkdata):

@dd.schedule_daily()
def checkdata(ar):
    """Run all data checkers."""
    check_data(fix=False)

The code for the example above should be in your plugin’s models.py modules.

Plugin configuration

linod.use_channels

Whether to use channels and daphne to run in asynchronous mode.

linod.background_sleep_time

How many seconds the background task runner should sleep when there is nothing to do.

Background procedures

>>> rt.show(linod.Procedures)
============================== ============================== ============================== ================== ================================
 value                          name                           text                           Task class         Suggested recurrency
------------------------------ ------------------------------ ------------------------------ ------------------ --------------------------------
 event_notification_scheduler   event_notification_scheduler   event_notification_scheduler   linod.SystemTask   every=300, every_unit=secondly
 generate_calendar_entries      generate_calendar_entries      generate_calendar_entries      linod.SystemTask   every=1, every_unit=daily
 delete_older_changes           delete_older_changes           delete_older_changes           linod.SystemTask   every=1, every_unit=daily
 checksummaries                 checksummaries                 checksummaries                 linod.SystemTask   every=1, every_unit=daily
 checkdata                      checkdata                      checkdata                      linod.SystemTask   every=1, every_unit=daily
 send_pending_emails_often      send_pending_emails_often      send_pending_emails_often      linod.SystemTask   every=10, every_unit=secondly
 send_pending_emails_daily      send_pending_emails_daily      send_pending_emails_daily      linod.SystemTask   every=1, every_unit=daily
 clear_seen_messages            clear_seen_messages            clear_seen_messages            linod.SystemTask   every=1, every_unit=daily
 run_invoicing_tasks            run_invoicing_tasks            run_invoicing_tasks            invoicing.Task     every=1, every_unit=daily
 update_publisher_pages         update_publisher_pages         update_publisher_pages         linod.SystemTask   every=1, every_unit=daily
============================== ============================== ============================== ================== ================================

While the procedures are in a choicelist (i.e. end users cannot edit them), the list of system tasks is configurable. The default situation is that every procedure has created one system task:

Logging levels

>>> rt.show(linod.LogLevels)
========== ========== ===============
 value      text       Numeric value
---------- ---------- ---------------
 DEBUG      DEBUG      10
 INFO       INFO       20
 WARNING    WARNING    30
 ERROR      ERROR      40
 CRITICAL   CRITICAL   50
========== ========== ===============

DEBUG means to include detailed debug messages. You should not set this for a longer period on a production site because it bloats the log files.

INFO means to show informative messages.

WARNING is the recommended value for most tasks. Only warnings and error messages are logged.

The levels ERROR and CRITICAL (log only errors and critical messages) exist only for exceptional situations. You should probably not use them.

System tasks

>>> rt.show(linod.SystemTasks) 
===== ============================== =============== ========== ======================= ==============================
 No.   Name                           Logging level   Disabled   Status                  Background procedure
----- ------------------------------ --------------- ---------- ----------------------- ------------------------------
 1     event_notification_scheduler   WARNING         No         Scheduled to run asap   event_notification_scheduler
 2     generate_calendar_entries      INFO            No         Scheduled to run asap   generate_calendar_entries
 3     delete_older_changes           INFO            No         Scheduled to run asap   delete_older_changes
 4     checksummaries                 INFO            No         Scheduled to run asap   checksummaries
 5     checkdata                      INFO            No         Scheduled to run asap   checkdata
 6     send_pending_emails_often      WARNING         No         Scheduled to run asap   send_pending_emails_often
 7     send_pending_emails_daily      INFO            No         Scheduled to run asap   send_pending_emails_daily
 8     clear_seen_messages            INFO            No         Scheduled to run asap   clear_seen_messages
 9     update_publisher_pages         INFO            No         Scheduled to run asap   update_publisher_pages
===== ============================== =============== ========== ======================= ==============================
>>> rt.show(invoicing.Task) 
===== ============ ============================= ===================================== =============== ========== =========== =======================
 No.   Author       Target journal                Invoice generators                    Logging level   Disabled   When        Status
----- ------------ ----------------------------- ------------------------------------- --------------- ---------- ----------- -----------------------
 1     Robin Rood   Service reports (SRV)         working.Session                       INFO            No         Every day   Scheduled to run asap
 2     Robin Rood   Sales invoices (SLS)          storage.Filler, trading.InvoiceItem   INFO            No         Every day   Scheduled to run asap
 3     Robin Rood   Subscription invoices (SUB)   subscriptions.SubscriptionPeriod      INFO            No         Every day   Scheduled to run asap
===== ============ ============================= ===================================== =============== ========== =========== =======================

Dependencies

When linod.use_channels is True, the lino.modlib.linod plugin requires the django-channels and channels-redis Python packages to be installed, as well as a running redis-server.

To install redis on a Debian-based Linux distribution, run the following command as root:

$ apt update
$ apt install redis

To install the required Python packages, run the following command after activating your Python environment:

$ pm install
>>> list(dd.plugins.linod.get_requirements(settings.SITE))
['channels', 'channels_redis', 'daphne']

Usage for developers

To run the Lino daemon in a development environment, run pm linod in your project directory:

$ cd ~/lino/lino_local/mysite
$ pm linod

This process will run as long as you don’t kill it, e.g. until you hit Ctrl-C.

Another way to kill the linod process is using the kill command:

$ kill -s SIGTERM 123456

If you kill linod with another signal than SIGTERM, Lino will not run it shutdown method, which is responsible e.g. for removing the socket file of the log server.

You may change the logging level by setting LINO_LOGLEVEL:

$ LINO_LOGLEVEL=debug pm linod

Testing instructions for developers

General remarks:

  • The following demo projects are useful when testing linod:

  • linod.use_channels changes the way pm linod works.

  • For testing the log server you need to create a log directory, and don’t forget to remove it after your tests because a log directory causes different output for certain commands and would cause the unit test suite to fail if you forget to delete it.

  • When you start pm runserver before pm linod, runserver will write directly to the lino.log file because there is no socket file. Two processes writing to the same file is likely to cause unpredictable results.

  • You can set Site.log_each_action_request to True

Example testing session 1

In terminal 1:

go noi1e
mkdir settings/log
LINO_LOGLEVEL=debug pm linod

In terminal 2:

go noi1r
pm runserver

In your browser: sign in as robin, go to Configure ‣ System ‣ System tasks, click “Run now” on one of them. The linod process in terminal 1 should run the task.

In terminal 1, hit Ctrl-C to stop the linod. Then do something in the browser and verify that runserver no longer writes to the lino.log. That’s normal because the runserver process believes that a socket server is running. Now restart the linod process and verify that runserver is again being logged. The socket file did not exist for some time and now it’s a new socket file, but this doesn’t disturb logging.

In terminal 1:

go noi1e
rm -rf settings/log

If you remove the log directory before stopping the linod, you will get the following exception when linod stops:

FileNotFoundError: [Errno 2] No such file or directory: '.../noi1e/settings/log/lino.log'

Example testing session 2

In terminal 1:

go cms1
LINO_LOGLEVEL=debug pm linod

Expected output:

actors.discover() : registering 135 actors
actors.initialize()
Analyzing Tables...
Analyze 22 slave tables...
Discovering choosers for database fields...
No log server because there is no directory .../lino_book/projects/cms1/log.
Start task runner using <Logger lino (DEBUG)>...
Start next task runner loop.
Too early to start System task #1 (update_publisher_pages)
Too early to start System task #2 (checkdata)
Let task runner sleep for 4.996284 seconds.
...
(etc until you hit Ctrl-C)

Class reference

class lino.modlib.linod.Procedure

A callable function designed to run in background at default interval given by every_unit and every_value.

The default interval can be overridden by SystemTask.

func

The function to run as a system task.

Type:

Callable[[BaseRequest], None]

every_unit

The default unit of the interval at which the task func will run.

Type:

str

every_value

The default value of the interval at which the task func will run.

Type:

int

start_datetime

The time at which this task should run first.

Type:

datetime.datetime

run(self, ar)

Calls the function stored in func passing ar as a positional argument.

Parameters:

ar – an instance of BaseRequest

class lino.modlib.linod.Procedures

The choicelist of background procedures available in this application.

class lino.modlib.linod.LogLevels

A choicelist of logging levels available in this application.

See Logging levels

class lino.modlib.linod.SystemTask

Django model used to represent a background task.

Overrides the recurrent rule of a Procedure.

A subclass of Sequenced and RecurrenceSet.

procedure

Pointer to an instance of Procedure.

start_datetime

Tells at what time exactly this job started.

Type:

datetime.datetime

message

Stores information about the job, mostly logs.

disabled

Tells whether the task should be ignored.

Lino sets this to True when the tasks fails and raises an exception. But it can also be checked by an end user in the web interface.

log_level

The logging level to apply when running this task.

See LogLevels.

run(self, ar, lgr=None) Job

Performs a routine job.

  • Calls self.procedure.run.

  • Cancels the rule in case of a failure.

  • Creates an instance of Job

Parameters:
  • ar – An instance of BaseRequest

  • lgr – Logger obtained by calling logging.getLogger.

Returns:

An instance of Job.

class lino.modlib.linod.SystemTasks

The default actor for the SystemTask model.

Don’t read this

>>> from logging import getLevelName
>>> from asgiref.sync import async_to_sync
>>> bt = linod.SystemTask.objects.get(procedure=linod.Procedures.update_publisher_pages)
>>> bt.status
'Scheduled to run asap'
>>> ar = rt.login("robin")
>>> print(getLevelName(ar.logger.level))
INFO
>>> ar.logger.setLevel("DEBUG")
>>> print(getLevelName(ar.logger.level))
DEBUG
>>> ar.logger.handlers
[<StreamHandler (INFO)>, <AdminEmailHandler (ERROR)>]
>>> [getLevelName(h.level) for h in ar.logger.handlers]
['INFO', 'ERROR']
>>> ar.logger.handlers[0].setLevel("DEBUG")
>>> async_to_sync(bt.start_task)(ar)
Start System task #9 (update_publisher_pages) with logging level INFO
Update publisher pages...
72 pages have been updated.
Successfully terminated System task #9 (update_publisher_pages)
>>> bt.disabled
False
>>> bt.status  
'Scheduled to run at ... (... from now)'
>>> bt = linod.SystemTask.objects.get(procedure=linod.Procedures.delete_older_changes)
>>> bt.status
'Scheduled to run asap'
>>> async_to_sync(bt.start_task)(ar)
Start System task #3 (delete_older_changes) with logging level INFO
Successfully terminated System task #3 (delete_older_changes)
>>> bt.disabled
False
>>> bt.status  
'Scheduled to run at ... (... from now)'
>>> bt.run_now.run_from_ui(ar)
>>> bt.message  
'Robin Rood requested to run this task at ....'
>>> bt.status  
'Scheduled to run asap'

Restore database state:

>>> for obj in linod.SystemTask.objects.all():
...     obj.last_start_time = None
...     obj.disabled = False
...     obj.save()