Welcome | Get started | Dive into Lino | Contribute | Topics | Reference | More

linod: The Lino daemon

We assume that you have read linod : Background tasks.

It defines the linod admin command, which is responsible for running the background tasks.

The "d" stands for "daemon", like in sshd, cupsd, systemd and other background processes on a Linux system.

When linod.use_channels is True, this plugin uses channels to provide an ASGI application, and the linod command then includes Channels' runworker command and a socket-based log server to avoid concurrency issues when multiple processes log to the same lino.log file.

Usage

Other plugins can register a background task using the dd.api.schedule_often() or dd.api.schedule_daily() decorators. For example (taken from lino.modlib.checkdata):

@dd.schedule_daily()
def checkdata(ar):
    """Run all data checkers."""
    check_data(fix=False)

The code for the example above should be in your plugin's models.py modules.

>>> import lino
>>> lino.startup('lino_book.projects.noi1e.settings.demo')
>>> from lino.api.doctest import *

Background procedures

>>> rt.show(linod.Procedures)
============================== ============================== ============================== ================== ================================
 value                          name                           text                           Task class         Suggested recurrency
------------------------------ ------------------------------ ------------------------------ ------------------ --------------------------------
 event_notification_scheduler   event_notification_scheduler   event_notification_scheduler   linod.SystemTask   every=300, every_unit=secondly
 generate_calendar_entries      generate_calendar_entries      generate_calendar_entries      linod.SystemTask   every=1, every_unit=daily
 delete_older_changes           delete_older_changes           delete_older_changes           linod.SystemTask   every=1, every_unit=daily
 checksummaries                 checksummaries                 checksummaries                 linod.SystemTask   every=1, every_unit=daily
 checkdata                      checkdata                      checkdata                      linod.SystemTask   every=1, every_unit=daily
 send_pending_emails_often      send_pending_emails_often      send_pending_emails_often      linod.SystemTask   every=10, every_unit=secondly
 send_pending_emails_daily      send_pending_emails_daily      send_pending_emails_daily      linod.SystemTask   every=1, every_unit=daily
 clear_seen_messages            clear_seen_messages            clear_seen_messages            linod.SystemTask   every=1, every_unit=daily
 run_invoicing_tasks            run_invoicing_tasks            run_invoicing_tasks            invoicing.Task     every=1, every_unit=daily
 update_publisher_pages         update_publisher_pages         update_publisher_pages         linod.SystemTask   every=1, every_unit=daily
============================== ============================== ============================== ================== ================================

While the procedures are in a choicelist (i.e. end users cannot edit them), the list of system tasks is configurable. The default situation is that every procedure has created one system task:

Logging levels

>>> rt.show(linod.LogLevels)
========== ========== ===============
 value      text       Numeric value
---------- ---------- ---------------
 DEBUG      DEBUG      10
 INFO       INFO       20
 WARNING    WARNING    30
 ERROR      ERROR      40
 CRITICAL   CRITICAL   50
========== ========== ===============

DEBUG means to include detailed debug messages. You should not set this for a longer period on a production site because it bloats the log files.

INFO means to show informative messages.

WARNING is the recommended value for most tasks. Only warnings and error messages are logged.

The levels ERROR and CRITICAL (log only errors and critical messages) exist only for exceptional situations. You should probably not use them.

System tasks

>>> rt.show(linod.SystemTasks) 
===== ============================== =============== ========== =======================
 No.   Background procedure           Logging level   Disabled   Status
----- ------------------------------ --------------- ---------- -----------------------
 1     event_notification_scheduler   WARNING         No         Scheduled to run asap
 2     generate_calendar_entries      INFO            No         Scheduled to run asap
 3     delete_older_changes           INFO            No         Scheduled to run asap
 4     checksummaries                 INFO            No         Scheduled to run asap
 5     checkdata                      INFO            No         Scheduled to run asap
 6     send_pending_emails_often      WARNING         No         Scheduled to run asap
 7     send_pending_emails_daily      INFO            No         Scheduled to run asap
 8     clear_seen_messages            INFO            No         Scheduled to run asap
 9     update_publisher_pages         INFO            No         Scheduled to run asap
===== ============================== =============== ========== =======================

>>> rt.show(invoicing.Task) 
===== ============ ============================= ===================================== =============== ========== =========== =======================
 No.   Author       Target journal                Invoice generators                    Logging level   Disabled   When        Status
----- ------------ ----------------------------- ------------------------------------- --------------- ---------- ----------- -----------------------
 1     Robin Rood   Service reports (SRV)         working.Session                       INFO            No         Every day   Scheduled to run asap
 2     Robin Rood   Sales invoices (SLS)          storage.Filler, trading.InvoiceItem   INFO            No         Every day   Scheduled to run asap
 3     Robin Rood   Subscription invoices (SUB)   subscriptions.SubscriptionPeriod      INFO            No         Every day   Scheduled to run asap
===== ============ ============================= ===================================== =============== ========== =========== =======================

Dependencies

When linod.use_channels is True, the lino.modlib.linod plugin requires the django-channels and channels-redis Python packages to be installed, as well as a running redis-server.

To install redis on a Debian-based Linux distribution, run the following command as root:

$ apt update
$ apt install redis

To install the required Python packages, run the following command after activating your Python environment:

$ pm install

>>> list(dd.plugins.linod.get_requirements(settings.SITE))
['channels', 'channels_redis', 'daphne']

Usage for developers

To run the Lino daemon in a development environment, run pm linod in your project directory:

$ cd ~/lino/lino_local/mysite
$ pm linod

This process will run as long as you don't kill it, e.g. until you hit Ctrl-C.

Another way to kill the linod process is using the kill command:

$ kill -s SIGTERM 123456

If you kill linod with another signal than SIGTERM, Lino will not run it shutdown method, which is responsible e.g. for removing the socket file of the log server.

You may change the logging level by setting LINO_LOGLEVEL:

$ LINO_LOGLEVEL=debug pm linod

Testing instructions for developers

  • Two demo projects are useful when testing linod because they have some background tasks:

    • cms1 has linod.use_channels set to False

    • noi1r has linod.use_channels set to True.

  • chatter : an instant messaging system has no background tasks, but it has linod.use_channels set to True.

  • linod.use_channels changes the way pm linod works. Without channels there is no log server, but background tasks are processed as with channels.

  • For testing the log server you need to create a log directory, and don't forget to remove it after your tests, because a log directory causes different output for certain commands and thus the unit test suite would fail if you forget to delete the.m

  • When you start pm runserver before pm linod, runserver will write directly to the lino.log file

  • You can set Site.log_each_action_request to True

Example testing session

In terminal 1:

go noi1e
mkdir settings/log
LINO_LOGLEVEL=debug pm linod

In terminal 2:

go noi1r
pm runserver

In your browser: sign in as robin, go to Configure ‣ System ‣ System tasks, click "Run now" on one of them. The linod process in terminal 1 should run the task.

In terminal 1, hit Ctrl-C to stop the linod. Then do something in the browser and verify that runserver no longer writes to the lino.log. That's normal because the runserver process believes that a socket server is running. Now restart the linod process and verify that runserver is again being logged. The socket file did not exist for some time and now it's a new socket file, but this doesn't disturb logging.

In terminal 1:

go noi1e
rm -rf settings/log

If you remove the log directory before stopping the linod, you will get the following exception when linod stops:

FileNotFoundError: [Errno 2] No such file or directory: '.../noi1e/settings/log/lino.log'

Class reference

class lino.modlib.linod.Procedure

A callable function designed to run in background at default interval given by every_unit and every_value.

The default interval can be overridden by SystemTask.

func

The function to run as a system task.

Type:

Callable[[BaseRequest], None]

every_unit

The default unit of the interval at which the task func will run.

Type:

str

every_value

The default value of the interval at which the task func will run.

Type:

int

start_datetime

The time at which this task should run first.

Type:

datetime.datetime

run(self, ar)

Calls the function stored in func passing ar as a positional argument.

Parameters:

ar -- an instance of BaseRequest

class lino.modlib.linod.Procedures

The choicelist of background procedures available in this application.

class lino.modlib.linod.LogLevels

A choicelist of logging levels available in this application.

See Logging levels

class lino.modlib.linod.SystemTask

Django model used to represent a background task.

Overrides the recurrent rule of a Procedure.

A subclass of Sequenced and RecurrenceSet.

procedure

Pointer to an instance of Procedure.

start_datetime

Tells at what time exactly this job started.

Type:

datetime.datetime

message

Stores information about the job, mostly logs.

disabled

Tells whether the task should be ignored.

Lino sets this to True when the tasks fails and raises an exception. But it can also be checked by an end user in the web interface.

log_level

The logging level to apply when running this task.

See LogLevels.

run(self, ar, lgr=None) → Job

Performs a routine job.

  • Calls self.procedure.run.

  • Cancels the rule in case of a failure.

  • Creates an instance of Job

Parameters:

  • ar -- An instance of BaseRequest

  • lgr -- Logger obtained by calling logging.getLogger.

Returns:

An instance of Job.

class lino.modlib.linod.SystemTasks

The default actor for the SystemTask model.

Don't read this

>>> from asgiref.sync import async_to_sync
>>> bt = linod.SystemTask.objects.get(procedure=linod.Procedures.update_publisher_pages)
>>> bt.status
'Scheduled to run asap'
>>> ar = rt.login("robin")
>>> async_to_sync(bt.start_task)(ar)
Start System task #9 update_publisher_pages with logging level INFO
Update published pages...
72 pages have been updated.
Successfully terminated System task #9 update_publisher_pages
>>> bt.disabled
False
>>> bt.status  
'Scheduled to run at ... (... from now)'

>>> bt = linod.SystemTask.objects.get(procedure=linod.Procedures.delete_older_changes)
>>> bt.status
'Scheduled to run asap'
>>> ar = rt.login("robin")
>>> async_to_sync(bt.start_task)(ar)
Start System task #3 delete_older_changes with logging level INFO
Successfully terminated System task #3 delete_older_changes
>>> bt.disabled
False
>>> bt.status  
'Scheduled to run at ... (... from now)'

>>> bt.run_now.run_from_ui(ar)
>>> bt.message  
'Robin Rood requested to run this task at ....'

>>> bt.status  
'Scheduled to run asap'

Restore database state:

>>> for obj in linod.SystemTask.objects.all():
...     obj.last_start_time = None
...     obj.disabled = False
...     obj.save()