`memo` : The memo parser¶

The lino.modlib.memo plugin adds application-specific markup to text fields. One facet of this plugin is a simple built-in memo markup format, another facet are suggesters. A usage example is documented in memo in Noi.

This page contains code snippets (lines starting with >>>), which are being tested during our development workflow. The following snippet initializes the demo project used throughout this page.

>>> from lino_book.projects.noi1r.startup import *

Glossary ¶

suggester¶: A suggester is when you define that a “trigger text” will pop up a list of suggestions for auto-completion. For example # commonly refers to a topic or a ticket, or @ refers to another site user.
memo markup¶: A simple markup language that replaces “memo commands” (expressions between [square brackets]) by their result. See Built-in memo commands.

Basic usage ¶

The lino.modlib.memo.utils.Parser is a simple markup parser that expands “commands” found in an input string to produce a resulting output string. Commands are in the form [COMMAND ARGS]. The caller defines itself all commands, there are no predefined commands.

Let’s instantiate parser:

>>> from lino.modlib.memo.utils import Parser
>>> p = Parser(error_tolerance=2, with_eval=True)

We declare a command handler function url2html and register it:

>>> def url2html(parser, s, cmdname, mentions, context):
...     print("[DEBUG] url2html() got %r" % s)
...     if not s: return "XXX"
...     url, text = s.split(None,1)
...     return '<a href="%s">%s</a>' % (url,text)
>>> p.register_command('url', url2html)

The intended usage of our example handler is [url URL TEXT], where URL is the URL to link to, and TEXT is the label of the link:

>>> print(p.parse('This is a [url http://xyz.com test].'))
[DEBUG] url2html() got 'http://xyz.com test'
This is a <a href="http://xyz.com">test</a>.

A command handler will be called with one parameter: the portion of text between the COMMAND and the closing square bracket. Not including the whitespace after the COMMAND. It must return the text that is to replace the [COMMAND ARGS] fragment. It is responsible for parsing the text that it receives as parameter.

If an exception occurs during the command handler, the final exception message is inserted into the result.

To demonstrate this, our example implementation has a bug, it doesn’t support the case of having only a URL without TEXT:

>>> print(p.parse('This is a [url http://xyz.com].'))
[DEBUG] url2html() got 'http://xyz.com'
This is a [ERROR ... in ...'[url http://xyz.com]' at position 10-30].

We use an ellipsis in above code because the error message varies with Python versions.

Newlines preceded by a backslash will be removed before the command handler is called:

>>> print(p.parse('''This is [url http://xy\
... z.com another test].'''))
[DEBUG] url2html() got 'http://xyz.com another test'
This is <a href="http://xyz.com">another test</a>.

The whitespace between the COMMAND and ARGS can be any whitespace, including newlines:

>>> print(p.parse('''This is a [url
... http://xyz.com test].'''))
[DEBUG] url2html() got 'http://xyz.com test'
This is a <a href="http://xyz.com">test</a>.

The ARGS part is optional (it’s up to the command handler to react accordingly, our handler function returns XXX in that case):

>>> print(p.parse('''This is a [url] test.'''))
[DEBUG] url2html() got ''
This is a XXX test.

The ARGS part may contain pairs of square brackets:

>>> print(p.parse('''This is a [url
... http://xyz.com test with [more] brackets].'''))
[DEBUG] url2html() got 'http://xyz.com test with [more] brackets'
This is a <a href="http://xyz.com">test with [more] brackets</a>.

Fragments of text between brackets that do not match any registered command will be left unchanged:

>>> print(p.parse('''This is a [1] test.'''))
This is a [1] test.

>>> print(p.parse('''This is a [foo bar] test.'''))
This is a [foo bar] test.

>>> print(p.parse('''Text with only [opening square bracket.'''))
Text with only [opening square bracket.

Special handling ¶

Leading and trailing spaces are always removed from command text:

>>> print(p.parse("[url http://example.com Trailing space  ]."))
[DEBUG] url2html() got 'http://example.com Trailing space'
<a href="http://example.com">Trailing space</a>.

>>> print(p.parse("[url http://example.com   Leading space]."))
[DEBUG] url2html() got 'http://example.com   Leading space'
<a href="http://example.com">Leading space</a>.

Non-breaking and zero-width spaces are treated like normal spaces:

>>> print(p.parse(u"[url\u00A0http://example.com example.com]."))
[DEBUG] url2html() got 'http://example.com example.com'
<a href="http://example.com">example.com</a>.

>>> print(p.parse(u"[url \u200bhttp://example.com example.com]."))
[DEBUG] url2html() got 'http://example.com example.com'
<a href="http://example.com">example.com</a>.

>>> print(p.parse(u"[url&nbsp;http://example.com example.com]."))
[DEBUG] url2html() got 'http://example.com example.com'
<a href="http://example.com">example.com</a>.

Limits ¶

A single closing square bracket as part of ARGS will not produce the desired result:

>>> print(p.parse(r'''This is a [url
... http://xyz.com The character "\]"].'''))
[DEBUG] url2html() got 'http://xyz.com The character "\\'
This is a <a href="http://xyz.com">The character "\</a>"].

Execution flow statements like [if …] and [endif …] or [for ...] and [endfor ...] would be nice.

The `[=expression]` form ¶

>>> print(p.parse('''<ul>[="".join(['<li>%s</li>' % (i+1) for i in range(5)])]</ul>'''))
<ul><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li></ul>

You can specify a run-time context:

>>> ctx = { 'a': 3 }
>>> print(p.parse('''\
... The answer is [=a*a*5-a].''', context=ctx))
The answer is 42.

The `Previewable` mixin ¶

The Previewable model mixin adds three database fields body, body_short_preview and body_full_preview. The two preview fields contain the parsed version of the body, they are read-only and get updated automatically when the body is updated. body_short_preview contains only the first paragraph and a “more” indication if the full preview has more. See also truncate_comment().

This mixin is used for example by lino.modlib.comments.Comment, lino.modlib.publisher.Page and lino_xl.lib.blog.Entry.

>>> from lino.modlib.memo.mixins import Previewable
>>> print("\n".join([full_model_name(m) for m in rt.models_by_base(Previewable)]))
comments.Comment

>>> def test(body):
...     short, full = comments.Comment().parse_previews(body, "body")
...     print(short)
...     print("------")
...     print(full)

>>> test("Foo bar baz")
Foo bar baz
------
Foo bar baz

>>> test("<p>Foo</p><p>bar baz</p>")
Foo bar baz
------
<p>Foo</p><p>bar baz</p>

>>> test("Foo\n\nbar baz")
Foo

bar baz
------
Foo

bar baz

Built-in memo commands ¶

ref¶

Insert a clickable reference to a database row.

Syntax: [ref <model>:<key>] where <model> is the memo name of a model and <key> the primary key or a natural key to identify a database row.

Examples:

>>> print(dd.plugins.memo.parser.parse("[ref ticket:1]"))
<a href="/#/api/tickets/Tickets/1" title="Föö fails to bar when baz" style="text-decoration:none">#1</a>

>>> print(dd.plugins.memo.parser.parse("[ref foo]"))
[ERROR Malformed reference 'foo' (must be <modelspec>:<pk>) in '[ref foo]' at position 0-9]

include¶

Insert a summary of a database row.

Syntax: [include <model>:<key>] where <model> is the memo name of a model and <key> the primary key or a natural key to identify a database row.

Examples:

>>> dd.plugins.memo.parser.error_tolerance = 0
>>> print(dd.plugins.memo.parser.parse("[include ticket:1]"))
<a href="/#/api/tickets/Tickets/1" style="text-decoration:none">#1 (Föö fails to
bar when baz)</a> ( <span class="l-text-prioritaire">50</span> by <a
href="/#/api/users/AllUsers/6" style="text-decoration:none">Luc</a> in <a
href="/#/api/groups/Groups/2" style="text-decoration:none">Managers</a>)

>>> print(dd.plugins.memo.parser.parse("[include upload:1]"))
<a href="/#/api/uploads/Uploads/1" target="_blank"><img
src="/media/volumes/screenshots/Screenshot_20250124_104858.png"
title="Screenshot 20250124 104858.png"
style="max-height:10em;max-width:10em;width:auto;height:auto;padding:1pt;"/></a>

Only models inheriting from MemoReferrable are candidate to get referred to by a ref or include memo command.

>>> from lino.modlib.memo.mixins import MemoReferrable
>>> pprint({m.memo_command: m for m in rt.models_by_base(MemoReferrable) if m.memo_command})
{'comment': <class 'lino.modlib.comments.models.Comment'>,
 'company': <class 'lino_xl.lib.contacts.models.Company'>,
 'group': <class 'lino_noi.lib.groups.models.Group'>,
 'person': <class 'lino_xl.lib.contacts.models.Person'>,
 'product': <class 'lino_xl.lib.products.models.Product'>,
 'ticket': <class 'lino_noi.lib.tickets.models.Ticket'>,
 'upload': <class 'lino.modlib.uploads.models.Upload'>}

url¶

20250605: This command doesn’t exist any more, see Handling URLs in text fields.

Insert a link to an external web page. The first argument is the URL (mandatory). If no other argument is given, the URL is used as text. Otherwise the remaining text is used as the link text.

The link will always open in a new window (target="_blank")

Usage examples:

[url http://www.example.com]
[url http://www.example.com example]
[url http://www.example.com another example]

py¶

20260324: This command exists only when you set with_eval to True, which is a security risk because end users might enter Python expressions that access your server’s file system.

Refer to a Python object. This is not being used on the field. A fundamental problem is that it works only in the currently running Python environment.

Usage examples:

[py lino]
[py lino.modlib.memo.utils]
[py lino_xl.lib.tickets.models.Ticket]
[py lino_xl.lib.tickets.models.Ticket tickets.Ticket]

The global memo parser contains two “built-in commands”:

>>> p = dd.plugins.memo.parser

The py command is disabled since 20240920 because I don’t know anybody who used it (except myself a few times for testing it) and because it requires SETUP_INFO, which has an uncertain future.

>>> print(p.parse("[py lino]."))
<a href="https://gitlab.com/lino-framework/lino/blob/master/lino/__init__.py" target="_blank">lino</a>.

>>> print(p.parse("[py lino_xl.lib.tickets.models.Ticket]."))
<a href="https://gitlab.com/lino-framework/xl/blob/master/lino_xl/lib/tickets/models.py" target="_blank">lino_xl.lib.tickets.models.Ticket</a>.

>>> print(p.parse("[py lino_xl.lib.tickets.models.Ticket.foo]."))
<a href="Error in Python code (type object 'Ticket' has no attribute 'foo')" target="_blank">lino_xl.lib.tickets.models.Ticket.foo</a>.

>>> print(p.parse("[py lino_xl.lib.tickets.models.Ticket Ticket]."))
<a href="https://gitlab.com/lino-framework/xl/blob/master/lino_xl/lib/tickets/models.py" target="_blank">Ticket</a>.

Non-breaking spaces are removed from command text:

>>> print(p.parse("[py lino]."))
<a href="https://gitlab.com/lino-framework/lino/blob/master/lino/__init__.py" target="_blank">lino</a>.

Configuration ¶

lino.modlib.memo.front_end¶

The front end to use when rendering memo commands.

If this is None, Lino will use the primary front end of this site (lino.core.site.Site.primary_front_end).

Used on sites that are available through more than one web front ends. The server administrator must then decide which front end is the primary one.

lino.modlib.memo.short_preview_length¶

How many characters to show in the short preview.

Default is 300, cms sets it to 1200.

>>> dd.plugins.memo.short_preview_length
300

lino.modlib.memo.error_tolerance¶

How tolerant to be with exceptions during evaluation of a memo command.

This will be used as the error_tolerance for :data: parser

>>> dd.plugins.memo.error_tolerance
2

lino.modlib.memo.with_eval¶

>>> dd.plugins.memo.with_eval
False

lino.modlib.memo.short_preview_image_height¶: This setting was removed 2025-02-09. Default value is 8em.

lino.modlib.memo.parser¶

The parser used by Lino for rendering memo commands.

Do not configure this attribute yourself. It is set during startup. It is an instance of lino.modlib.memo.utils.Parser.

>>> dd.plugins.memo.parser
<lino.modlib.memo.utils.Parser object at ...>

Mentions ¶

>>> obj = comments.Comment.objects.filter(body__contains="[person 13]").first()
>>> print(obj.body)
<p>This is a comment about [person 15] and [person 13].</p>

>>> print(obj.body_short_preview)
This is a comment about <a href="/api/contacts/Persons/15"
style="text-decoration:none">Mr Hans Altenberg</a> and <a
href="/api/contacts/Persons/13" style="text-decoration:none">Mr Andreas
Arens</a>.

>>> hans = contacts.Person.objects.get(pk=13)
>>> rt.show(memo.MentionsByTarget, hans)
`⏏ <…>`__ | `Comment #122 <…>`__

>>> rt.show(memo.Mentions)
====================== ===============================================
 Referrer               Target
---------------------- -----------------------------------------------
 `Comment #1 <…>`__     `Screenshot 20250124 104858.png <…>`__
 `Comment #2 <…>`__     `screenshot-toolbar.png <…>`__
 `Comment #11 <…>`__    `Rumma & Ko OÜ <…>`__
 `Comment #11 <…>`__    `Bäckerei Ausdemwald <…>`__
 `Comment #12 <…>`__    `Bäckerei Mießen <…>`__
 `Comment #12 <…>`__    `Bäckerei Schmitz <…>`__
 `Comment #21 <…>`__    `Screenshot 20250124 104858.png <…>`__
 `Comment #22 <…>`__    `Garage Mergelsberg <…>`__
 `Comment #22 <…>`__    `Donderweer BV <…>`__
 `Comment #31 <…>`__    `screenshot-toolbar.png <…>`__
 `Comment #32 <…>`__    `Screenshot 20250124 104858.png <…>`__
 `Comment #41 <…>`__    `Hans Flott & Co <…>`__
 `Comment #41 <…>`__    `Van Achter NV <…>`__
 `Comment #42 <…>`__    `Bernd Brechts Bücherladen <…>`__
 `Comment #42 <…>`__    `Reinhards Baumschule <…>`__
 ...
 `Comment #472 <…>`__   `Managers <…>`__
 `Comment #472 <…>`__   `Sales team <…>`__
 `Comment #481 <…>`__   `screenshot-toolbar.png <…>`__
 `Comment #482 <…>`__   `Screenshot 20250124 104858.png <…>`__
 `Comment #491 <…>`__   `screenshot-toolbar.png <…>`__
 `Comment #492 <…>`__   `Screenshot 20250124 104858.png <…>`__
 `Comment #501 <…>`__   `#1 (Föö fails to bar when baz) <…>`__
 `Comment #501 <…>`__   `#2 (Bar is not always baz) <…>`__
 `Comment #502 <…>`__   `#3 (Baz sucks) <…>`__
 `Comment #502 <…>`__   `#4 (Foo and bar don't baz) <…>`__
====================== ===============================================

For API details about editor-dependent parsing and revision comparison, see EditorTypes, FormatPreviewable, CompareRevisions and HistoryAwarePreviewable below.

Technical reference ¶

lino.modlib.memo.truncate_comment(html_str, max_length=300)¶

Return the first paragraph of a string that can be either HTML or plain text, containing at most one paragraph with at most max_p_len characters.

Html_str:: the raw string of html
Max_length:: max number of characters to leave in the paragraph.

See usage examples in comments : The comments framework and memo in Noi and Truncating HTML texts.

lino.modlib.memo.rich_text_to_elems(ar, description)¶: A RichTextField can contain either HTML markup or plain text.

lino.modlib.memo.body_subject_to_elems(ar, title, description)¶

Convert the given title and description to a list of HTML elements.

Used by lino.modlib.notify and by lino_xl.lib.trading

class lino.modlib.memo.Previewable¶

See The Previewable mixin.

Adds three rich text fields and their behaviour.

body¶

An editable text body.

This is a lino.core.fields.PreviewTextField.

body_short_preview¶: A read-only preview of the first paragraph of body.

body_full_preview¶: A read-only full preview of body.

parse_previews(src, field_name, ar=None, mentions=None, save=False, **context)¶

class lino.modlib.memo.BabelPreviewable¶: A Previewable where the body field is a babel field.

class lino.modlib.memo.PreviewableChecker¶: Check for previewables needing update.

class lino.modlib.memo.Mention¶

Django model to represent a mention, i.e. the fact that some memo text of the owner points to some other database row.

owner¶: The database row that mentions another one in a memo text.

source¶: The mentioned database row.

class lino.modlib.memo.MemoReferrable¶

Makes your model referable by a memo command.

Overrides lino.core.model.Model.on_analyze() to call parser.Parser.register_django_model() when memo_command is given.

memo_command¶: The name of the memo command to define.

class lino.modlib.memo.EditorTypes¶

Choice list of source editor types used by FormatPreviewable.

Supported values are html, md, rst and plain.

parse_source_text(source, editor_type, ar=None, mentions=None, context=None)¶

Convert source text to HTML according to editor_type:

html: parses memo markup using the global memo parser
md: renders Markdown
rst: renders ReStructuredText
plain: wraps text into <pre>

Related class: EditorType.

class lino.modlib.memo.FormatPreviewable¶

Mixin for previewable models whose rich text can be written in different source formats.

It introduces an editor_type choice field and controls which fields are parsed with that editor type.

format_fields¶

List of field names where editor_type applies.

When left as None, on_analyze() initializes it automatically from BasePreviewable / BabelPreviewable.

editor_type¶: The selected editor type (EditorTypes) used to parse the source text of fields listed in format_fields.

on_analyze(site)¶: Initializes format_fields when needed.

class lino.modlib.memo.CompareRevisions¶

Action that lets users compare two saved revisions of one rich text field.

Instances are created with a required field_name argument and are usually attached dynamically by HistoryAwarePreviewable as compare_revision_of_<field_name>.

class lino.modlib.memo.HistoryAwarePreviewable¶

Mixin for previewable documents that keep revision history and can render side-by-side (or stacked) diffs between two revisions.

Intended usage:

configure a change watcher for the model (e.g. using lino.modlib.changes.utils.watch_changes())
include revision_compare_helpers() in your plugin head lines to load required CSS/JS
inherit this mixin on the model

on_analyze(site)¶

Adds one CompareRevisions action per editable lino.core.fields.RichTextField that is watched by the model’s change watcher.

Also installs ResetToRevision actions:

one action for all FormatPreviewable.format_fields when the model inherits FormatPreviewable
one action per other watched rich text field not covered by format_fields

get_content_changes(doc, field_name)¶: Returns the ordered changes.Change queryset for a given document field (create/update entries containing that field).

get_content_changes_for_fields(doc, field_names)¶

Returns the ordered changes.Change queryset for multiple fields.

The filter is an OR over the given field names, equivalent to combining Q(changed_fields__contains=<field>) conditions.

revive_content_at_change(doc, field_name, target_change)¶

Rebuilds the raw field content for target_change by replaying history from an empty string.

It takes all changes for that field with timestamp less than or equal to the target change (and id <= target.id when timestamps are equal), orders them ascending by time and id, then applies their stored diffs in forward direction.

compute_content_at_change(doc, field_name, target_change)¶: Rebuilds the raw field content at a given revision by rolling back newer diffs.

get_editor_type_name_at_change(doc, field_name, target_change)¶: Resolves the editor type name recorded in diff metadata (with fallback to current document editor type).

get_parsed_content_at_change(doc, field_name, target_change, ar=None, **context)¶: Returns parsed HTML for the target revision, including memo parsing, marker/toc handling and editor-type specific rendering when applicable.

render_comparison_content(ar, field_name, left_change, right_change)¶: Produces diff HTML and returns (html, max_line_digits).

render_stacked_diff(left_lines, right_lines, left_label, right_label)¶: Produces the stacked/mobile-friendly diff view.

get_diff_container(ar, max_digits=2)¶: Wraps comparison HTML into a responsive container with view-mode toggle controls.

other_revisions_html(ar, field_name)¶: In publisher frontend, renders links to all available revisions for quick navigation.

class lino.modlib.memo.ResetToRevision¶

Action that resets one or more rich text fields of the selected document to a chosen revision.

Instances are created with a required field_names argument (list of strings) and expose a target_revision parameter.

For each configured field, the action restores content at the selected revision (or the closest earlier available state for that field) using HistoryAwarePreviewable.revive_content_at_change().

The `[url]` memo command has been removed ¶

When we introduced the clickable URLs feature, we decided to remove the old [url] memo command because it wasn’t used very much and because keeping it would make things complicated.

To remove [url] memo tags from existing data, the site maintainer can run the removeurls admin command.

removeurls¶: Convert [url] memo commands in the text fields of this database into <a href> tags.

>>> from atelier.sheller import Sheller
>>> shell = Sheller(settings.SITE.project_dir)

>>> shell('python manage.py removeurls --help')
...
usage: manage.py removeurls [-h] [-b] [--version] [-v {0,1,2,3}]
                            [--settings SETTINGS] [--pythonpath PYTHONPATH]
                            [--traceback] [--no-color] [--force-color]
                            [--skip-checks]

Convert [url] memo commands in the text fields of this database into <a href>
tags.

options:
  -h, --help            show this help message and exit
  -b, --batch, --noinput
                        Do not prompt for input of any kind.
  --version             Show program's version number and exit.
  -v..., --verbosity {0,1,2,3}
                        Verbosity level; 0=minimal output, 1=normal output,
                        2=verbose output, 3=very verbose output
  --settings SETTINGS   The Python path to a settings module, e.g.
                        "myproject.settings.main". If this isn't provided, the
                        DJANGO_SETTINGS_MODULE environment variable will be
                        used.
  --pythonpath PYTHONPATH
                        A directory to add to the Python path, e.g.
                        "/home/djangoprojects/myproject".
  --traceback           Display a full stack trace on CommandError exceptions.
  --no-color            Don't colorize the command output.
  --force-color         Force colorization of the command output.
  --skip-checks         Skip system checks.

>>> shell('python manage.py removeurls --batch')
...
Search for [url] memo commands in accounting.PaymentTerm...
Search for [url] memo commands in comments.Comment...
Search for [url] memo commands in groups.Group...
Search for [url] memo commands in jinja.TextFieldTemplate...
Search for [url] memo commands in products.Category...
Search for [url] memo commands in products.Product...
Search for [url] memo commands in tickets.Ticket...
Search for [url] memo commands in topics.Interest...
Search for [url] memo commands in trading.InvoiceItem...
Search for [url] memo commands in working.Session...

Here is what you get it you forget to convert existing data:

>>> from lino.utils.soup import url2a
>>> print(url2a('Here is [url https://www.example.com an example].'))
Here is [url <a href="https://www.example.com" target="_blank">www.example.com</a> an example].

memo : The memo parser¶

ref¶

include¶

url¶

py¶

`memo` : The memo parser¶