`search` : Add search functionality¶

The lino.modlib.search plugin adds functionality for doing site-wide searches across all tables of the application.

This plugin can optionally use an ElasticSearch server to have faster and intelligent search functionalities.

Table of contents:

Side note: Code snippets (lines starting with >>>) in this document get tested as part of our development workflow. The following initialization snippet tells you which demo project is being used in this document.

>>> import lino
>>> lino.startup('lino_book.projects.noi1e.settings.demo')
>>> from lino.api.doctest import *

Configuration settings ¶

use_elasticsearch¶

Whether to use ElasticSearch.

This is a site setting.

search.elasticsearch_url¶

The URL of the ElasticSearch server.

This is a plugin setting.

The `SiteSearch` table ¶

class lino.modlib.search.SiteSearch¶

A virtual table that searches in all database tables.

Open it using the menu command Site ‣ Search or the quick link [Search]

>>> #ses.show_menu_path('search.SiteSearch')

The base version uses the default Django ORM filtering.

>>> ses = rt.login("robin")
>>> ses.show('search.SiteSearch', quick_search="foo", limit=5)
... 
- **[Comment #2](Detail)** :  Who| What| Done?
  ---|---|---
  Him| Bar|
  Her| Foo the Bar|  **x**
  Them| Floop the pig
  | x


- **[Comment #5](Detail)** : breaking **De :**
  [lino@foo.net](mailto:lino@foo.net) [[mailto:foo@bar.com](mailto:foo@bar.com)]
  **Envoyé :** mardi 18 octobre 2016 08:52
  **À :** [eexample@foo.com](mailto:Far@baz.net)
  **Objet :** [welcht] YOU modified FOO BAR Dear Aurélie , this is to notify /
  BAR BAR modified TODO: include a summary of the modifications. Any subsequent
  notifications about ...


- **[Comment #10](Detail)** :  Who| What| Done?
  ---|---|---
  Him| Bar|
  Her| Foo the Bar|  **x**
  Them| Floop the pig
  | x


- **[Comment #13](Detail)** : breaking **De :**
  [lino@foo.net](mailto:lino@foo.net) [[mailto:foo@bar.com](mailto:foo@bar.com)]
  **Envoyé :** mardi 18 octobre 2016 08:52
  **À :** [eexample@foo.com](mailto:Far@baz.net)
  **Objet :** [welcht] YOU modified FOO BAR Dear Aurélie , this is to notify /
  BAR BAR modified TODO: include a summary of the modifications. Any subsequent
  notifications about ...


- **[Comment #18](Detail)** :  Who| What| Done?
  ---|---|---
  Him| Bar|
  Her| Foo the Bar|  **x**
  Them| Floop the pig
  | x

If your search contains more than one word, it shows all rows containing both words.

>>> ses.show('search.SiteSearch', quick_search="est land", limit=5)
... 
- **Estonia**


- **Flandre de l'Est**


- **Flandre de l'Ouest**


- **[Comment #3](Detail)** : Lorem ipsum **dolor sit amet** , consectetur
  adipiscing elit. Nunc cursus felis nisi, eu pellentesque lorem lobortis non.
  Aenean non sodales neque, vitae venenatis lectus. In eros dui, gravida et
  dolor at, pellentesque hendrerit magna. Quisque vel lectus dictum, rhoncus
  massa feugiat, condimentum sem. ...


- **[Comment #11](Detail)** : Lorem ipsum **dolor sit amet** , consectetur
  adipiscing elit. Nunc cursus felis nisi, eu pellentesque lorem lobortis non.
  Aenean non sodales neque, vitae venenatis lectus. In eros dui, gravida et
  dolor at, pellentesque hendrerit magna. Quisque vel lectus dictum, rhoncus
  massa feugiat, condimentum sem. ...

>>> ses.show('search.SiteSearch', quick_search="123", limit=5)  
===================================================== ========================
 Description                                           Matches
----------------------------------------------------- ------------------------
 *Arens Andreas* (Partner)                             phone:+32 87**123**456
 *Arens Annette* (Partner)                             phone:+32 87**123**457
 *Dobbelstein-Demeulenaere Dorothée* (Partner)         id:123
 *Mr Andreas Arens* (Person)                           phone:+32 87**123**456
 *Mrs Annette Arens* (Person)                          phone:+32 87**123**457
 *Mrs Dorothée Dobbelstein-Demeulenaere* (Person)      id:123
 *+32 87123456* (Contact detail)                       value:+32 87**123**456
 *+32 87123457* (Contact detail)                       value:+32 87**123**457
 *Diner (09.05.2015 13:30)* (Calendar entry)           id:123
 *SLS 9.2* (Movement)                                  id:123
 *DOBBELSTEIN-DEMEULENAERE Dorothée (123)* (Patient)   id:123
===================================================== ========================

Dobbelstein-Demeulenaere Dorothée (id 123) is Partner, Person and Patient. All three instances are listed in the SiteSearch.

The `ElasticSiteSearch` table ¶

class lino.modlib.search.ElasticSiteSearch¶

A virtual table used to search on this Lino site using ElasticSearch.

It exists only when settings.SITE.use_elasticsearch is True.

Install an ElasticSearch server ¶

This section explains how an application developer can install ElasticSearch on their machine.

General instructions are here. For our demonstration purposes we will install ElasticSearch in a docker container using the instructions provided here.

Pull the docker image of ElasticSearch:

$ sudo docker pull docker.elastic.co/elasticsearch/elasticsearch:7.15.0

Run an ElasticSearch instance on a docker container:

$ sudo docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.15.0

PS: Always try to use the latest ElasticSearch version and replace 7.15.0 with current x.xx.x version.

To run a docker container instance with minimal security enabled, use the following command:

$ sudo docker run -p 127.0.0.1:9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "xpack.security.enabled=true" -e ELASTIC_PASSWORD="elastic" docker.elastic.co/elasticsearch/elasticsearch:7.15.0

Here we replaced -p 9200:9200 with -p 127.0.0.1:9200:9200 so that port 9200 is only accessible from the localhost and not open to the external network. The option -e “xpack.security.enabled=true” enables the security feature of the docker container and -e ELASTIC_PASSWORD=”elastic” set the password to elastic for the default user: elastic.

Source: https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html

Preparing Indexes ¶

lino.modlib.search provides a management command python manage.py createindexes (recommended to run before publishing a site) that has the following grounds on a lino site:

The following method search.utils.ESResolver.resolve_es_indexes() gathers all the indexes, defined in the model definitions as lino.core.model.Model.ES_indexes, into a dictionary for further uses in the app elasticsearch_django’s settings:

>>> from lino.modlib.search.utils import ESResolver
>>> indexes, _ = ESResolver.resolve_es_indexes()
>>> indexes
{'comment': {'models': [<class 'lino.modlib.comments.models.Comment'>]}, 'ticket': {'models': [<class 'lino_noi.lib.tickets.models.Ticket'>]}}
>>> ESResolver.read_indexes()
{'comment': {'models': ['comments.Comment']}, 'ticket': {'models': ['tickets.Ticket']}}

URL to ElasticSearch instance:

>>> es_url = dd.plugins.search.ES_url
>>> print(es_url)
https://elastic:mMh6KlFP0UAooywwsWPLJ3ae@lino.es.us-central1.gcp.cloud.es.io:9243

Create ElasticSearch index mapping files:

>>> # ESResolver.create_index_mapping_files()
>>> mappings_dir = dd.plugins.search.mappings_dir
>>> print(mappings_dir) 
/.../lino/modlib/search/search/mappings

Content of the mapping files:

>>> import json
>>> def get_mappings_from_files(mappings_dir):
...     maps = dict()
...     for index in ESResolver.get_indexes():
...         filename = mappings_dir / (index + '.json')
...         with open(filename, 'r') as f:
...             obj = json.load(f)
...         maps[index] = obj
...     print(maps)
>>> get_mappings_from_files(mappings_dir)
{'comment': {'mappings': {'properties': {'body': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}, 'analyzer': 'autocomplete', 'search_analyzer': 'autocomplete_search'}, 'body_full_preview': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}, 'analyzer': 'autocomplete', 'search_analyzer': 'autocomplete_search'}, 'body_short_preview': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}, 'analyzer': 'autocomplete', 'search_analyzer': 'autocomplete_search'}, 'model': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}, 'analyzer': 'autocomplete', 'search_analyzer': 'autocomplete_search'}, 'modified': {'type': 'date'}, 'owner_id': {'type': 'long'}, 'owner_type': {'type': 'long'}, 'private': {'type': 'boolean'}, 'user': {'type': 'long'}}}}, 'ticket': {'mappings': {'properties': {'closed': {'type': 'boolean'}, 'description': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}, 'analyzer': 'autocomplete', 'search_analyzer': 'autocomplete_search'}, 'end_user': {'type': 'long'}, 'feedback': {'type': 'boolean'}, 'model': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}, 'analyzer': 'autocomplete', 'search_analyzer': 'autocomplete_search'}, 'priority': {'type': 'long'}, 'private': {'type': 'boolean'}, 'site': {'type': 'long'}, 'standby': {'type': 'boolean'}, 'state': {'properties': {'text': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}}, 'value': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}}}}, 'summary': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}, 'analyzer': 'autocomplete', 'search_analyzer': 'autocomplete_search'}, 'ticket_type': {'type': 'long'}, 'upgrade_notes': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}, 'analyzer': 'autocomplete', 'search_analyzer': 'autocomplete_search'}, 'user': {'type': 'long'}, 'waiting_for': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}, 'analyzer': 'autocomplete', 'search_analyzer': 'autocomplete_search'}}}}}

Don’t read on ¶

The remaining part of this page is just technical stuff.

>>> from lino.core.utils import get_models
>>> ar = rt.login()
>>> user_type = ar.get_user().user_type
>>> count = 0
>>> for model in get_models():
...     t = model.get_default_table()
...     if t and not t.get_view_permission(user_type):
...         print("Not visible: {}".format(t))
...     count += 1
>>> print("Verified {} models".format(count))
Verified 110 models

>>> rt.models.contacts.Person.quick_search_fields_digit
(<django.db.models.fields.BigAutoField: id>,)

The following request caused problems before 20180920 (#2544 SiteSearch fails when it finds a name containing “&”)

>>> ses.show('search.SiteSearch', quick_search="rumma & ko")
... 
- [Partner #100](…) : [Rumma & Ko OÜ](…) (Uus tn 1, Vigala vald, 78003 Rapla
  maakond, Estonia)

...

AnonymousUser does NOT have row permission on Partner

>>> rt.show('search.SiteSearch', quick_search="rumma")

Using Solr as search backend ¶

Lino utilizes django-haystack app along with apache Solr backend server to provide intelligent search functionalities.

Installing Solr and dependencies ¶

The following command will set the application developer for a compatible Solr - django-haystack environment.

Install pysolr:

$ pip install pysolr

Install django-haystack:

$ pip install django-haystack

Install java-1.8 (Linux):

jre1.8 is available for download here however, to make it more command line friendly, lino team made downloading jre1.8 available from another source for Linux (x64) based systems. Follow the commands below:
$ cd ~
$ curl -o jre-8u311-linux-x64.tar.gz "link"
$ tar -xvf jre-8u311-linux-x64.tar.gz
Change JAVA_HOME to our downloaded jre1.8:
$ export JAVA_HOME="$HOME/jre1.8.0_311"

Install solr-8.11.0:

$ cd ~
$ curl -o solr-8.11.0.zip "https://dlcdn.apache.org/lucene/solr/8.11.0/solr-8.11.0.zip"
$ unzip solr-8.11.0.zip

Add solr-6.6.6 to unix PATH:

$ export PATH="$HOME/solr-8.11.0/bin:$PATH"

Run solr and create a collection named lino:

$ solr start
$ solr create -c lino -n basic_config

Now we are ready to use solr with a lino project. Go to the root directory of you lino project (where manage.py is located) and run the following command:

$ python manage.py build_solr_schema_v2 --configure-directory="$HOME/solr-8.11.0/server/lino/conf" --reload-core=lino

The last command will build solr schema and reload the solr core name lino. And now your solr core/collection lino is ready for indexing and searching search documents.

search : Add search functionality¶

`search` : Add search functionality¶