`search` : Add search functionality¶

The lino.modlib.search plugin adds functionality for doing site-wide searches across all tables of the application.

This plugin can optionally use an ElasticSearch server to have faster and intelligent search functionalities.

Table of contents:

This is a tested document. The following instructions are used for initialization:

>>> import lino
>>> lino.startup('lino_book.projects.noi1e.settings.demo')
>>> from lino.api.doctest import *

Configuration settings ¶

use_elasticsearch¶

Whether to use ElasticSearch.

This is a site setting.

search.elasticsearch_url¶

The URL of the ElasticSearch server.

This is a plugin setting.

The `SiteSearch` table ¶

class lino.modlib.search.SiteSearch¶: The virtual table used to implement site-wide searches using default Django ORM filtering.

>>> rt.show('search.SiteSearch', quick_search="foo", limit=5)
... 
- **Comment #245** : breaking De : lino@foo.net [mailto:foo@bar.com] Envoyé :
  mardi 18 octobre 2016 08:52 À : eexample@foo.com Objet : [welcht] YOU modified
  FOO BAR Dear Aurélie , this is to notify / BAR BAR modified TODO: include a
  summary of the modifications. Any subsequent notifications about foo/ un...


- **Comment #242** :  Who| What| Done?
  ---|---|---
  Him| Bar|
  Her| Foo the Bar|  **x**
  Them| Floop the pig
  | x


- **Comment #237** : breaking De : lino@foo.net [mailto:foo@bar.com] Envoyé :
  mardi 18 octobre 2016 08:52 À : eexample@foo.com Objet : [welcht] YOU modified
  FOO BAR Dear Aurélie , this is to notify / BAR BAR modified TODO: include a
  summary of the modifications. Any subsequent notifications about foo/ un...


- **Comment #234** :  Who| What| Done?
  ---|---|---
  Him| Bar|
  Her| Foo the Bar|  **x**
  Them| Floop the pig
  | x


- **Comment #229** : breaking De : lino@foo.net [mailto:foo@bar.com] Envoyé :
  mardi 18 octobre 2016 08:52 À : eexample@foo.com Objet : [welcht] YOU modified
  FOO BAR Dear Aurélie , this is to notify / BAR BAR modified TODO: include a
  summary of the modifications. Any subsequent notifications about foo/ un...

If your search contains more than one word, it shows all rows containing both words.

>>> rt.show('search.SiteSearch', quick_search="est land", limit=5)
... 
- **Comment #243** : Lorem ipsum dolor sit amet, consectetur adipiscing elit.
  Nunc cursus felis nisi, eu pellentesque lorem lobortis non. Aenean non sodales
  neque, vitae venenatis lectus. In eros dui, gravida et dolor at, pellentesque
  hendrerit magna. Quisque vel lectus dictum, rhoncus massa feugiat, condimentum
  sem. Do...


- **Comment #235** : Lorem ipsum dolor sit amet, consectetur adipiscing elit.
  Nunc cursus felis nisi, eu pellentesque lorem lobortis non. Aenean non sodales
  neque, vitae venenatis lectus. In eros dui, gravida et dolor at, pellentesque
  hendrerit magna. Quisque vel lectus dictum, rhoncus massa feugiat, condimentum
  sem. Do...


- **Comment #227** : Lorem ipsum dolor sit amet, consectetur adipiscing elit.
  Nunc cursus felis nisi, eu pellentesque lorem lobortis non. Aenean non sodales
  neque, vitae venenatis lectus. In eros dui, gravida et dolor at, pellentesque
  hendrerit magna. Quisque vel lectus dictum, rhoncus massa feugiat, condimentum
  sem. Do...


- **Comment #203** : Lorem ipsum dolor sit amet, consectetur adipiscing elit.
  Nunc cursus felis nisi, eu pellentesque lorem lobortis non. Aenean non sodales
  neque, vitae venenatis lectus. In eros dui, gravida et dolor at, pellentesque
  hendrerit magna. Quisque vel lectus dictum, rhoncus massa feugiat, condimentum
  sem. Do...


- **Comment #187** : Lorem ipsum dolor sit amet, consectetur adipiscing elit.
  Nunc cursus felis nisi, eu pellentesque lorem lobortis non. Aenean non sodales
  neque, vitae venenatis lectus. In eros dui, gravida et dolor at, pellentesque
  hendrerit magna. Quisque vel lectus dictum, rhoncus massa feugiat, condimentum
  sem. Do...

>>> rt.show('search.SiteSearch', quick_search="123", limit=5)  
===================================================== ========================
 Description                                           Matches
----------------------------------------------------- ------------------------
 *Arens Andreas* (Partner)                             phone:+32 87**123**456
 *Arens Annette* (Partner)                             phone:+32 87**123**457
 *Dobbelstein-Demeulenaere Dorothée* (Partner)         id:123
 *Mr Andreas Arens* (Person)                           phone:+32 87**123**456
 *Mrs Annette Arens* (Person)                          phone:+32 87**123**457
 *Mrs Dorothée Dobbelstein-Demeulenaere* (Person)      id:123
 *+32 87123456* (Contact detail)                       value:+32 87**123**456
 *+32 87123457* (Contact detail)                       value:+32 87**123**457
 *Diner (09.05.2015 13:30)* (Calendar entry)           id:123
 *SLS 9.2* (Movement)                                  id:123
 *DOBBELSTEIN-DEMEULENAERE Dorothée (123)* (Patient)   id:123
===================================================== ========================

Dobbelstein-Demeulenaere Dorothée (id 123) is Partner, Person and Patient. All three instances are listed in the SiteSearch.

The `ElasticSiteSearch` table ¶

class lino.modlib.search.ElasticSiteSearch¶

A virtual table used to search on this Lino site using ElasticSearch.

It exists only when settings.SITE.use_elasticsearch is True.

Install an ElasticSearch server ¶

This section explains how an application developer can install ElasticSearch on their machine.

General instructions are here. For our demonstration purposes we will install ElasticSearch in a docker container using the instructions provided here.

Pull the docker image of ElasticSearch:

$ sudo docker pull docker.elastic.co/elasticsearch/elasticsearch:7.15.0

Run an ElasticSearch instance on a docker container:

$ sudo docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.15.0

PS: Always try to use the latest ElasticSearch version and replace 7.15.0 with current x.xx.x version.

To run a docker container instance with minimal security enabled, use the following command:

$ sudo docker run -p 127.0.0.1:9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "xpack.security.enabled=true" -e ELASTIC_PASSWORD="elastic" docker.elastic.co/elasticsearch/elasticsearch:7.15.0

Here we replaced -p 9200:9200 with -p 127.0.0.1:9200:9200 so that port 9200 is only accessible from the localhost and not open to the external network. The option -e "xpack.security.enabled=true" enables the security feature of the docker container and -e ELASTIC_PASSWORD="elastic" set the password to elastic for the default user: elastic.

Source: https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html

Preparing Indexes ¶

lino.modlib.search provides a management command python manage.py createindexes (recommended to run before publishing a site) that has the following grounds on a lino site:

The following method search.utils.ESResolver.resolve_es_indexes() gathers all the indexes, defined in the model definitions as lino.core.model.Model.ES_indexes, into a dictionary for further uses in the app elasticsearch_django's settings:

>>> from lino.modlib.search.utils import ESResolver
>>> indexes, _ = ESResolver.resolve_es_indexes()
>>> indexes
{'comment': {'models': [<class 'lino.modlib.comments.models.Comment'>]}, 'ticket': {'models': [<class 'lino_noi.lib.tickets.models.Ticket'>]}}
>>> ESResolver.read_indexes()
{'comment': {'models': ['comments.Comment']}, 'ticket': {'models': ['tickets.Ticket']}}

URL to ElasticSearch instance:

>>> es_url = dd.plugins.search.ES_url
>>> print(es_url)
https://elastic:mMh6KlFP0UAooywwsWPLJ3ae@lino.es.us-central1.gcp.cloud.es.io:9243

Create ElasticSearch index mapping files:

>>> # ESResolver.create_index_mapping_files()
>>> mappings_dir = dd.plugins.search.mappings_dir
>>> print(mappings_dir) 
/.../lino/modlib/search/search/mappings

Content of the mapping files:

>>> import json
>>> def get_mappings_from_files(mappings_dir):
...     maps = dict()
...     for index in ESResolver.get_indexes():
...         filename = mappings_dir / (index + '.json')
...         with open(filename, 'r') as f:
...             obj = json.load(f)
...         maps[index] = obj
...     print(maps)
>>> get_mappings_from_files(mappings_dir)
{'comment': {'mappings': {'properties': {'body': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}, 'analyzer': 'autocomplete', 'search_analyzer': 'autocomplete_search'}, 'body_full_preview': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}, 'analyzer': 'autocomplete', 'search_analyzer': 'autocomplete_search'}, 'body_short_preview': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}, 'analyzer': 'autocomplete', 'search_analyzer': 'autocomplete_search'}, 'model': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}, 'analyzer': 'autocomplete', 'search_analyzer': 'autocomplete_search'}, 'modified': {'type': 'date'}, 'owner_id': {'type': 'long'}, 'owner_type': {'type': 'long'}, 'private': {'type': 'boolean'}, 'user': {'type': 'long'}}}}, 'ticket': {'mappings': {'properties': {'closed': {'type': 'boolean'}, 'description': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}, 'analyzer': 'autocomplete', 'search_analyzer': 'autocomplete_search'}, 'end_user': {'type': 'long'}, 'feedback': {'type': 'boolean'}, 'model': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}, 'analyzer': 'autocomplete', 'search_analyzer': 'autocomplete_search'}, 'priority': {'type': 'long'}, 'private': {'type': 'boolean'}, 'site': {'type': 'long'}, 'standby': {'type': 'boolean'}, 'state': {'properties': {'text': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}}, 'value': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}}}}, 'summary': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}, 'analyzer': 'autocomplete', 'search_analyzer': 'autocomplete_search'}, 'ticket_type': {'type': 'long'}, 'upgrade_notes': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}, 'analyzer': 'autocomplete', 'search_analyzer': 'autocomplete_search'}, 'user': {'type': 'long'}, 'waiting_for': {'type': 'text', 'fields': {'keyword': {'type': 'keyword', 'ignore_above': 256}}, 'analyzer': 'autocomplete', 'search_analyzer': 'autocomplete_search'}}}}}

Don't read on ¶

The remaining part of this page is just technical stuff.

>>> from lino.core.utils import get_models
>>> ar = rt.login()
>>> user_type = ar.get_user().user_type
>>> count = 0
>>> for model in get_models():
...     t = model.get_default_table()
...     if not t.get_view_permission(user_type):
...         print("Not visible: {}".format(t))
...     count += 1
>>> print("Verified {} models".format(count))
Verified 90 models

>>> rt.models.contacts.Person.quick_search_fields_digit
(<django.db.models.fields.BigAutoField: id>,)

The following request caused problems before 20180920 (#2544 SiteSearch fails when it finds a name containing "&")

>>> rt.show('search.SiteSearch', quick_search="rumma")

Using Solr as search backend ¶

Lino utilizes django-haystack app along with apache Solr backend server to provide intelligent search functionalities.

Installing Solr and dependencies ¶

The following command will set the application developer for a compatible Solr - django-haystack environment.

Install pysolr:

$ pip install pysolr

Install django-haystack:

$ pip install django-haystack

Install java-1.8 (Linux):

jre1.8 is available for download here however, to make it more command line friendly, lino team made downloading jre1.8 available from another source for Linux (x64) based systems. Follow the commands below:
$ cd ~
$ curl -o jre-8u311-linux-x64.tar.gz "link"
$ tar -xvf jre-8u311-linux-x64.tar.gz
Change JAVA_HOME to our downloaded jre1.8:
$ export JAVA_HOME="$HOME/jre1.8.0_311"

Install solr-8.11.0:

$ cd ~
$ curl -o solr-8.11.0.zip "https://dlcdn.apache.org/lucene/solr/8.11.0/solr-8.11.0.zip"
$ unzip solr-8.11.0.zip

Add solr-6.6.6 to unix PATH:

$ export PATH="$HOME/solr-8.11.0/bin:$PATH"

Run solr and create a collection named lino:

$ solr start
$ solr create -c lino -n basic_config

Now we are ready to use solr with a lino project. Go to the root directory of you lino project (where manage.py is located) and run the following command:

$ python manage.py build_solr_schema_v2 --configure-directory="$HOME/solr-8.11.0/server/lino/conf" --reload-core=lino

The last command will build solr schema and reload the solr core name lino. And now your solr core/collection lino is ready for indexing and searching search documents.

search : Add search functionality¶

`search` : Add search functionality¶