Welcome | Get started | Dive into Lino | Contribute | Topics | Reference | More

dump2py : make a Python dump of a Lino database

pm dump2py

Usage:

$ pm dump2py [options] OUTPUT_DIR

This will create a Python dump of your database in the directory OUTPUT_DIR.

The directory will contain a series of .py files, one for every database model, plus one extra file named restore.py, which execfile()s the .py files.

Options:

--noinput

Do not prompt for user input of any kind.

--tolerate

Tolerate database errors. This can help making a partial snapshot of a database that is not (fully) synced with the application code.

--simulate

Just simulate, don't actually write any data. This is used for verifying whether thte database structure has changed after an upgrade.

--overwrite

Don't complain if the TARGET directory already exists. This will potentially overwrite existing files.

--max-row-count <NUM>

Change the maximum number of rows per source file from its default value (50000) to NUM.

See The --max-row-count option below.

This command is defined by the lino core plugin.

The --max-row-count option

When a table contains many rows, the resulting .py file can become so large that it doesn't fit into memory, causing the Python process to get killed when it tries to restore the data. To avoid this limitation, dump2py distributes the content over several files if a table contains are more than NUM rows.

The default value has been "clinically tested" and should be small enough for most machines.

Hint: When your process gets killed, before using this option, consider restarting the web services on your server and trying again. The web services can occupy considerable amounts of memory on a long-running production site. A simple reload_services.sh can fix your issue.

Glossary

Python dump

A dump of a Lino database created using pm dump2py. Python dumps are used for creating daily backups or before an upgrade with data migration.

Files

restore.py

The main script of a Python dump generated by the pm dump2py command.

This script is run during a site upgrade.

About Python dumps

A database dump is an image of the data in a database which you can use e.g. to backup and restore your data. While Django's dumpdata command lets you make database dumps in json and xml format, Lino extends this by letting you make a database dump in Python format. This is what we call a Python dump.

To make a Python dump of your database, you use the pm dump2py command. This django-admin command creates a directory of Python modules with one main module restore.py.

To restore a dump created using pm dump2py to your database, you execute the restore.py script using the pm run command:

$ pm run mydump/restore.py

Another important thing is that you can use such a backup for data migrations.

More concrete usage exmples in The dumps demo project.

Related work

The django-extensions repository has a Django-admin command named dumpscript, which is comparable to pm dump2py.

This page is a tested document and the following instructions are used for initialization:

>>> import lino
>>> lino.startup('lino_book.projects.min1.settings')
>>> from lino.api.doctest import *
>>> from atelier.sheller import Sheller
>>> shell = Sheller(settings.SITE.project_dir)

>>> shell("django-admin dump2py --help")
... 
usage: django-admin dump2py [-h] [--noinput] [-t] [-s] [-o] [-m MAX_ROW_COUNT] [--version] [-v {0,1,2,3}] [--settings SETTINGS] [--pythonpath PYTHONPATH] [--traceback] [--no-color] [--force-color]
                            [--skip-checks]
                            output_dir

positional arguments:
  output_dir            The directory where to write output files.

option...s:
  -h, --help            show this help message and exit
  --noinput             Do not prompt for input of any kind.
  -t, --tolerate        Tolerate database errors.
  -s, --simulate        Just simulate, don't actually write any data.
  -o, --overwrite       Overwrite existing files.
  -m MAX_ROW_COUNT, --max-row-count MAX_ROW_COUNT
                        Maximum number of rows per file.
  --version             Show program's version number and exit.
  -v {0,1,2,3}, --verbosity {0,1,2,3}
                        Verbosity level; 0=minimal output, 1=normal output, 2=verbose output, 3=very verbose output
  --settings SETTINGS   The Python path to a settings module, e.g. "myproject.settings.main". If this isn't provided, the DJANGO_SETTINGS_MODULE environment variable will be used.
  --pythonpath PYTHONPATH
                        A directory to add to the Python path, e.g. "/home/djangoprojects/myproject".
  --traceback           Raise on CommandError exceptions.
  --no-color            Don't colorize the command output.
  --force-color         Force colorization of the command output.
  --skip-checks         Skip system checks.