Welcome | Get started | Dive | Contribute | Topics | Reference | Changes | More

The dumps demo project

This document shows a series of usage examples for Python dumps. It verifies whether #2204 (AmbiguousTimeError) is fixed, it demonstrates three methods of writing demo multilingual data for babel fields, and it demonstrates behaviour of choicelists in Python dumps.

See also Data migrations à la Lino and dump2py : make a Python dump of a Lino database.

This document uses the lino_book.projects.dumps demo project.

>>> from atelier.sheller import Sheller
>>> shell = Sheller("lino_book/projects/dumps")

Database structure

Here is our models.py file:

# -*- coding: UTF-8 -*-
# Copyright 2016-2018 Rumma & Ko Ltd
# License: GNU Affero General Public License v3 (see file COPYING for details)

from django.db import models
from lino.api import dd, _
from lino.mixins import BabelDesignated
from .choicelists import Bars


class Foo(BabelDesignated):

    class Meta(object):
        app_label = 'dumps'

    #     verbose_name = _("Foo")
    #     verbose_name_plural = _("Foos")

    last_visit = models.DateTimeField(_("Last visit"), editable=False)
    bar = Bars.field(default='sale')


class Foos(dd.Table):
    model = Foo

We also have a choicelist:

# -*- coding: UTF-8 -*-
# Copyright 2018 Rumma & Ko Ltd
# License: GNU Affero General Public License v3 (see file COPYING for details)

from lino.api import dd, _


class Bars(dd.ChoiceList):
    verbose_name = _("Bar")
    verbose_name_plural = _("Bars")


add = Bars.add_item
add('10', _("Sale"), 'sale')
add('20', _("Purchase"), 'purchase')
add('30', _("Profit"), 'profit')
add('40', _("Loss"), 'loss')

And we have a Python fixture in file fixtures/demo.py, which adds three records:

# -*- coding: UTF-8 -*-
# Copyright 2016-2018 Rumma & Ko Ltd
# License: GNU Affero General Public License v3 (see file COPYING for details)
"""The demo fixture for this tutorial.
"""

from datetime import datetime
from django.conf import settings
from django.utils.timezone import make_aware
from datetime import timezone

utc = timezone.utc
from lino.api import dd, _
from lino_book.projects.dumps.models import Foo

if settings.USE_TZ:

    def dt(*args):
        # return make_aware(datetime(*args))
        return make_aware(datetime(*args), timezone=utc)
else:

    def dt(*args):
        return datetime(*args)


def objects():
    # three methods for specifying content of babelfields in fixtures:
    yield Foo(designation="First",
              designation_de="Erster",
              designation_fr="Premier",
              last_visit=dt(2016, 7, 2, 23, 55, 12))
    yield Foo(last_visit=dt(2016, 7, 3, 0, 10, 23),
              **dd.str2kw('designation', _("January")))
    yield Foo(
        # last_visit=dt(2016, 10, 30, 4, 34, 0),
        last_visit=dt(2017, 10, 29, 3, 16, 6),
        # last_visit=dt(2012, 10, 28, 4, 34, 0),
        **dd.babelkw('designation',
                     en="Three",
                     de="Drei",
                     fr="Trois",
                     et="Kolm"))

First site : without time zone

In a first example we have USE_TZ set to False. This is defined by the settings/a.py file:

"""Default settings for a :mod:`lino_book.projects.dump` site.

This module instantiates a :setting:`SITE` variable and thus is
designed to be used directly as a :setting:`DJANGO_SETTINGS_MODULE`.

"""

from ..settings import *
from lino.utils import i2d


class Site(Site):
    languages = "en de fr"

    the_demo_date = i2d(20160702)


SITE = Site(globals())
# SECRET_KEY = "20227"  # see :djangoticket:`20227`
DEBUG = True

We initialize our database from the demo fixture:

>>> shell("python manage_a.py prep --noinput")
... 
`initdb demo` started on database .../default.db.
...
Loading data from ...
Installed 3 object(s) from 1 fixture(s)

And here is the result:

>>> shell("python manage_a.py show dumps.Foos")
... 
==== ============= ================== ================== ===================== ======
 ID   Designation   Designation (de)   Designation (fr)   Last visit            Bar
---- ------------- ------------------ ------------------ --------------------- ------
 1    First         Erster             Premier            2016-07-02 23:55:12   Sale
 2    January       Januar             janvier            2016-07-03 00:10:23   Sale
 3    Three         Drei               Trois              2017-10-29 03:16:06   Sale
==== ============= ================== ================== ===================== ======

Note that our demo data contains an ambiguous time stamp. When somebody living in Brussels tells you “it was at on 2017-10-29 at 01:16:06”, then you don’t know whether they mean summer or winter time. Because their clock showed that particular time twice during that night. Every year on the last Sunday of October, all clocks in Europe are turned back at 2am by one hour to 1am again. The timestamp “2017-10-29 01:16:06” is ambiguous. Thanks to Ilian Iliev for publishing his blog post Django, pytz, NonExistentTimeError and AmbiguousTimeError.

Now we run dum2py to create a dump:

>>> shell("python manage_a.py dump2py tmp/a --overwrite")
... 
Writing .../lino_book/projects/dumps/tmp/a/restore.py...
Wrote 3 objects to .../lino_book/projects/dumps/tmp/a/restore.py and siblings.

The dump generated by pm dump2py consists of two files. First restore.py:

#!/usr/bin/env python
# -*- coding: UTF-8 -*-

# This is a Python dump created using dump2py.
# DJANGO_SETTINGS_MODULE was 'lino_book.projects.dumps.settings.a', TIME_ZONE was 'UTC'.


from lino import logger

SOURCE_VERSION = 'None'

import os
import six
from decimal import Decimal
from datetime import datetime
from datetime import time, date
from django.conf import settings
from django.utils.timezone import make_aware
from datetime import timezone; utc = timezone.utc
from django.core.management import call_command
# from django.contrib.contenttypes.models import ContentType
from lino.utils.dpy import create_mti_child
from lino.utils.dpy import DpyLoader
from lino.core.utils import resolve_model

if settings.USE_TZ:
    def dt(*args):
        return make_aware(datetime(*args), timezone=utc)
else:
    def dt(*args):
        return datetime(*args)

def new_content_type_id(m):
    if m is None: return m
    ct = settings.SITE.models.contenttypes.ContentType.objects.get_for_model(m)
    if ct is None: return None
    return ct.pk

def pmem():
    # Thanks to https://stackoverflow.com/questions/938733/total-memory-used-by-python-process
    process = psutil.Process(os.getpid())
    print(process.memory_info().rss)

def execfile(fn, *args):
    logger.info("Execute file %s ...", fn)
    six.exec_(compile(open(fn, "rb").read(), fn, 'exec'), *args)
    # pmem()  # requires pip install psutil


def bv2kw(fieldname, values):
    """
    Needed if `Site.languages` changed between dumpdata and loaddata
    """
    return settings.SITE.babelkw(fieldname, en=values[0],de=values[1],fr=values[2])

dumps_Foo = resolve_model("dumps.Foo")


def create_dumps_foo(id, designation, last_visit, bar):
#    if bar: bar = settings.SITE.models.dumps.Bars.get_by_value(bar)
    kw = dict()
    kw.update(id=id)
    if designation is not None: kw.update(bv2kw('designation',designation))
    kw.update(last_visit=last_visit)
    kw.update(bar=bar)
    return dumps_Foo(**kw)




def main(args):
    loader = DpyLoader(globals(), quick=args.quick)
    from django.core.management import call_command
    call_command('initdb', interactive=args.interactive)
    os.chdir(os.path.dirname(__file__))
    loader.initialize()
    args = (globals(), locals())

    execfile("dumps_foo.py", *args)
    loader.finalize()
    logger.info("Loaded %d objects", loader.count_objects)
    call_command('resetsequences')

if __name__ == '__main__':
    import argparse
    parser = argparse.ArgumentParser(description='Restore the data.')
    parser.add_argument('--noinput', dest='interactive',
        action='store_false', default=True,
        help="Don't ask for confirmation before flushing the database.")
    parser.add_argument('--quick', dest='quick',
        action='store_true',default=False,
        help='Do not call full_clean() on restored instances.')

    args = parser.parse_args()
    main(args)

and the second is dumps_foo.py (app label and model name)

# -*- coding: UTF-8 -*-
logger.info("Loading 3 objects to table dumps_foo...")
# fields: id, designation, last_visit, bar
loader.save(create_dumps_foo(1,['First', 'Erster', 'Premier'],dt(2016,7,2,23,55,12),'10'))
loader.save(create_dumps_foo(2,['January', 'Januar', 'janvier'],dt(2016,7,3,0,10,23),'10'))
loader.save(create_dumps_foo(3,['Three', 'Drei', 'Trois'],dt(2017,10,29,3,16,6),'10'))

loader.flush_deferred_objects()

Let’s verify whether the newly created dump is as expected:

>>> shell("diff a tmp/a")  # "diff -I OTHER_SOURCES a tmp/a"
... 

Load the dump, dump again and verify that both dumps are the same:

>>> shell("python manage_a.py run a/restore.py --noinput")
... 
Unversioned Site instance : no database migration
`initdb ` started on database .../default.db.
...
Execute file dumps_foo.py ...
Loading 3 objects to table dumps_foo...
Loaded 3 objects
>>> shell("python manage_a.py dump2py tmp/a --overwrite")
... 
Writing .../lino_book/projects/dumps/tmp/a/restore.py...
Wrote 3 objects to .../lino_book/projects/dumps/tmp/a/restore.py and siblings.
>>> shell("diff a tmp/a")
... 

Second suite

Now the same with b, i.e. with USE_TZ enabled.

The settings/b.py file:

from .a import *

USE_TZ = True
TIME_ZONE = "Europe/Tallinn"

We load our demo data:

>>> shell("python manage_b.py prep --noinput")
... 
`initdb demo` started on database .../default.db.
...
Loading data from ...
Installed 3 object(s) from 1 fixture(s)

The result as seen by the user is the same as in a.

>>> shell("python manage_b.py show dumps.Foos")
... 
==== ============= ================== ================== =========================== ======
 ID   Designation   Designation (de)   Designation (fr)   Last visit                  Bar
---- ------------- ------------------ ------------------ --------------------------- ------
 1    First         Erster             Premier            2016-07-02 23:55:12+00:00   Sale
 2    January       Januar             janvier            2016-07-03 00:10:23+00:00   Sale
 3    Three         Drei               Trois              2017-10-29 03:16:06+00:00   Sale
==== ============= ================== ================== =========================== ======
>>> shell("python manage_b.py dump2py tmp/b --overwrite")
... 
Writing .../lino_book/projects/dumps/tmp/b/restore.py...
Wrote 3 objects to .../lino_book/projects/dumps/tmp/b/restore.py and siblings.

Verify that the newly created dump is as expected:

>>> shell("diff b tmp/b")
... 

Load the dump, dump again and verify that both dumps are the same:

>>> shell("python manage_b.py run b/restore.py --noinput")
... 
Unversioned Site instance : no database migration
`initdb ` started on database .../default.db.
...
Execute file dumps_foo.py ...
Loading 3 objects to table dumps_foo...
Loaded 3 objects
>>> shell("python manage_b.py dump2py tmp/b --overwrite")
... 
Writing .../lino_book/projects/dumps/tmp/b/restore.py...
Wrote 3 objects to .../lino_book/projects/dumps/tmp/b/restore.py and siblings.
>>> shell("diff b tmp/b")
... 

Third suite

Here we test the –max-row-count option. Since we have only three rows, we change the value from its default 50000 to 2 in order to trigger the situation:

>>> shell("python manage_b.py dump2py tmp/c --overwrite -m 2")
... 
Writing .../lino_book/projects/dumps/tmp/c/restore.py...
Wrote 3 objects to .../lino_book/projects/dumps/tmp/c/restore.py and siblings.

Verify that the newly created dump is as expected:

>>> shell("diff c tmp/c")
... 

Error messages

Here we try to call dump2y in some invalid ways, just to demonstrate the possible error messages.

>>> shell("python manage_a.py dump2py")
... 
usage: manage_a.py dump2py [-h] ...
manage_a.py dump2py: error: ...
>>> shell("python manage_a.py dump2py a")
... 
CommandError: Specified output_dir ...lino_book/projects/dumps/a already exists. Delete it yourself if you dare!

Bibliography

Thanks to Ilian Iliev Django, pytz, NonExistentTimeError and AmbiguousTimeError