Welcome | Get started | Dive into Lino | Contribute | Topics | Reference | More
The dumps demo project¶
This document shows a series of usage examples for Python dumps. It verifies whether #2204 (AmbiguousTimeError) is fixed, it demonstrates three methods of writing demo multilingual data for babel fields, and it demonstrates behaviour of choicelists in Python dumps.
See also Data migrations à la Lino and dump2py : make a Python dump of a Lino database.
This document uses the lino_book.projects.dumps demo project.
>>> from atelier.sheller import Sheller
>>> shell = Sheller("lino_book/projects/dumps")
Database structure¶
Here is our models.py file:
# -*- coding: UTF-8 -*-
# Copyright 2016-2018 Rumma & Ko Ltd
# License: GNU Affero General Public License v3 (see file COPYING for details)
from django.db import models
from lino.api import dd, _
from lino.mixins import BabelDesignated
from .choicelists import Bars
class Foo(BabelDesignated):
class Meta(object):
app_label = 'dumps'
# verbose_name = _("Foo")
# verbose_name_plural = _("Foos")
last_visit = models.DateTimeField(_("Last visit"), editable=False)
bar = Bars.field(default='sale')
class Foos(dd.Table):
model = Foo
We also have a choicelist:
# -*- coding: UTF-8 -*-
# Copyright 2018 Rumma & Ko Ltd
# License: GNU Affero General Public License v3 (see file COPYING for details)
from lino.api import dd, _
class Bars(dd.ChoiceList):
verbose_name = _("Bar")
verbose_name_plural = _("Bars")
add = Bars.add_item
add('10', _("Sale"), 'sale')
add('20', _("Purchase"), 'purchase')
add('30', _("Profit"), 'profit')
add('40', _("Loss"), 'loss')
And we have a Python fixture in file fixtures/demo.py, which adds three
records:
# -*- coding: UTF-8 -*-
# Copyright 2016-2018 Rumma & Ko Ltd
# License: GNU Affero General Public License v3 (see file COPYING for details)
"""The demo fixture for this tutorial.
"""
from datetime import datetime
from django.conf import settings
from django.utils.timezone import make_aware
from datetime import timezone
utc = timezone.utc
from lino.api import dd, _
from lino_book.projects.dumps.models import Foo
if settings.USE_TZ:
def dt(*args):
# return make_aware(datetime(*args))
return make_aware(datetime(*args), timezone=utc)
else:
def dt(*args):
return datetime(*args)
def objects():
# three methods for specifying content of babelfields in fixtures:
yield Foo(designation="First",
designation_de="Erster",
designation_fr="Premier",
last_visit=dt(2016, 7, 2, 23, 55, 12))
yield Foo(last_visit=dt(2016, 7, 3, 0, 10, 23),
**dd.str2kw('designation', _("January")))
yield Foo(
# last_visit=dt(2016, 10, 30, 4, 34, 0),
last_visit=dt(2017, 10, 29, 3, 16, 6),
# last_visit=dt(2012, 10, 28, 4, 34, 0),
**dd.babelkw('designation',
en="Three",
de="Drei",
fr="Trois",
et="Kolm"))
First site : without time zone¶
In a first example we have USE_TZ set to False. This is
defined by the settings/a.py file:
"""Default settings for a :mod:`lino_book.projects.dump` site.
This module instantiates a :setting:`SITE` variable and thus is
designed to be used directly as a :setting:`DJANGO_SETTINGS_MODULE`.
"""
from ..settings import *
from lino.utils import i2d
class Site(Site):
languages = "en de fr"
the_demo_date = i2d(20160702)
SITE = Site(globals())
# SECRET_KEY = "20227" # see :djangoticket:`20227`
DEBUG = True
We initialize our database from the demo fixture:
>>> shell("python manage_a.py prep --noinput")
...
`initdb demo` started on database .../default.db.
...
Loading data from ...
Installed 3 object(s) from 1 fixture(s)
And here is the result:
>>> shell("python manage_a.py show dumps.Foos")
...
==== ============= ================== ================== ===================== ======
ID Designation Designation (de) Designation (fr) Last visit Bar
---- ------------- ------------------ ------------------ --------------------- ------
1 First Erster Premier 2016-07-02 23:55:12 Sale
2 January Januar janvier 2016-07-03 00:10:23 Sale
3 Three Drei Trois 2017-10-29 03:16:06 Sale
==== ============= ================== ================== ===================== ======
Note that our demo data contains an ambigous time stamp. When somebody living in Brussels tells you "it was at on 2017-10-29 at 01:16:06", then you don't know whether they mean summer or winter time. Because their clock showed that particular time twice during that night. Every year on the last Sunday of October, all clocks in Europe are turned back at 2am by one hour to 1am again. The timestamp "2017-10-29 01:16:06" is ambigous. Thanks to Ilian Iliev for publishing his blog post Django, pytz, NonExistentTimeError and AmbiguousTimeError.
Now we run dum2py to create a dump:
>>> shell("python manage_a.py dump2py tmp/a --overwrite")
...
Writing .../lino_book/projects/dumps/tmp/a/restore.py...
Wrote 3 objects to .../lino_book/projects/dumps/tmp/a/restore.py and siblings.
The dump generated by pm dump2py consists of two files. First
restore.py:
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
# This is a Python dump created using dump2py.
# DJANGO_SETTINGS_MODULE was 'lino_book.projects.dumps.settings.a', TIME_ZONE was 'UTC'.
from lino import logger
SOURCE_VERSION = 'None'
OTHER_SOURCES = {'lino': '24.1.5', 'lino_book': '20.1.1'}
import os
import six
from decimal import Decimal
from datetime import datetime
from datetime import time, date
from django.conf import settings
from django.utils.timezone import make_aware
from datetime import timezone; utc = timezone.utc
from django.core.management import call_command
# from django.contrib.contenttypes.models import ContentType
from lino.utils.dpy import create_mti_child
from lino.utils.dpy import DpyLoader
from lino.core.utils import resolve_model
if settings.USE_TZ:
def dt(*args):
return make_aware(datetime(*args), timezone=utc)
else:
def dt(*args):
return datetime(*args)
def new_content_type_id(m):
if m is None: return m
ct = settings.SITE.models.contenttypes.ContentType.objects.get_for_model(m)
if ct is None: return None
return ct.pk
def pmem():
# Thanks to https://stackoverflow.com/questions/938733/total-memory-used-by-python-process
process = psutil.Process(os.getpid())
print(process.memory_info().rss)
def execfile(fn, *args):
logger.info("Execute file %s ...", fn)
six.exec_(compile(open(fn, "rb").read(), fn, 'exec'), *args)
# pmem() # requires pip install psutil
def bv2kw(fieldname, values):
"""
Needed if `Site.languages` changed between dumpdata and loaddata
"""
return settings.SITE.babelkw(fieldname, en=values[0],de=values[1],fr=values[2])
dumps_Foo = resolve_model("dumps.Foo")
def create_dumps_foo(id, designation, last_visit, bar):
# if bar: bar = settings.SITE.models.dumps.Bars.get_by_value(bar)
kw = dict()
kw.update(id=id)
if designation is not None: kw.update(bv2kw('designation',designation))
kw.update(last_visit=last_visit)
kw.update(bar=bar)
return dumps_Foo(**kw)
def main(args):
loader = DpyLoader(globals(), quick=args.quick)
from django.core.management import call_command
call_command('initdb', interactive=args.interactive)
os.chdir(os.path.dirname(__file__))
loader.initialize()
args = (globals(), locals())
execfile("dumps_foo.py", *args)
loader.finalize()
logger.info("Loaded %d objects", loader.count_objects)
call_command('resetsequences')
if __name__ == '__main__':
import argparse
parser = argparse.ArgumentParser(description='Restore the data.')
parser.add_argument('--noinput', dest='interactive',
action='store_false', default=True,
help="Don't ask for confirmation before flushing the database.")
parser.add_argument('--quick', dest='quick',
action='store_true',default=False,
help='Do not call full_clean() on restored instances.')
args = parser.parse_args()
main(args)
and the second is dumps_foo.py (app label and model name)
# -*- coding: UTF-8 -*-
logger.info("Loading 3 objects to table dumps_foo...")
# fields: id, designation, last_visit, bar
loader.save(create_dumps_foo(1,['First', 'Erster', 'Premier'],dt(2016,7,2,23,55,12),'10'))
loader.save(create_dumps_foo(2,['January', 'Januar', 'janvier'],dt(2016,7,3,0,10,23),'10'))
loader.save(create_dumps_foo(3,['Three', 'Drei', 'Trois'],dt(2017,10,29,3,16,6),'10'))
loader.flush_deferred_objects()
Verify that the newly created dump is as expected:
>>> shell("diff -I OTHER_SOURCES a tmp/a")
...
Load the dump, dump again and verify that both dumps are the same:
>>> shell("python manage_a.py run a/restore.py --noinput")
...
Unversioned Site instance : no database migration
`initdb ` started on database .../default.db.
...
Execute file dumps_foo.py ...
Loading 3 objects to table dumps_foo...
Loaded 3 objects
>>> shell("python manage_a.py dump2py tmp/a --overwrite")
...
Writing .../lino_book/projects/dumps/tmp/a/restore.py...
Wrote 3 objects to .../lino_book/projects/dumps/tmp/a/restore.py and siblings.
>>> shell("diff -I OTHER_SOURCES a tmp/a")
...
Second suite¶
Now the same with b, i.e. with USE_TZ enabled.
The settings/b.py file:
from .a import *
USE_TZ = True
TIME_ZONE = "Europe/Tallinn"
We load our demo data:
>>> shell("python manage_b.py prep --noinput")
...
`initdb demo` started on database .../default.db.
...
Loading data from ...
Installed 3 object(s) from 1 fixture(s)
The result as seen by the user is the same as in a.
>>> shell("python manage_b.py show dumps.Foos")
...
==== ============= ================== ================== =========================== ======
ID Designation Designation (de) Designation (fr) Last visit Bar
---- ------------- ------------------ ------------------ --------------------------- ------
1 First Erster Premier 2016-07-02 23:55:12+00:00 Sale
2 January Januar janvier 2016-07-03 00:10:23+00:00 Sale
3 Three Drei Trois 2017-10-29 03:16:06+00:00 Sale
==== ============= ================== ================== =========================== ======
>>> shell("python manage_b.py dump2py tmp/b --overwrite")
...
Writing .../lino_book/projects/dumps/tmp/b/restore.py...
Wrote 3 objects to .../lino_book/projects/dumps/tmp/b/restore.py and siblings.
Verify that the newly created dump is as expected:
>>> shell("diff -I OTHER_SOURCES b tmp/b")
...
Load the dump, dump again and verify that both dumps are the same:
>>> shell("python manage_b.py run b/restore.py --noinput")
...
Unversioned Site instance : no database migration
`initdb ` started on database .../default.db.
...
Execute file dumps_foo.py ...
Loading 3 objects to table dumps_foo...
Loaded 3 objects
>>> shell("python manage_b.py dump2py tmp/b --overwrite")
...
Writing .../lino_book/projects/dumps/tmp/b/restore.py...
Wrote 3 objects to .../lino_book/projects/dumps/tmp/b/restore.py and siblings.
>>> shell("diff -I OTHER_SOURCES b tmp/b")
...
Third suite¶
Here we test the --max-row-count option. Since we have only three rows, we change the value from its default 50000 to 2 in order to trigger the situation:
>>> shell("python manage_b.py dump2py tmp/c --overwrite -m 2")
...
Writing .../lino_book/projects/dumps/tmp/c/restore.py...
Wrote 3 objects to .../lino_book/projects/dumps/tmp/c/restore.py and siblings.
Verify that the newly created dump is as expected:
>>> shell("diff -I OTHER_SOURCES c tmp/c")
...
Error messages¶
Here we try to call dump2y in some invalid ways, just to
demonstrate the possible error messages.
>>> shell("python manage_a.py dump2py")
...
usage: manage_a.py dump2py [-h] ...
manage_a.py dump2py: error: ...
>>> shell("python manage_a.py dump2py a")
...
CommandError: Specified output_dir ...lino_book/projects/dumps/a already exists. Delete it yourself if you dare!
Bibliography¶
Thanks to Ilian Iliev Django, pytz, NonExistentTimeError and AmbiguousTimeError