DataTaps

Perhaps the funky usage of this code follows closely to the command line usage.

Example usage:

from datatap.dataps import JSONDataTap, StreamDataTap, FileDataTap, ModelDataTap, ZipFileDataTap

#with django models
iostream = BytesIO()
tap = StreamDataTap(JSONDataTap(ModelDataTap([ContentType])))
tap.send(iostream)

#read a model fixture
tap = ModelDataTap(JSONDataTap(FileDataTap('fixtures.json')))
for item in tap:
    print item #a deserialized object with a save method
tap.commit() #save all the items

#write a zip archive of active users and groups
tap = FileDataTap(ZipFileDataTap(ModelDataTap([User.objects.filter(is_active=True), Group])))
tap.send('users.zip')

#comming soon:
tap = S3DataTap(ZipFileDataTap(ModelDataTap([User.objects.filter(is_active=True), Group])))
tap.send('exports/users.zip') #sends it to your s3 storage bucket

DataTap

class datatap.datataps.DataTap(instream=None, filetap=None, domain=None)

get_domain()

Detect the target domain typically based on the instream.

domain

Returns the target domain this data tap will be exporting to

close()

Closes the DataTap

read(size=None)

Read a number of chunks read from this datatap.

get_item_stream()

Returns an iterable based on the instream. By default return the iterable defined by the method get_<domain>_stream where domain is the result of get_domain.

send(datatap)

Send this datatap stream to another datatap or stream

detect_originating_datatap()

Returns the originating datatap class or None if it doesn’t know. If this is stored datatap open for reading then it returns the datatap class that was used for populating this datatap

get_logger()

Returns a logger for logging events

classmethod load_from_command_line(arglist, instream=None)

Retuns an instantiated DataTap with the provided arguments from commandline

classmethod load_from_command_line_for_write(arglist, instream)

Retuns an instantiated DataTap with the provided arguments from commandline

MemoryDataTap

class datatap.datataps.MemoryDataTap(instream=None, filetap=None, domain=None)

Bases: datatap.datataps.base.DataTap

Reads and writes from a que stored in memory

get_item_stream()

Iterates through the objects in the instream

StreamDataTap

class datatap.datataps.StreamDataTap(instream=None, filetap=None, domain=None)

Bases: datatap.datataps.base.DataTap

A data tap that fowards the instream or saves the instream to a real stream

FileDataTap

class datatap.datataps.FileDataTap(filename=None, **kwargs)

Bases: datatap.datataps.streams.StreamDataTap

A stream data tap that opens files for io

FileDT(JSONDT(ModelDT)).write(filename) => write to filename FileDT(filename=filename) => bytes stream

JSONDataTap

class datatap.datataps.JSONDataTap(instream=None, filetap=None, domain=None)

Bases: datatap.datataps.base.DataTap

A data tap that converts primitive objects to and from a text datatap

JSONDT(ModelDT) => json representation of models JSONDT(FileDT(filename)) => decoded json from filename

ZipFileDataTap

class datatap.datataps.ZipFileDataTap(instream=None, filetap=None, domain=None)

Bases: datatap.datataps.base.DataTap

Reads and writes objects from a zipfile with asset storage support

FileDT(ZipDT(ModelDT)).write(filename) => write models to filename ModelDT(ZipDT(FileDT(filename))) => read from filename into models

ModelDataTap

class datatap.datataps.ModelDataTap(instream=None, use_natural_keys=True, track_uncommitted=True, **kwargs)

Bases: datatap.datataps.base.DataTap

Reads and writes from Django’s ORM

ModelDT([MyModel, Queryset, ModelInstance]) => primitive representation of sources ModelDT(ZipDT(...)) => deserialized objects from the zip datatap ModelDT(ZipDT(...)).commit() => save the deserialized objects

get_primitive_stream(instream)

Convert various model sources to primitive objects

get_deserialized_model_stream(instream)

Convert primitive objects to deserialized model instances

get_model_stream(instream)

Convert primitive objects to saved model instances