dill module documentation

dill module

dill: a utility for serialization of python objects

Based on code written by Oren Tirosh and Armin Ronacher. Extended to a (near) full set of the builtin types (in types module), and coded to the pickle interface, by <mmckerns@caltech.edu>. Initial port to python3 by Jonathan Dobson, continued by mmckerns. Test against “all” python types (Std. Lib. CH 1-15 @ 2.7) by mmckerns. Test against CH16+ Std. Lib. … TBD.

dump(obj, file, protocol=None, byref=None, fmode=None, recurse=None)

pickle an object to a file

dumps(obj, protocol=None, byref=None, fmode=None, recurse=None)

pickle an object to a string

load(file, ignore=None)

unpickle an object from a file

loads(str, ignore=None)

unpickle an object from a string

dump_session(filename='/tmp/session.pkl', main=None, byref=False)

pickle the current state of __main__ to a file

load_session(filename='/tmp/session.pkl', main=None)

update the __main__ module with the state from the session file

class Pickler(*args, **kwds)

Bases: pickle.Pickler

python’s Pickler extended to interpreter sessions

__init__(*args, **kwds)
__module__ = 'dill._dill'
_byref = False
_fmode = 0
_main = None
_recurse = False
_session = False
_strictio = False
dispatch = {<type 'instancemethod'>: <function save_instancemethod0 at 0x7fdfc361a6e0>, <type 'instance'>: <function save_inst at 0x7fdfc86efe60>, <type 'classobj'>: <function save_classobj at 0x7fdfc361a0c8>, <type 'file'>: <function save_file at 0x7fdfc361a578>, <type 'float'>: <function save_float at 0x7fdfc86efa28>, <type 'int'>: <function save_int at 0x7fdfc86ef938>, <type 'list'>: <function save_list at 0x7fdfc86efc80>, <type 'long'>: <function save_long at 0x7fdfc86ef9b0>, <type 'dict'>: <function save_module_dict at 0x7fdfc361a050>, <type 'module'>: <function save_module at 0x7fdfc361ab90>, <type 'NotImplementedType'>: <function save_singleton at 0x7fdfc361aaa0>, <type 'NoneType'>: <function save_none at 0x7fdfc86ef848>, <type 'xrange'>: <function save_singleton at 0x7fdfc361aaa0>, <type 'slice'>: <function save_slice at 0x7fdfc361a8c0>, <type 'ellipsis'>: <function save_singleton at 0x7fdfc361aaa0>, <type 'str'>: <function save_string at 0x7fdfc86efaa0>, <type 'tuple'>: <function save_tuple at 0x7fdfc86efb90>, <type 'super'>: <function save_super at 0x7fdfc361a5f0>, <type 'type'>: <function save_type at 0x7fdfc361ac80>, <type 'weakcallableproxy'>: <function save_weakproxy at 0x7fdfc361ac08>, <type 'weakproxy'>: <function save_weakproxy at 0x7fdfc361ac08>, <type 'weakref'>: <function save_weakref at 0x7fdfc361ab18>, <type 'unicode'>: <function save_unicode at 0x7fdfc86efb18>, <type 'thread.lock'>: <function save_lock at 0x7fdfc361a140>, <type 'bool'>: <function save_bool at 0x7fdfc86ef8c0>, <type 'cell'>: <function save_cell at 0x7fdfc361a7d0>, <type 'code'>: <function save_code at 0x7fdfc3616f50>, <type 'property'>: <function save_property at 0x7fdfc361acf8>, <type 'method-wrapper'>: <function save_instancemethod at 0x7fdfc361a758>, <type 'dictproxy'>: <function save_dictproxy at 0x7fdfc361a848>, <type 'wrapper_descriptor'>: <function save_wrapper_descriptor at 0x7fdfc361a938>, <type 'getset_descriptor'>: <function save_wrapper_descriptor at 0x7fdfc361a938>, <type 'member_descriptor'>: <function save_wrapper_descriptor at 0x7fdfc361a938>, <type 'method_descriptor'>: <function save_wrapper_descriptor at 0x7fdfc361a938>, <type 'staticmethod'>: <function save_classmethod at 0x7fdfc361ade8>, <type 'classmethod'>: <function save_classmethod at 0x7fdfc361ade8>, <type 'function'>: <function save_function at 0x7fdfc361ad70>, <type 'builtin_function_or_method'>: <function save_builtin_method at 0x7fdfc361a668>, <class 'threading._RLock'>: <function save_rlock at 0x7fdfc361a1b8>, <class '_pyio.BufferedReader'>: <function save_file at 0x7fdfc361a500>, <class '_pyio.BufferedWriter'>: <function save_file at 0x7fdfc361a500>, <class '_pyio.BufferedRandom'>: <function save_file at 0x7fdfc361a500>, <class '_pyio.TextIOWrapper'>: <function save_file at 0x7fdfc361a500>, <type 'cStringIO.StringI'>: <function save_stringi at 0x7fdfc361a398>, <type 'cStringIO.StringO'>: <function save_stringo at 0x7fdfc361a410>, <type 'functools.partial'>: <function save_functor at 0x7fdfc361a488>, <type 'operator.attrgetter'>: <function save_attrgetter at 0x7fdfc361a2a8>, <type 'operator.itemgetter'>: <function save_itemgetter at 0x7fdfc361a230>}
settings = {'byref': False, 'fmode': 0, 'ignore': False, 'protocol': 2, 'recurse': False}
class Unpickler(*args, **kwds)

Bases: pickle.Unpickler

python’s Unpickler extended to interpreter sessions and more types

__init__(*args, **kwds)
__module__ = 'dill._dill'
_ignore = False
_main = None
_session = False
find_class(module, name)
settings = {'byref': False, 'fmode': 0, 'ignore': False, 'protocol': 2, 'recurse': False}
register(t)
copy(obj, *args, **kwds)

use pickling to ‘copy’ an object

pickle(t, func)

expose dispatch table for user-created extensions

pickles(obj, exact=False, safe=False, **kwds)

quick check if object pickles with dill

check(obj, *args, **kwds)

check pickling of an object across another process

exception PicklingError

Bases: pickle.PickleError

This exception is raised when an unpicklable object is passed to the dump() method.

__module__ = 'pickle'
exception UnpicklingError

Bases: pickle.PickleError

This exception is raised when there is a problem unpickling an object, such as a security violation.

Note that other exceptions may also be raised during unpickling, including (but not necessarily limited to) AttributeError, EOFError, ImportError, and IndexError.

__module__ = 'pickle'

detect module

Methods for detecting objects leading to pickling failures.

baditems(obj, exact=False, safe=False)

get items in object that fail to pickle

badobjects(obj, depth=0, exact=False, safe=False)

get objects that fail to pickle

badtypes(obj, depth=0, exact=False, safe=False)

get types for objects that fail to pickle

code(func)

get the code object for the given function or method

NOTE: use dill.source.getsource(CODEOBJ) to get the source code

errors(obj, depth=0, exact=False, safe=False)

get errors for objects that fail to pickle

freevars(func)

get objects defined in enclosing code that are referred to by func

returns a dict of {name:object}

getmodule(object, _filename=None, force=False)

get the module of the object

globalvars(func, recurse=True, builtin=False)

get objects defined in global scope that are referred to by func

return a dict of {name:object}

nestedcode(func, recurse=True)

get the code objects for any nested functions (e.g. in a closure)

nestedglobals(func, recurse=True)

get the names of any globals found within func

outermost(func)

get outermost enclosing object (i.e. the outer function in a closure)

NOTE: this is the object-equivalent of getsource(func, enclosing=True)

referredglobals(func, recurse=True, builtin=False)

get the names of objects in the global scope referred to by func

referrednested(func, recurse=True)

get functions defined inside of func (e.g. inner functions in a closure)

NOTE: results may differ if the function has been executed or not. If len(nestedcode(func)) > len(referrednested(func)), try calling func(). If possible, python builds code objects, but delays building functions until func() is called.

trace(boolean)

print a trace through the stack when pickling; useful for debugging

varnames(func)

get names of variables defined by func

returns a tuple (local vars, local vars referrenced by nested functions)

objtypes module

all Python Standard Library object types (currently: CH 1-15 @ 2.7) and some other common object types (i.e. numpy.ndarray)

to load more objects and types, use dill.load_types()

pointers module

parent(obj, objtype, ignore=())
>>> listiter = iter([4,5,6,7])
>>> obj = parent(listiter, list)
>>> obj == [4,5,6,7]  # actually 'is', but don't have handle any longer
True

NOTE: objtype can be a single type (e.g. int or list) or a tuple of types.

WARNING: if obj is a sequence (e.g. list), may produce unexpected results. Parent finds one parent (e.g. the last member of the sequence).

reference(obj)

get memory address of proxy’s reference object

at(address, module=None)

get object located at the given memory address (inverse of id(obj))

parents(obj, objtype, depth=1, ignore=())

Find the chain of referents for obj. Chain will end with obj.

objtype: an object type or tuple of types to search for depth: search depth (e.g. depth=2 is ‘grandparents’) ignore: an object or tuple of objects to ignore in the search

children(obj, objtype, depth=1, ignore=())

Find the chain of referrers for obj. Chain will start with obj.

objtype: an object type or tuple of types to search for depth: search depth (e.g. depth=2 is ‘grandchildren’) ignore: an object or tuple of objects to ignore in the search

NOTE: a common thing to ignore is all globals, ‘ignore=(globals(),)’

NOTE: repeated calls may yield different results, as python stores the last value in the special variable ‘_’; thus, it is often good to execute something to replace ‘_’ (e.g. >>> 1+1).

settings module

global settings for Pickler

source module

Extensions to python’s ‘inspect’ module, which can be used to retrieve information from live python objects. The methods defined in this module are augmented to facilitate access to source code of interactively defined functions and classes, as well as provide access to source code for objects defined in a file.

findsource(object)

Return the entire source file and starting line number for an object. For interactively-defined objects, the ‘file’ is the interpreter’s history.

The argument may be a module, class, method, function, traceback, frame, or code object. The source code is returned as a list of all the lines in the file and the line number indexes a line in that list. An IOError is raised if the source code cannot be retrieved, while a TypeError is raised for objects where the source code is unavailable (e.g. builtins).

getsourcelines(object, lstrip=False, enclosing=False)

Return a list of source lines and starting line number for an object. Interactively-defined objects refer to lines in the interpreter’s history.

The argument may be a module, class, method, function, traceback, frame, or code object. The source code is returned as a list of the lines corresponding to the object and the line number indicates where in the original source file the first line of code was found. An IOError is raised if the source code cannot be retrieved, while a TypeError is raised for objects where the source code is unavailable (e.g. builtins).

If lstrip=True, ensure there is no indentation in the first line of code. If enclosing=True, then also return any enclosing code.

getsource(object, alias='', lstrip=False, enclosing=False, force=False, builtin=False)

Return the text of the source code for an object. The source code for interactively-defined objects are extracted from the interpreter’s history.

The argument may be a module, class, method, function, traceback, frame, or code object. The source code is returned as a single string. An IOError is raised if the source code cannot be retrieved, while a TypeError is raised for objects where the source code is unavailable (e.g. builtins).

If alias is provided, then add a line of code that renames the object. If lstrip=True, ensure there is no indentation in the first line of code. If enclosing=True, then also return any enclosing code. If force=True, catch (TypeError,IOError) and try to use import hooks. If builtin=True, force an import for any builtins

indent(code, spaces=4)

indent a block of code with whitespace (default is 4 spaces)

outdent(code, spaces=None, all=True)

outdent a block of code (default is to strip all leading whitespace)

_wrap(f)

encapsulate a function and it’s __import__

dumpsource(object, alias='', new=False, enclose=True)

‘dump to source’, where the code includes a pickled object.

If new=True and object is a class instance, then create a new instance using the unpacked class source code. If enclose, then create the object inside a function enclosure (thus minimizing any global namespace pollution).

getname(obj, force=False, fqn=False)

get the name of the object. for lambdas, get the name of the pointer

_namespace(obj); return namespace hierarchy (as a list of names)

for the given object. For an instance, find the class hierarchy.

For example:

>>> from functools import partial
>>> p = partial(int, base=2)
>>> _namespace(p)
['functools', 'partial']
getimport(obj, alias='', verify=True, builtin=False, enclosing=False)

get the likely import string for the given object

obj is the object to inspect If verify=True, then test the import string before returning it. If builtin=True, then force an import for builtins where possible. If enclosing=True, get the import for the outermost enclosing callable. If alias is provided, then rename the object on import.

_importable(obj, alias='', source=None, enclosing=False, force=True, builtin=True, lstrip=True)

get an import string (or the source code) for the given object

This function will attempt to discover the name of the object, or the repr of the object, or the source code for the object. To attempt to force discovery of the source code, use source=True, to attempt to force the use of an import, use source=False; otherwise an import will be sought for objects not defined in __main__. The intent is to build a string that can be imported from a python file. obj is the object to inspect. If alias is provided, then rename the object with the given alias.

If source=True, use these options:
If enclosing=True, then also return any enclosing code. If force=True, catch (TypeError,IOError) and try to use import hooks. If lstrip=True, ensure there is no indentation in the first line of code.
If source=False, use these options:
If enclosing=True, get the import for the outermost enclosing callable. If force=True, then don’t test the import string before returning it. If builtin=True, then force an import for builtins where possible.
importable(obj, alias='', source=None, builtin=True)

get an importable string (i.e. source code or the import string) for the given object, including any required objects from the enclosing and global scope

This function will attempt to discover the name of the object, or the repr of the object, or the source code for the object. To attempt to force discovery of the source code, use source=True, to attempt to force the use of an import, use source=False; otherwise an import will be sought for objects not defined in __main__. The intent is to build a string that can be imported from a python file.

obj is the object to inspect. If alias is provided, then rename the object with the given alias. If builtin=True, then force an import for builtins where possible.

isdynamic(obj)

check if object was built in the interpreter

isfrommain(obj)

check if object was built in __main__

temp module

Methods for serialized objects (or source code) stored in temporary files and file-like objects.

dump_source(object, **kwds)

write object source to a NamedTemporaryFile (instead of dill.dump) Loads with “import” or “dill.temp.load_source”. Returns the filehandle.

>>> f = lambda x: x**2
>>> pyfile = dill.temp.dump_source(f, alias='_f')
>>> _f = dill.temp.load_source(pyfile)
>>> _f(4)
16
>>> f = lambda x: x**2
>>> pyfile = dill.temp.dump_source(f, dir='.')
>>> modulename = os.path.basename(pyfile.name).split('.py')[0]
>>> exec('from %s import f as _f' % modulename)
>>> _f(4)
16
Optional kwds:

If ‘alias’ is specified, the object will be renamed to the given string.

If ‘prefix’ is specified, the file name will begin with that prefix, otherwise a default prefix is used.

If ‘dir’ is specified, the file will be created in that directory, otherwise a default directory is used.

If ‘text’ is specified and true, the file is opened in text mode. Else (the default) the file is opened in binary mode. On some operating systems, this makes no difference.

NOTE: Keep the return value for as long as you want your file to exist !

dump(object, **kwds)

dill.dump of object to a NamedTemporaryFile. Loads with “dill.temp.load”. Returns the filehandle.

>>> dumpfile = dill.temp.dump([1, 2, 3, 4, 5])
>>> dill.temp.load(dumpfile)
[1, 2, 3, 4, 5]
Optional kwds:

If ‘suffix’ is specified, the file name will end with that suffix, otherwise there will be no suffix.

If ‘prefix’ is specified, the file name will begin with that prefix, otherwise a default prefix is used.

If ‘dir’ is specified, the file will be created in that directory, otherwise a default directory is used.

If ‘text’ is specified and true, the file is opened in text mode. Else (the default) the file is opened in binary mode. On some operating systems, this makes no difference.

NOTE: Keep the return value for as long as you want your file to exist !

dumpIO_source(object, **kwds)

write object source to a buffer (instead of dill.dump) Loads by with dill.temp.loadIO_source. Returns the buffer object.

>>> f = lambda x:x**2
>>> pyfile = dill.temp.dumpIO_source(f, alias='_f')
>>> _f = dill.temp.loadIO_source(pyfile)
>>> _f(4)
16
Optional kwds:
If ‘alias’ is specified, the object will be renamed to the given string.
dumpIO(object, **kwds)

dill.dump of object to a buffer. Loads with “dill.temp.loadIO”. Returns the buffer object.

>>> dumpfile = dill.temp.dumpIO([1, 2, 3, 4, 5])
>>> dill.temp.loadIO(dumpfile)
[1, 2, 3, 4, 5]
load_source(file, **kwds)

load an object that was stored with dill.temp.dump_source

file: filehandle alias: string name of stored object mode: mode to open the file, one of: {‘r’, ‘rb’}

>>> f = lambda x: x**2
>>> pyfile = dill.temp.dump_source(f, alias='_f')
>>> _f = dill.temp.load_source(pyfile)
>>> _f(4)
16
load(file, **kwds)

load an object that was stored with dill.temp.dump

file: filehandle mode: mode to open the file, one of: {‘r’, ‘rb’}

>>> dumpfile = dill.temp.dump([1, 2, 3, 4, 5])
>>> dill.temp.load(dumpfile)
[1, 2, 3, 4, 5]
loadIO_source(buffer, **kwds)

load an object that was stored with dill.temp.dumpIO_source

buffer: buffer object alias: string name of stored object

>>> f = lambda x:x**2
>>> pyfile = dill.temp.dumpIO_source(f, alias='_f')
>>> _f = dill.temp.loadIO_source(pyfile)
>>> _f(4)
16
loadIO(buffer, **kwds)

load an object that was stored with dill.temp.dumpIO

buffer: buffer object

>>> dumpfile = dill.temp.dumpIO([1, 2, 3, 4, 5])
>>> dill.temp.loadIO(dumpfile)
[1, 2, 3, 4, 5]
capture(*args, **kwds)

builds a context that temporarily replaces the given stream name

>>> with capture('stdout') as out:
...   print "foo!"
...
>>> print out.getvalue()
foo!