dill module documentation

detect module

Methods for detecting objects leading to pickling failures.

baditems(obj, exact=False, safe=False): get items in object that fail to pickle

badobjects(obj, depth=0, exact=False, safe=False): get objects that fail to pickle

badtypes(obj, depth=0, exact=False, safe=False): get types for objects that fail to pickle

code(func)

get the code object for the given function or method

NOTE: use dill.source.getsource(CODEOBJ) to get the source code

errors(obj, depth=0, exact=False, safe=False): get errors for objects that fail to pickle

freevars(func)

get objects defined in enclosing code that are referred to by func

returns a dict of {name:object}

getmodule(object, _filename=None, force=False): get the module of the object

globalvars(func, recurse=True, builtin=False)

get objects defined in global scope that are referred to by func

return a dict of {name:object}

nestedcode(func, recurse=True): get the code objects for any nested functions (e.g. in a closure)

nestedglobals(func, recurse=True): get the names of any globals found within func

outermost(func)

get outermost enclosing object (i.e. the outer function in a closure)

NOTE: this is the object-equivalent of getsource(func, enclosing=True)

referredglobals(func, recurse=True, builtin=False): get the names of objects in the global scope referred to by func

referrednested(func, recurse=True)

get functions defined inside of func (e.g. inner functions in a closure)

NOTE: results may differ if the function has been executed or not. If len(nestedcode(func)) > len(referrednested(func)), try calling func(). If possible, python builds code objects, but delays building functions until func() is called.

trace(arg=None, *, mode='a')

print a trace through the stack when pickling; useful for debugging

With a single boolean argument, enable or disable the tracing.

Example usage:

>>> import dill
>>> dill.detect.trace(True)
>>> dill.dump_session()

Alternatively, trace() can be used as a context manager. With no arguments, it just takes care of restoring the tracing state on exit. Either a file handle, or a file name and (optionally) a file mode may be specitfied to redirect the tracing output in the with block context. A log function is yielded by the manager so the user can write extra information to the file.

Example usage:

>>> from dill import detect
>>> D = {'a': 42, 'b': {'x': None}}
>>> with detect.trace():
>>>     dumps(D)
┬ D2: <dict object at 0x7f2721804800>
├┬ D2: <dict object at 0x7f27217f5c40>
│└ # D2 [8 B]
└ # D2 [22 B]
>>> squared = lambda x: x**2
>>> with detect.trace('output.txt', mode='w') as log:
>>>     log("> D = %r", D)
>>>     dumps(D)
>>>     log("> squared = %r", squared)
>>>     dumps(squared)

Parameters:

arg (bool | TextIO | str | PathLike | None) – a boolean value, or an optional file-like or path-like object for the context manager
mode (str) – mode string for open() if a file name is passed as the first argument

Return type:

None

varnames(func)

get names of variables defined by func

returns a tuple (local vars, local vars referrenced by nested functions)

logger module

Logging utilities for dill.

The ‘logger’ object is dill’s top-level logger.

The ‘adapter’ object wraps the logger and implements a ‘trace()’ method that generates a detailed tree-style trace for the pickling call at log level INFO.

The ‘trace()’ function sets and resets dill’s logger log level, enabling and disabling the pickling trace.

The trace shows a tree structure depicting the depth of each object serialized with dill save functions, but not the ones that use save functions from ‘pickle._Pickler.dispatch’. If the information is available, it also displays the size in bytes that the object contributed to the pickle stream (including its child objects). Sample trace output:

>>> import dill, dill.tests
>>> dill.detect.trace(True)
>>> dill.dump_session(main=dill.tests)
┬ M1: <module 'dill.tests' from '.../dill/tests/__init__.py'>
├┬ F2: <function _import_module at 0x7f0d2dce1b80>
│└ # F2 [32 B]
├┬ D2: <dict object at 0x7f0d2e98a540>
│├┬ T4: <class '_frozen_importlib.ModuleSpec'>
││└ # T4 [35 B]
│├┬ D2: <dict object at 0x7f0d2ef0e8c0>
││├┬ T4: <class '_frozen_importlib_external.SourceFileLoader'>
│││└ # T4 [50 B]
││├┬ D2: <dict object at 0x7f0d2e988a40>
│││└ # D2 [84 B]
││└ # D2 [413 B]
│└ # D2 [763 B]
└ # M1 [813 B]

objtypes module

all Python Standard Library object types (currently: CH 1-15 @ 2.7) and some other common object types (i.e. numpy.ndarray)

to load more objects and types, use dill.load_types()

pointers module

at(address, module=None): get object located at the given memory address (inverse of id(obj))

children(obj, objtype, depth=1, ignore=())

Find the chain of referrers for obj. Chain will start with obj.

objtype: an object type or tuple of types to search for depth: search depth (e.g. depth=2 is ‘grandchildren’) ignore: an object or tuple of objects to ignore in the search

NOTE: a common thing to ignore is all globals, ‘ignore=(globals(),)’

NOTE: repeated calls may yield different results, as python stores the last value in the special variable ‘_’; thus, it is often good to execute something to replace ‘_’ (e.g. >>> 1+1).

parent(obj, objtype, ignore=())

>>> listiter = iter([4,5,6,7])
>>> obj = parent(listiter, list)
>>> obj == [4,5,6,7]  # actually 'is', but don't have handle any longer
True

NOTE: objtype can be a single type (e.g. int or list) or a tuple of types.

WARNING: if obj is a sequence (e.g. list), may produce unexpected results. Parent finds one parent (e.g. the last member of the sequence).

parents(obj, objtype, depth=1, ignore=())

Find the chain of referents for obj. Chain will end with obj.

objtype: an object type or tuple of types to search for depth: search depth (e.g. depth=2 is ‘grandparents’) ignore: an object or tuple of objects to ignore in the search

reference(obj): get memory address of proxy’s reference object

session module

Pickle and restore the intepreter session.

dump_module(filename=None, module=None, refimported=False, **kwds)

Pickle the current state of __main__ or another module to a file.

Save the contents of __main__ (e.g. from an interactive interpreter session), an imported module, or a module-type object (e.g. built with ModuleType), to a file. The pickled module can then be restored with the function load_module().

Parameters:

filename (str | PathLike | None) – a path-like object or a writable stream. If None (the default), write to a named file in a temporary directory.
module (module | str | None) – a module object or the name of an importable module. If None (the default), __main__ is saved.
refimported (bool) – if True, all objects identified as having been imported into the module’s namespace are saved by reference. Note: this is similar but independent from dill.settings[`byref`], as refimported refers to virtually all imported objects, while byref only affects select objects.
**kwds – extra keyword arguments passed to Pickler().

Raises:

PicklingError – if pickling fails.

Return type:

None

Examples

Save current interpreter session state:

>>> import dill
>>> squared = lambda x: x*x
>>> dill.dump_module() # save state of __main__ to /tmp/session.pkl

Save the state of an imported/importable module:

>>> import dill
>>> import pox
>>> pox.plus_one = lambda x: x+1
>>> dill.dump_module('pox_session.pkl', module=pox)

Save the state of a non-importable, module-type object:

>>> import dill
>>> from types import ModuleType
>>> foo = ModuleType('foo')
>>> foo.values = [1,2,3]
>>> import math
>>> foo.sin = math.sin
>>> dill.dump_module('foo_session.pkl', module=foo, refimported=True)

Restore the state of the saved modules:

>>> import dill
>>> dill.load_module()
>>> squared(2)
4
>>> pox = dill.load_module('pox_session.pkl')
>>> pox.plus_one(1)
2
>>> foo = dill.load_module('foo_session.pkl')
>>> [foo.sin(x) for x in foo.values]
[0.8414709848078965, 0.9092974268256817, 0.1411200080598672]

Use refimported to save imported objects by reference:

>>> import dill
>>> from html.entities import html5
>>> type(html5), len(html5)
(dict, 2231)
>>> import io
>>> buf = io.BytesIO()
>>> dill.dump_module(buf) # saves __main__, with html5 saved by value
>>> len(buf.getvalue()) # pickle size in bytes
71665
>>> buf = io.BytesIO()
>>> dill.dump_module(buf, refimported=True) # html5 saved by reference
>>> len(buf.getvalue())
438

Changed in version 0.3.6: Function dump_session() was renamed to dump_module(). Parameters main and byref were renamed to module and refimported, respectively.

Note

Currently, dill.settings['byref'] and dill.settings['recurse'] don’t apply to this function.

dump_session(filename=None, main=None, byref=False, **kwds)

Pickle the current state of __main__ or another module to a file.

Save the contents of __main__ (e.g. from an interactive interpreter session), an imported module, or a module-type object (e.g. built with ModuleType), to a file. The pickled module can then be restored with the function load_module().

Parameters:

filename – a path-like object or a writable stream. If None (the default), write to a named file in a temporary directory.
module – a module object or the name of an importable module. If None (the default), __main__ is saved.
refimported – if True, all objects identified as having been imported into the module’s namespace are saved by reference. Note: this is similar but independent from dill.settings[`byref`], as refimported refers to virtually all imported objects, while byref only affects select objects.
**kwds – extra keyword arguments passed to Pickler().

Raises:

PicklingError – if pickling fails.

Examples

Save current interpreter session state:

>>> import dill
>>> squared = lambda x: x*x
>>> dill.dump_module() # save state of __main__ to /tmp/session.pkl

Save the state of an imported/importable module:

>>> import dill
>>> import pox
>>> pox.plus_one = lambda x: x+1
>>> dill.dump_module('pox_session.pkl', module=pox)

Save the state of a non-importable, module-type object:

>>> import dill
>>> from types import ModuleType
>>> foo = ModuleType('foo')
>>> foo.values = [1,2,3]
>>> import math
>>> foo.sin = math.sin
>>> dill.dump_module('foo_session.pkl', module=foo, refimported=True)

Restore the state of the saved modules:

>>> import dill
>>> dill.load_module()
>>> squared(2)
4
>>> pox = dill.load_module('pox_session.pkl')
>>> pox.plus_one(1)
2
>>> foo = dill.load_module('foo_session.pkl')
>>> [foo.sin(x) for x in foo.values]
[0.8414709848078965, 0.9092974268256817, 0.1411200080598672]

Use refimported to save imported objects by reference:

>>> import dill
>>> from html.entities import html5
>>> type(html5), len(html5)
(dict, 2231)
>>> import io
>>> buf = io.BytesIO()
>>> dill.dump_module(buf) # saves __main__, with html5 saved by value
>>> len(buf.getvalue()) # pickle size in bytes
71665
>>> buf = io.BytesIO()
>>> dill.dump_module(buf, refimported=True) # html5 saved by reference
>>> len(buf.getvalue())
438

Changed in version 0.3.6: Function dump_session() was renamed to dump_module(). Parameters main and byref were renamed to module and refimported, respectively.

Note

Currently, dill.settings['byref'] and dill.settings['recurse'] don’t apply to this function.

load_module(filename=None, module=None, **kwds)

Update the selected module (default is __main__) with the state saved at filename.

Restore a module to the state saved with dump_module(). The saved module can be __main__ (e.g. an interpreter session), an imported module, or a module-type object (e.g. created with ModuleType).

When restoring the state of a non-importable module-type object, the current instance of this module may be passed as the argument main. Otherwise, a new instance is created with ModuleType and returned.

Parameters:

filename (str | PathLike | None) – a path-like object or a readable stream. If None (the default), read from a named file in a temporary directory.
module (module | str | None) – a module object or the name of an importable module; the module name and kind (i.e. imported or non-imported) must match the name and kind of the module stored at filename.
**kwds – extra keyword arguments passed to Unpickler().

Raises:

UnpicklingError – if unpickling fails.
ValueError – if the argument main and module saved at filename are incompatible.

Returns:

A module object, if the saved module is not __main__ or a module instance wasn’t provided with the argument main.

Return type:

module | None

Examples

Save the state of some modules:

>>> import dill
>>> squared = lambda x: x*x
>>> dill.dump_module() # save state of __main__ to /tmp/session.pkl
>>>
>>> import pox # an imported module
>>> pox.plus_one = lambda x: x+1
>>> dill.dump_module('pox_session.pkl', module=pox)
>>>
>>> from types import ModuleType
>>> foo = ModuleType('foo') # a module-type object
>>> foo.values = [1,2,3]
>>> import math
>>> foo.sin = math.sin
>>> dill.dump_module('foo_session.pkl', module=foo, refimported=True)

Restore the state of the interpreter:

>>> import dill
>>> dill.load_module() # updates __main__ from /tmp/session.pkl
>>> squared(2)
4

Load the saved state of an importable module:

>>> import dill
>>> pox = dill.load_module('pox_session.pkl')
>>> pox.plus_one(1)
2
>>> import sys
>>> pox in sys.modules.values()
True

Load the saved state of a non-importable module-type object:

>>> import dill
>>> foo = dill.load_module('foo_session.pkl')
>>> [foo.sin(x) for x in foo.values]
[0.8414709848078965, 0.9092974268256817, 0.1411200080598672]
>>> import math
>>> foo.sin is math.sin # foo.sin was saved by reference
True
>>> import sys
>>> foo in sys.modules.values()
False

Update the state of a non-importable module-type object:

>>> import dill
>>> from types import ModuleType
>>> foo = ModuleType('foo')
>>> foo.values = ['a','b']
>>> foo.sin = lambda x: x*x
>>> dill.load_module('foo_session.pkl', module=foo)
>>> [foo.sin(x) for x in foo.values]
[0.8414709848078965, 0.9092974268256817, 0.1411200080598672]

Changed in version 0.3.6: Function load_session() was renamed to load_module(). Parameter main was renamed to module.

settings module

global settings for Pickler

source module

Extensions to python’s ‘inspect’ module, which can be used to retrieve information from live python objects. The methods defined in this module are augmented to facilitate access to source code of interactively defined functions and classes, as well as provide access to source code for objects defined in a file.

_importable(obj, alias='', source=None, enclosing=False, force=True, builtin=True, lstrip=True)

get an import string (or the source code) for the given object

This function will attempt to discover the name of the object, or the repr of the object, or the source code for the object. To attempt to force discovery of the source code, use source=True, to attempt to force the use of an import, use source=False; otherwise an import will be sought for objects not defined in __main__. The intent is to build a string that can be imported from a python file. obj is the object to inspect. If alias is provided, then rename the object with the given alias.

If source=True, use these options:: If enclosing=True, then also return any enclosing code. If force=True, catch (TypeError,IOError) and try to use import hooks. If lstrip=True, ensure there is no indentation in the first line of code.
If source=False, use these options:: If enclosing=True, get the import for the outermost enclosing callable. If force=True, then don’t test the import string before returning it. If builtin=True, then force an import for builtins where possible.

_namespace(obj); return namespace hierarchy (as a list of names)

for the given object. For an instance, find the class hierarchy.

For example:

>>> from functools import partial
>>> p = partial(int, base=2)
>>> _namespace(p)
['functools', 'partial']

_wrap(f): encapsulate a function and it’s __import__

dumpsource(object, alias='', new=False, enclose=True)

‘dump to source’, where the code includes a pickled object.

If new=True and object is a class instance, then create a new instance using the unpacked class source code. If enclose, then create the object inside a function enclosure (thus minimizing any global namespace pollution).

findsource(object)

Return the entire source file and starting line number for an object. For interactively-defined objects, the ‘file’ is the interpreter’s history.

The argument may be a module, class, method, function, traceback, frame, or code object. The source code is returned as a list of all the lines in the file and the line number indexes a line in that list. An IOError is raised if the source code cannot be retrieved, while a TypeError is raised for objects where the source code is unavailable (e.g. builtins).

getimport(obj, alias='', verify=True, builtin=False, enclosing=False)

get the likely import string for the given object

obj is the object to inspect If verify=True, then test the import string before returning it. If builtin=True, then force an import for builtins where possible. If enclosing=True, get the import for the outermost enclosing callable. If alias is provided, then rename the object on import.

getname(obj, force=False, fqn=False): get the name of the object. for lambdas, get the name of the pointer

getsource(object, alias='', lstrip=False, enclosing=False, force=False, builtin=False)

Return the text of the source code for an object. The source code for interactively-defined objects are extracted from the interpreter’s history.

The argument may be a module, class, method, function, traceback, frame, or code object. The source code is returned as a single string. An IOError is raised if the source code cannot be retrieved, while a TypeError is raised for objects where the source code is unavailable (e.g. builtins).

If alias is provided, then add a line of code that renames the object. If lstrip=True, ensure there is no indentation in the first line of code. If enclosing=True, then also return any enclosing code. If force=True, catch (TypeError,IOError) and try to use import hooks. If builtin=True, force an import for any builtins

getsourcelines(object, lstrip=False, enclosing=False)

Return a list of source lines and starting line number for an object. Interactively-defined objects refer to lines in the interpreter’s history.

The argument may be a module, class, method, function, traceback, frame, or code object. The source code is returned as a list of the lines corresponding to the object and the line number indicates where in the original source file the first line of code was found. An IOError is raised if the source code cannot be retrieved, while a TypeError is raised for objects where the source code is unavailable (e.g. builtins).

If lstrip=True, ensure there is no indentation in the first line of code. If enclosing=True, then also return any enclosing code.

importable(obj, alias='', source=None, builtin=True)

get an importable string (i.e. source code or the import string) for the given object, including any required objects from the enclosing and global scope

This function will attempt to discover the name of the object, or the repr of the object, or the source code for the object. To attempt to force discovery of the source code, use source=True, to attempt to force the use of an import, use source=False; otherwise an import will be sought for objects not defined in __main__. The intent is to build a string that can be imported from a python file.

obj is the object to inspect. If alias is provided, then rename the object with the given alias. If builtin=True, then force an import for builtins where possible.

indent(code, spaces=4): indent a block of code with whitespace (default is 4 spaces)

isdynamic(obj): check if object was built in the interpreter

isfrommain(obj): check if object was built in __main__

outdent(code, spaces=None, all=True): outdent a block of code (default is to strip all leading whitespace)

temp module

Methods for serialized objects (or source code) stored in temporary files and file-like objects.

capture(stream='stdout')

builds a context that temporarily replaces the given stream name

>>> with capture('stdout') as out:
...   print ("foo!")
...
>>> print (out.getvalue())
foo!

dump(object, **kwds)

dill.dump of object to a NamedTemporaryFile. Loads with “dill.temp.load”. Returns the filehandle.

>>> dumpfile = dill.temp.dump([1, 2, 3, 4, 5])
>>> dill.temp.load(dumpfile)
[1, 2, 3, 4, 5]

Optional kwds:

If ‘suffix’ is specified, the file name will end with that suffix, otherwise there will be no suffix.

If ‘prefix’ is specified, the file name will begin with that prefix, otherwise a default prefix is used.

If ‘dir’ is specified, the file will be created in that directory, otherwise a default directory is used.

If ‘text’ is specified and true, the file is opened in text mode. Else (the default) the file is opened in binary mode. On some operating systems, this makes no difference.

NOTE: Keep the return value for as long as you want your file to exist !

dumpIO(object, **kwds)

dill.dump of object to a buffer. Loads with “dill.temp.loadIO”. Returns the buffer object.

>>> dumpfile = dill.temp.dumpIO([1, 2, 3, 4, 5])
>>> dill.temp.loadIO(dumpfile)
[1, 2, 3, 4, 5]

dumpIO_source(object, **kwds)

write object source to a buffer (instead of dill.dump) Loads by with dill.temp.loadIO_source. Returns the buffer object.

>>> f = lambda x:x**2
>>> pyfile = dill.temp.dumpIO_source(f, alias='_f')
>>> _f = dill.temp.loadIO_source(pyfile)
>>> _f(4)
16

Optional kwds:: If ‘alias’ is specified, the object will be renamed to the given string.

dump_source(object, **kwds)

write object source to a NamedTemporaryFile (instead of dill.dump) Loads with “import” or “dill.temp.load_source”. Returns the filehandle.

>>> f = lambda x: x**2
>>> pyfile = dill.temp.dump_source(f, alias='_f')
>>> _f = dill.temp.load_source(pyfile)
>>> _f(4)
16

>>> f = lambda x: x**2
>>> pyfile = dill.temp.dump_source(f, dir='.')
>>> modulename = os.path.basename(pyfile.name).split('.py')[0]
>>> exec('from %s import f as _f' % modulename)
>>> _f(4)
16

Optional kwds:

If ‘alias’ is specified, the object will be renamed to the given string.

If ‘prefix’ is specified, the file name will begin with that prefix, otherwise a default prefix is used.

If ‘dir’ is specified, the file will be created in that directory, otherwise a default directory is used.

If ‘text’ is specified and true, the file is opened in text mode. Else (the default) the file is opened in binary mode. On some operating systems, this makes no difference.

NOTE: Keep the return value for as long as you want your file to exist !

load(file, **kwds)

load an object that was stored with dill.temp.dump

file: filehandle mode: mode to open the file, one of: {‘r’, ‘rb’}

>>> dumpfile = dill.temp.dump([1, 2, 3, 4, 5])
>>> dill.temp.load(dumpfile)
[1, 2, 3, 4, 5]

loadIO(buffer, **kwds)

load an object that was stored with dill.temp.dumpIO

buffer: buffer object

>>> dumpfile = dill.temp.dumpIO([1, 2, 3, 4, 5])
>>> dill.temp.loadIO(dumpfile)
[1, 2, 3, 4, 5]

loadIO_source(buffer, **kwds)

load an object that was stored with dill.temp.dumpIO_source

buffer: buffer object alias: string name of stored object

>>> f = lambda x:x**2
>>> pyfile = dill.temp.dumpIO_source(f, alias='_f')
>>> _f = dill.temp.loadIO_source(pyfile)
>>> _f(4)
16

load_source(file, **kwds)

load an object that was stored with dill.temp.dump_source

file: filehandle alias: string name of stored object mode: mode to open the file, one of: {‘r’, ‘rb’}

>>> f = lambda x: x**2
>>> pyfile = dill.temp.dump_source(f, alias='_f')
>>> _f = dill.temp.load_source(pyfile)
>>> _f(4)
16