shiv 🔪
====================
Shiv is a command line utility for building fully self-contained Python zipapps as outlined in `PEP 441 `_
but with all their dependencies included!
Shiv's primary goal is making distributing Python applications fast & easy.
How it works
------------
Internally, shiv includes two major components: a *builder* and a small *bootstrap* runtime.
Building
^^^^^^^^
In order to build self-contained, single-artifact executables, shiv leverages ``pip`` to stage your project's dependencies
and then uses the features described in `PEP 441 `_ to create a "zipapp".
The primary feature of PEP 441 that ``shiv`` uses is Python's ability to implicitly execute a `__main__.py` file inside of a zip archive.
Here's an example of the feature in action:
.. code-block:: sh
$ echo "print('hello world')" >> __main__.py
$ zip archive.zip __main__.py
adding: __main__.py (stored 0%)
$ python3 ./archive.zip
hello world
``shiv`` expands on this functionality by packing your dependencies into the same zip and adding a specialized `__main__.py` that instructs the Python interpreter to
unpack those dependencies to a known location. Then, at runtime, adds those dependencies to your interpreter's search path, and that's it!
.. note::
"Conventional" zipapps don't include any dependencies, which is what sets shiv apart from the stdlib zipapp module.
``shiv`` accepts only a few command line parameters of its own, `described here `_, and under the covers, **any unprocessed parameters are
delegated to** ``pip install``. This allows users to fully leverage all the functionality that pip provides.
For example, if you wanted to create an executable for ``flake8``, you'd specify the required
dependencies (in this case, simply ``flake8``), the callable (either via ``-e`` for a setuptools-style entry
point or ``-c`` for a bare console_script name), and the output file:
.. code-block:: sh
$ shiv -c flake8 -o ~/bin/flake8 flake8
Let's break this command down,
* ``shiv`` is the command itself.
* ``-c flake8`` specifies the ``console_script`` for flake8 (`defined here `_)
* ``-o ~/bin/flake8`` specifies the ``outfile``
* ``flake8`` is a dependency (this portion of the command is delegated to ``pip install``)
This creates an executable (``~/bin/flake8``) containing all the dependencies specified (``flake8``)
that, when invoked, executes the provided console_script (``flake8``)!
If you were to omit the entry point/console script flag, invoking the resulting executable would drop you into an interpreter that
is bootstrapped with the dependencies you've specified. This can be useful for creating a single-artifact executable
Python environment:
.. code-block:: sh
$ shiv httpx -o httpx.pyz --quiet
$ ./httpx.pyz
Python 3.7.7 (default, Mar 10 2020, 16:11:21)
[Clang 11.0.0 (clang-1100.0.33.12)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import httpx
>>> httpx.get("https://shiv.readthedocs.io")
This is particularly useful for running scripts without needing to create a virtual environment or contaminate your Python
environment, since the ``pyz`` files can be used as a shebang!
.. code-block:: sh
$ cat << EOF > tryme.py
> #!/usr/bin/env httpx.pyz
>
> import httpx
> url = "https://shiv.readthedocs.io"
> response = httpx.get(url)
> print(f"Got {response.status_code} from {url}!")
>
> EOF
$ chmod +x tryme.py
$ ./tryme.py
Got 200 from https://shiv.readthedocs.io!
Bootstrapping
^^^^^^^^^^^^^
As mentioned above, when you run an executable created with ``shiv``, a special bootstrap function is called.
This function unpacks the dependencies into a uniquely named subdirectory of ``~/.shiv`` and then runs your entry point
(or interactive interpreter) with those dependencies added to your interpreter's search path (``sys.path``).
To improve performance, once the dependencies have been extracted to disk, any further invocations will re-use the 'cached'
site-packages unless they are deleted or moved.
.. note::
Dependencies are extracted (rather than loaded into memory from the zipapp itself) for two reasons.
**1.) Because of limitations of third-party and binary dependencies.**
Just as an example, shared objects loaded via the dlopen syscall require a regular filesystem.
In addition, many libraries also expect a filesystem in order to do things like building paths via ``__file__`` (which doesn't work when a module is imported from a zip), etc.
To learn more, check out this resource about the setuptools `"zip_safe" flag `_.
**2.) Performance reasons**
Decompressing files takes time, and if we loaded the dependencies from the zip file every time it would significantly slow down invocation speed.
Preamble
^^^^^^^^
As an application packager, you may want to run some sanity checks or clean up tasks when users execute a pyz.
For such a use case, ``shiv`` provides a ``--preamble`` flag.
Any executable script passed to that flag will be packed into the zipapp and invoked during bootstrapping (*after* extracting dependencies but *before* invoking an entry point / console script).
If the preamble file is written in Python (e.g. ends in ``.py``) then shiv will inject three variables into the runtime that may be useful to preamble authors:
* ``archive``: (a string) path to the current PYZ file
* ``env``: an instance of the `Environment `_ object.
* ``site_packages``: a :py:class:`pathlib.Path` of the directory where the current PYZ's site_packages were extracted to during bootstrap.
For an example, a preamble file that cleans up prior extracted ``~/.shiv`` directories might look like:
.. code-block:: py
#!/usr/bin/env python3
import shutil
from pathlib import Path
# These variables are injected by shiv.bootstrap
site_packages: Path
env: "shiv.bootstrap.environment.Environment"
# Get a handle of the current PYZ's site_packages directory
current = site_packages.parent
# The parent directory of the site_packages directory is our shiv cache
cache_path = current.parent
name, build_id = current.name.split('_')
if __name__ == "__main__":
for path in cache_path.iterdir():
if path.name.startswith(f"{name}_") and not path.name.endswith(build_id):
shutil.rmtree(path)
Hello World
^^^^^^^^^^^
Here's an example of how to set up a hello-world executable using ``shiv``.
First, create a new project:
.. code-block:: sh
$ mkdir hello-world
$ cd hello-world
Add some code.
.. code-block:: python
:caption: hello.py
def main():
print("Hello world")
if __name__ == "__main__":
main()
Second, create a Python package using your preferred workflow (for this example, I'll simply create a minimal ``setup.py`` file).
.. code-block:: python
:caption: setup.py
from setuptools import setup
setup(
name="hello-world",
version="0.0.1",
description="Greet the world.",
py_modules=["hello"],
entry_points={
"console_scripts": ["hello=hello:main"],
},
)
That's it! We now have a proper Python package, so we can use ``shiv`` to create a single-artifact executable for it.
.. code-block:: sh
$ shiv -c hello -o hello .
.. note::
Notice the ``.`` at the end of the ``shiv`` invocation. That is referring to the local package that we just created.
You can think of it as analogous to running ``pip install .``
That's it! Our example should now execute as expected.
.. code-block:: sh
$ ./hello
Hello world
Influencing Runtime
-------------------
Whenever you are creating a zipapp with ``shiv``, you can specify a few flags that influence the runtime.
For example, the ``-c/--console-script`` and ``-e/--entry-point`` options already mentioned in this doc.
To see the full list of command line options, see this page.
In addition to options that are settable during zipapp creation, there are a number of environment variables
you can specify to influence a zipapp created with shiv at run time.
SHIV_ROOT
^^^^^^^^^
This should be populated with a full path, it overrides ``~/.shiv`` as the default base dir for shiv's extraction cache.
This is useful if you want to collect the contents of a zipapp to inspect them, or if you want to make a quick edit to
a source file, but don't want to taint the extraction cache.
SHIV_INTERPRETER
^^^^^^^^^^^^^^^^
This is a boolean that bypasses and console_script or entry point baked into your pyz. Useful for
dropping into an interactive session in the environment of a built cli utility.
SHIV_ENTRY_POINT / SHIV_MODULE
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. note:: Same functionality as ``-e/--entry-point`` at build time
This should be populated with a setuptools-style callable, e.g. "module.main:main". This will
execute the pyz with whatever callable entry point you supply. Useful for sharing a single pyz
across many callable 'scripts'.
SHIV_CONSOLE_SCRIPT
^^^^^^^^^^^^^^^^^^^
.. note:: Same functionality as ``-c/--console-script` at build time
Similar to the SHIV_ENTRY_POINT and SHIV_MODULE environment variables, SHIV_CONSOLE_SCRIPT overrides any value
provided at build time.
SHIV_FORCE_EXTRACT
^^^^^^^^^^^^^^^^^^
This forces re-extraction of dependencies even if they've already been extracted. If you make
hotfixes/modifications to the 'cached' dependencies, this will overwrite them.
SHIV_EXTEND_PYTHONPATH
^^^^^^^^^^^^^^^^^^^^^^
.. note:: Same functionality as ``-E/--extend-pythonpath`` at build time.
This is a boolean that adds the modules bundled into the zipapp into the ``PYTHONPATH`` environment
variable. It is not needed for most applications, but if an application calls Python as a
subprocess, expecting to be able to import the modules bundled in the zipapp, this will allow it
to do so successfully.
Reproducibility
^^^^^^^^^^^^^^^
``shiv`` supports the ability to create reproducible artifacts. By using the ``--reproducible`` command line option or
by setting the ``SOURCE_DATE_EPOCH`` environment variable during zipapp creation. When this option is selected, if the
inputs do not change, the output should be idempotent.
For more information, see https://reproducible-builds.org/.
Table of Contents
=================
.. toctree::
:maxdepth: 2
cli-reference
history
api
django
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`