April 27, 2023·4 min read·engineeringsqlitedatasettepythonpackaging

Datasette declares the packaging tools its runtime uses

Datasette 0.64.3 adds pip and setuptools as dependencies after Rye exposes two assumptions about its environment. Startup and plugin installation fail for different reasons.

Installing a Python application does not establish that every packaging tool it expects is present at runtime. Simon Willison's April 27 report about installing Datasette with Rye demonstrates two separate failures. Datasette initially could not start because it could not import pkg_resources. After adding the missing dependency, its plugin installation command failed because it could not import pip.

Datasette 0.64.3 fixes both by declaring setuptools and pip as explicit dependencies. The release is small, but the underlying issue is useful for anyone distributing an application whose operational commands depend on Python packaging machinery. A tool available in the developer's environment can still be absent from the environment in which a user runs the installed application.

I like SQLite partly because its deployment model can avoid a separate database service. Tools that publish SQLite data still have their own runtime requirements. Keeping the database operationally small does not remove the need to state those requirements accurately.

A successful install followed by an import failure

The original report begins with rye install datasette. Invoking the installed Datasette shim then raises ModuleNotFoundError for pkg_resources. The traceback passes through the CLI module into the application module before reaching the failed import. The application therefore fails during startup, before it can serve a database.

Willison identifies the missing setuptools package as the likely cause and adds it to the dependencies. That is a different repair from asking Rye users to discover and install an extra package themselves. The application already depends on that import, so its package metadata needs to describe the requirement.

The 0.64.3 setup file records both additions in install_requires. It also distinguishes runtime dependencies from documentation and test extras. Setuptools is now part of the environment requested for an ordinary Datasette installation, regardless of whether a user also installs those extras.

That placement explains why this is a runtime packaging fix. The setup file uses setuptools to define the package, but the installed application also imports functionality associated with it. Providing enough machinery to construct or install a distribution does not necessarily provide everything that its execution will import.

Plugin installation has a second dependency

The issue did not end when startup worked. Willison next ran the Datasette command for installing a GraphQL plugin. That command failed with ImportError: No module named pip. The first correction made the main application executable, but a later operation reached another undeclared dependency.

The versioned CLI implementation makes the mechanism explicit. Its install command builds arguments beginning with pip install, assigns the argument list to sys.argv, and executes the pip module through run_module. The code invokes Python's module machinery inside the Datasette process.

This choice helps target the same environment as the application. A bare external pip executable could belong to a different Python installation. Datasette's command describes its purpose as installing plugins and packages into the same environment as Datasette. Loading the module gives that operation a direct dependency on pip being importable there.

The uninstall command has the corresponding requirement. It constructs pip uninstall arguments and invokes the same module. These commands are ordinary application capabilities, so pip cannot be treated as an incidental tool used only before Datasette starts.

After adding pip as a dependency, Willison repeats the installation and confirms that the plugin appears in the output of datasette plugins. The report therefore documents more than a successful import. It follows the user operation through installation into plugin discovery.

Dependencies follow the executable code

The plugin implementation at this release also imports pkg_resources. It uses resource functions when finding static and template directories for external plugins. The relationship to setuptools is present in installed runtime code, including code that helps extensions supply their assets.

This does not imply that every plugin uses the same resources or that all plugins need their own packaging tools. The relevant observation is narrower. Datasette's implementation imports and calls these facilities while handling plugins, so the application's dependency declaration must cover that behavior.

My preference is for an installation to contain the tools required by its advertised commands. A command that succeeds only because an unrelated package manager happened to seed a dependency leaves the application's installation contract incomplete. Declaring the dependency makes the requirement visible to the tool creating the environment.

There is a cost to that decision. Users receive packages that they might not invoke directly when all they want is to browse a database. In this version, however, Datasette includes plugin management commands that need pip. The dependency matches the capabilities the application distributes.

The patch does not change SQLite's file format or SQL behavior. It repairs the surrounding program's assumptions about its Python environment. For a SQLite publishing workflow, that can determine whether the program starts at all and whether its plugin commands operate in the intended installation. The dependency list now accounts for both paths exposed by the Rye report.