November 14, 2023·4 min read·engineeringpythonpypisecuritydeployment

The PyPI audit follows code through its deployment system

PyPI publishes its first external audit and remediation accounts for Warehouse and cabotage. The findings show why release infrastructure belongs inside the security review.

A package registry can validate its public requests correctly while its deployment machinery builds the wrong repository revision. PyPI's first external security audit examined both sides of that boundary. Its scope covered Warehouse, the application behind the index, and cabotage, the system deploying it. The remediation accounts are useful because they explain how a finding changes when traced through the actual production arrangement.

The November 14 announcement describes ten engineer-weeks of work by Trail of Bits, funded through the Open Technology Fund. The review included both codebases and their CI/CD configurations. PyPI reports no high-severity findings, while publishing separate explanations of Warehouse remediation and cabotage remediation.

I depend on Python's packages more than I enjoy managing their installation. That makes the systems controlling the registry's own releases relevant to my trust in the ecosystem. A review confined to the upload endpoint would omit code that determines what that endpoint runs.

A resolved commit is not enough repository evidence

One cabotage finding concerned repository confusion during image building. The remediation account explains that a GitHub reference lookup could resolve to a concrete commit belonging to a fork rather than the configured repository. A user could intentionally or accidentally cause a deployment to use that code.

The distinction is between locating a commit and establishing its membership in the authorized repository. A commit identifier makes the chosen content specific. It does not, on its own, establish that the content belongs to the source relationship the deployment configuration intended.

Cabotage added validation that the resolved SHA belongs to the configured repository. That preserves the meaning of the repository setting through the lookup. Without that validation, an apparently successful resolution could change the effective source of the deployment without changing the visible repository configuration.

This is a useful example of why validation has to follow references through external APIs. The input names a repository and a reference, but the returned identifier still needs to satisfy the original authorization assumption. Successfully obtaining an object is only part of that check.

Configuration becomes code at a generation boundary

Another finding involved process names in a Procfile. Cabotage incorporated user-supplied names into a generated Dockerfile. Crafted names containing newlines could alter the generated instructions. The remediation added validation of those names.

A process label and a line of Dockerfile syntax have different interpretation rules. Passing a value from one into the other without constraining its representation allows a data field to affect generated code. The relevant validation is therefore tied to the receiving syntax, not merely to whether the original configuration can be parsed.

The audit also identified shell=True usage around container building. Cabotage's account says the vulnerable path applied to local development mode and was not exploitable in its production deployment. The maintainers removed the shell usage anyway. That explanation separates the existence of a code-level injection path from a claim about production exposure.

I find this treatment more useful than a list of suspicious API names. It identifies the input, the context interpreting it and the deployment conditions under which it runs. Those details explain why one finding changes production source validation while another results in preventive cleanup of development code.

Remediation can remove obsolete functionality

Several cabotage findings came from code that no longer served the current build arrangement. An unauthenticated route once provided build context data. Direct GitHub contexts and Kubernetes secrets had made that route defunct, but the old code remained available to review and potentially expose information.

The maintainers removed it. They also removed obsolete log filtering associated with earlier private-repository support. A partial secret-filtering mechanism was unnecessary once the supported behavior no longer required that path. Removing the unused implementation reduced the amount of authentication-related code the system had to reason about.

An obsolete route can retain security consequences after its original caller disappears. Its continued presence may preserve assumptions about authentication or temporary storage that current deployments no longer exercise. The cabotage account makes the relationship explicit by explaining what replaced the old behavior before describing why removal was appropriate.

Dependency cleanup followed a similar principle. The remediation account describes moving to pip-tools to compile and pin the packages actually needed. It also records automated dependency management and updates. The audit distinguishes known vulnerable dependencies from vulnerabilities that could affect this application, which prevents a dependency count from being mistaken for an exploitability assessment.

Warehouse findings depend on surrounding systems

Warehouse's remediation account includes an XML processing issue resolved through its operating-system update. The audit began with a deployment using Debian 11 and an older Expat. During the review, the service moved to Debian 12, whose Expat version did not have the specific problem described by the finding.

That initially made the report difficult to reproduce in the updated environment. The maintainers traced the difference to the underlying library change and worked with the auditors to revise the recommendation. The relevant dependency was the parser actually loaded by Python, not only the application source under review.

A search-index locking finding also crossed a library boundary. Warehouse's custom context manager could believe it still held a lock after the Redis lock expired. The maintainers replaced it with the context manager supplied by redis-py. Local control flow and the external lock lifetime needed to describe the same protected interval.

Accepted findings need an explanation of their assumptions

The Warehouse account distinguishes accepted findings from remediated ones. For example, the temporary-file storage finding required a level of access that already enabled more direct attacks. The maintainers judged the implementation complexity disproportionate to the additional defense in their current arrangement.

The tar upload resource-consumption finding had another context. Dedicated upload workers limit the effect on the rest of the deployment, while determining the expansion of a compressed tar stream requires doing decompression work. The accepted status records a resource-isolation decision. It does not claim that a compressed input has stopped consuming resources, which is why the deployment architecture remains part of understanding that decision.