July 28, 2020·4 min read·engineeringrubyrubygemssecuritycredentials

RubyGems resets publishing keys exposed through RSS request logs

The July incident report traces 407 exposed API keys to personalized RSS URLs and separates immediate revocation from planned changes to credential scope.

A credential used to read a personal feed should not also authorize publishing a package. RubyGems.org's July 28 incident report describes what happened when a user API key served both purposes. Requests for personalized dashboard feeds put keys into URLs, and access logs sent those URLs to a publicly queryable dataset. The team identifies 407 exposed keys and says it has reset all of them.

The report traces the logging integration to October 2018 and discovery to July 18, 2020. RubyGems.org stopped sending new events and removed public access after discovering the leak. Its investigation of query history found no searches for the leaked API keys. That is evidence about recorded queries, without establishing an unlimited claim about every possible form of credential misuse.

I find Ruby's package ecosystem useful because libraries are easy to obtain and incorporate. Publishing credentials deserve a correspondingly clear scope. Reading a personalized list of package updates has a much smaller authorization requirement than changing what another application may install.

A URL became a credential-bearing log field

The report explains the client constraint behind the design. Many RSS readers cannot send authorization headers, so the personalized feed accepted a key in its query string. The logging integration scrubbed personal information such as client IP addresses but overlooked that credential-bearing GET request.

This is a distinction between identifying an account and authorizing an operation. A URL can look like ordinary request metadata while carrying a value that the server accepts as proof of authority. Filtering a list of recognized personal fields does not necessarily remove every value with that property.

RFC 6750, published in 2012, documents the same general risk for bearer tokens in URI query parameters. It discourages that transport because URLs containing tokens are likely to be logged. RubyGems' API key is not thereby an OAuth token, but the exposure mechanism is comparable. A credential copied into an access log acquires the log's storage and access rules.

The practical implication follows from that mechanism. Log processing is part of the path a URL can take after its request succeeds. Encrypting transport to the application does not prevent the application's own infrastructure from recording the URL. Credential placement must account for those authorized processing steps too.

Authority should match the client task

A feed reader needs permission to retrieve one user's personalized feed. A publisher needs permission to create package versions. Reusing one key means the less privileged activity transports the more privileged credential. A leak in the feed path can therefore affect operations that the feed reader never intended to perform.

The report's linked API-key design work proposes multiple keys with permissions attached to each key. That proposal predates this incident. The report also says a separate RSS key without API access will be introduced. That is a planned scope change, distinct from the resets already completed.

A feed-specific credential would still need protection. Its disclosure could reveal the private feed it authorizes. The advantage is limiting what possession permits. Losing that credential would not automatically grant the authority associated with a package-publishing key.

The same separation also improves revocation choices. If a credential belongs to one task, replacing it can affect that task without requiring every other client to change. This is an architectural consequence of separate keys, not a claim that all such behavior is already available in the service today.

The investigation had two retention periods

Honeycomb retained the dataset for up to sixty days. The team first scanned that retained data, then examined older HTTP-log archives to identify keys sent earlier in the integration's lifetime. The report identifies 155 keys in the retained dataset and another 252 through the historical search.

That distinction explains why searching only the active analytics dataset would not have identified every affected credential. Data retention in a downstream service and the period during which credentials entered that service are different intervals. Revocation has to consider the exposure history relevant to credentials that may still be valid.

The report says affected users were notified. Notification and revocation perform different jobs. Revocation removes the known credential's authority, while notification allows the owner to assess whether package activity needs review and to repair clients still using the old value.

I prefer the report's explicit account of those steps to an undifferentiated statement that logging was fixed. Stopping new events prevents additional exposure through that route. Removing public access changes who can query retained data. Resetting keys addresses authority already copied outside the intended authentication path.

A reset also changes the meaning of an old log entry. The recorded string remains a disclosure of historical data, but it should no longer authenticate a new API operation. That is why retention cleanup and credential revocation need separate treatment in an incident involving reusable credentials.

Package immutability limits one kind of damage

The incident report explains that an existing published gem version cannot be modified, even with valid credentials. A stolen key could still authorize a new malicious version or a yank, subject to the account's other protections. Immutability constrains replacement of an established artifact without eliminating publishing risk.

That difference matters to a consumer's interpretation of the incident. A package ecosystem can preserve the bytes of an existing version and still need to investigate unauthorized new releases. The relevant history includes operations permitted by the credential, not just whether an old archive changed in place.

The report also distinguishes two-factor authentication enabled for API operations from a key used alone. The authority of a copied key depends on the checks required for those operations. An account's authentication policy is therefore part of assessing exposure, alongside which credential was logged.

As of the report, API actions no longer accept query-string authentication. The dedicated RSS credential remains a stated next step. These changes address separate parts of the design, with one removing a risky authentication transport for API operations and the other reducing the authority needed by clients that can only authenticate through a feed URL.