The Terraform registry protocol, explained

HashiCorp documents the registry protocol, but nobody owns the readable explainer, the one that walks the actual HTTP requests and tells you where the sharp edges are. I have implemented this protocol, so this is that explainer, with real request and response shapes and the one mistake that will quietly make your registry's authentication decorative.

The protocol is smaller than you would guess. A registry is mostly three endpoints and a redirect trick.

Why a protocol exists at all

When you write source = "acme/vpc/aws" in a module block, Terraform has to turn that into "go to this host, ask for the versions of this module, pick one that satisfies the constraint, then download it." It cannot do that by convention alone, so there is a protocol: a well-known discovery document that points at the API, and a small set of endpoints the API must expose.

It starts with service discovery. Terraform requests /.well-known/terraform.json from the host and expects a document telling it where the module and provider APIs live:

GET /.well-known/terraform.json HTTP/1.1
Host: registry.terramantle.dev

{
  "modules.v1": "/v1/modules/",
  "providers.v1": "/v1/providers/"
}

That is the entire handshake. Terraform now knows to prefix module requests with /v1/modules/. This indirection is deliberate: the API can live anywhere, and the discovery document is the only fixed path.

The module protocol, walked through

Two endpoints do the work. First, list the versions:

GET /v1/modules/acme/vpc/aws/versions HTTP/1.1

{
  "modules": [
    {
      "versions": [
        { "version": "3.1.2" },
        { "version": "3.2.0" },
        { "version": "3.2.1" }
      ]
    }
  ]
}

Note what this is not. It is a flat list of versions. It does not resolve ~> 3.2 for you. Constraint resolution happens client-side, in Terraform, against this list. The registry's job is to be honest about what versions exist; the CLI picks the one that fits. A registry that tries to be clever here is solving a problem the protocol already assigns to the client.

Then, download a specific version:

GET /v1/modules/acme/vpc/aws/3.2.1/download HTTP/1.1
Authorization: Bearer <registry token>

HTTP/1.1 204 No Content
X-Terraform-Get: https://<store>/acme/vpc/aws/3.2.1.tar.gz?X-Amz-Signature=...

This is the part worth slowing down on. The download endpoint returns 204 No Content. There is no body. The actual location of the artifact comes back in the X-Terraform-Get header, and Terraform then fetches that URL to get the tarball. The download endpoint hands you a pointer, not bytes.

The gotcha that makes auth decorative

Here is the mistake, and it is an easy one to make because everything works in testing.

Your /download endpoint is authenticated. Good. You check the bearer token, confirm the caller can read this module, and return the X-Terraform-Get header. The natural thing to put in that header is the artifact's real URL: your S3 object, your git archive, wherever the tarball lives.

The problem is that Terraform fetches the X-Terraform-Get URL as a plain GET, and it does not attach your registry credentials to it. The download hop is a separate request to a separate URL, and the registry token stays behind. So if you returned a naked artifact URL, anyone who obtains that URL can pull the artifact with no auth at all. Your registry's authentication protected the pointer, not the thing it points at. It is decorative.

The failure mode is invisible in testing because in testing you are the one holding the URL, and it works. It only bites when a URL leaks, gets logged, ends up in a proxy cache, or is shared, and the artifact behind it turns out to have been world-readable the whole time.

The fix is to treat the download hop as untrusted and push authorisation into a time-limited capability URL. Do not return the artifact's real address. Return a short-lived signed URL that grants access to that one object for a few minutes. The store stays private, the link expires before it is worth stealing, and authorisation still happened at the registry layer where the trust rules live.

Terramantle returns a 15-minute presigned URL from object storage in that header. The bucket is private, the signature is scoped to the single object, and the link is dead long before it could leak usefully. When you self-host without object storage, the same 15-minute capability model is served by the worker itself with a signed URL instead of an S3 presign, but the shape is identical: expiring, single-object, private store behind it. The general lesson holds either way. The download URL is a capability, not an address, so make it expire.

sequenceDiagram
participant TF as terraform init
participant Reg as Registry API
participant Store as Private store
TF->>Reg: GET /download (Bearer token)
Reg->>Reg: verify token, authorise
Reg->>Store: mint 15-min presigned URL
Reg-->>TF: 204, X-Terraform-Get: presigned URL
Note over TF,Reg: registry token does NOT travel to the next hop
TF->>Store: GET presigned URL (no registry creds)
Store-->>TF: tarball bytes
Note over TF,Store: URL expires in minutes

The provider protocol, in one paragraph

Providers are harder because they are signed binaries, not source archives. The version list carries platforms (os/arch pairs), the download response points at the binary plus a SHA256SUMS file and its GPG signature, and Terraform verifies the checksum and the signature against the publisher's key before installing. Same discovery mechanism, same capability-URL concern on the file downloads, more moving parts because integrity and authenticity both have to be provable.

What the protocol does not cover

Two things people expect to find in the spec and do not. There is no authentication standard: the protocol says requests may require auth but not how, so every registry brings its own bearer or basic scheme. And there is no publish API: the protocol is entirely about consumption, so how modules get into a registry is implementation-defined, which is why publish flows differ wildly between products.

A minimal registry, and where it breaks

You can implement the read side in a couple of dozen lines. Static JSON files for discovery and the versions endpoint, tarballs in an S3 bucket, and a /download route that returns a 204 with the bucket URL in X-Terraform-Get. It will pass terraform init. It will also fall into the exact gotcha above: that bucket URL is either public, or it is signed but you have now hand-rolled URL signing, and the moment you make the bucket private without signing, init breaks. The minimal version works right up until you care about who can pull what, which is the moment you needed a registry rather than a bucket.

Where real products earn their place

Beyond serving the protocol, a registry adds the things the protocol leaves out: authentication and authorisation on reads, publish flows, immutability, deprecation, supply-chain scanning, and knowing which workspaces consume which version. The protocol gets a tarball to a terraform init. Everything that makes that safe to run at a company sits on top of it.

I have more implementation war stories than fit here, the kind you only collect by getting the download hop wrong once in a staging environment.

If you want the consumption side from the other direction, module versioning is what the versions endpoint is really serving, and the download-hop lesson is why a registry does not need to run your Terraform to be the trust boundary.