# Morcom: Technical Architecture and Stack

Federated, community-owned registry for open-weight models. Morcom never hosts weights. A model repo is a signed manifest; bytes move peer to peer. Morcom runs only an index, a tracker, and a signing/verification service, all publicly mirrorable.

## Decisions at a glance

| Decision | Choice | Rejected |
|---|---|---|
| Manifest + signing | OpenSSF Model Signing (OMS) on Sigstore | Custom format, raw TUF/in-toto at manifest level |
| Primary byte transport | BitTorrent v2 (libtorrent, BEP 52) | IPFS/Bitswap, tokenised storage |
| Fallback transport | Signed HTTP mirrors (range requests) | None, this is mandatory |
| Watch-list transport | Iroh (BLAKE3 over QUIC) | |
| Client compatibility | HF Hub REST surface via HF_ENDPOINT swap | Xet protocol (not needed for MVP) |
| Index | Postgres + Meilisearch | Elastic, custom search |
| Index mirroring | Signed append-only log + TUF-signed snapshots | Ad hoc dumps |
| Enterprise mirror path | OCI artifacts via ORAS (optional) | OCI as canonical distribution |

## 1. Manifest format and signing

**Adopt OMS verbatim.** It is a detached signature in the Sigstore bundle format wrapping an in-toto statement whose predicate is a per-file SHA-256 manifest. It is PKI-agnostic (keyless Sigstore with OIDC + Rekor transparency log, enterprise CA, or self-signed), which matches the federated seeder model: a university signs with its own institutional identity. NVIDIA has signed all NGC models with OMS since March 2025 and Google is prototyping it on Kaggle, so Morcom manifests are verifiable by tooling the ecosystem is already building.

**Morcom adds one thing:** a small JSON locator sidecar containing the BitTorrent v2 infohash and magnet, HTTP mirror URLs, optional Iroh ticket, file sizes, and license/provenance fields. The OMS signature covers the weight hashes; the locator is signed separately by the same key so mirrors can be added without re-signing weights.

**TUF** is applied at the index snapshot layer (section 5), not to individual manifests. in-toto is already inside OMS.

**OCI/ORAS:** OCI registries are proven for model artifacts (Harbor + CNAI, KAITO, ModelPack, Ollama), but OCI is a client-server pull model and recreates the central host. Support it as an optional mirror target an institution can expose, never as the canonical path.

## 2. P2P distribution layer

| Option | Large-file fit | Resumability | Chunk verification | NAT traversal | Institutional seeding | Verdict |
|---|---|---|---|---|---|---|
| BitTorrent v2 (libtorrent) | Excellent; per-file SHA-256 Merkle over 16 KiB leaves | Native | Fine-grained, metadata stays small at any file size | Moderate (DHT/tracker) | Strong; institutions already run torrent infra | Primary |
| WebTorrent | Good for browser reach | Native | v1 SHA-1 only | Good (WebRTC) | Weak at TB scale | Browser adjunct only |
| IPFS / libp2p / Bitswap | Poor; 6-block parallel fetch, small windows, HTTP often faster | Partial | CID Merkle DAG | Good | Weak, high overhead | Reject |
| Iroh (BLAKE3/QUIC) | Good in principle, TB-scale blobs, range requests | Native, verified streaming | BLAKE3, roughly 6% metadata overhead at 1 KiB chunks | Excellent (dial-by-key, hole punching, relays) | Immature; single stream measured at 42 to 50 MB/s on a 110 MB/s link | Watch-list |
| Signed HTTP mirrors | Excellent (proven at HF scale) | HTTP range | Via manifest hash | N/A | Strong; any lab with a bucket | Universal fallback |

**Why BitTorrent v2:** BEP 52's per-file Merkle tree is exactly what a multi-hundred-GB safetensors file needs. Any 16 KiB block verifies on arrival, torrents stay small, and identical files across repos share root hashes, giving natural dedup in the same spirit as HF's Xet content addressing. libtorrent 2.0+ is mature and embeddable. Academic Torrents has run this exact HTTP + swarm hybrid since 2013 and moves over 3 TB/day.

**Why HTTP fallback is non-negotiable:** it guarantees the system degrades to "just works" behind hostile firewalls and on the long tail, and it is what makes the `HF_ENDPOINT` swap transparent.

**Iroh:** prototype in parallel, do not block on it. Measure libtorrent large-swarm throughput directly; no primary-source Gbps benchmarks were located.

## 3. Seeding incentives and durability

- **Seed-node operator program** with public badges on model pages and org profiles, tied to measurable SLAs (uptime, bytes served, models pinned).
- **Morcom Anchor tier** for institutions committing to seed a defined model set under an availability SLA, analogous to Debian official mirrors. Anchors get a governance voice.
- **Long-tail fallback pinning:** models with zero live seeders fall back to a foundation-run store on zero-egress object storage. Budget a few hundred TB. This prevents the classic dead-swarm failure.
- **Seed-node daemon:** packaged libtorrent-based service for institutions, configurable pin sets, reports metrics to the index for badge and SLA tracking.

## 4. Abuse, malware, and licensing

This is the highest-risk area.

1. **Safetensors-preferred policy.** Pickle files are warned on and quarantined. Run `picklescan` + ClamAV on ingestion, but treat scanning as necessary and not sufficient: 7z evasion and ShadowPickle-class attacks defeat multiple scanners as of 2026. Trusted-signer provenance is the real defence.
2. **Mandatory SPDX/OMS license field** in every manifest. Refuse to index without it.
3. **Notice-and-delist process** at the index layer. Morcom cannot remove bytes from a swarm, but it can delist a manifest, the same posture Meta took against the LLaMA torrent on HF.
4. **Namespace-reuse protection.** Never recycle deleted org names (Unit 42 model namespace reuse attack).

## 5. Hugging Face API compatibility

The `huggingface_hub` client builds file URLs as

```
{ENDPOINT}/{repo_id}/resolve/{revision}/{filename}
```

where `ENDPOINT` reads `HF_ENDPOINT`. One line redirects everything:

```bash
export HF_ENDPOINT=https://hub.morcom.ai
```

hf-mirror.com proves this works transparently for millions of users.

**Minimal gateway surface**

- `GET /api/models` and `GET /api/models/{repo_id}`: listing and metadata (siblings, tags, sha).
- `GET /api/models/{repo_id}/revision/{revision}`: revision resolution.
- `GET /{repo_id}/resolve/{revision}/{filename}`: the critical resolver. Small files served directly; large files 302-redirect to a signed HTTP mirror URL (LFS style). Expose magnet and Iroh ticket in a Morcom-specific header so Morcom-aware clients prefer P2P.
- `HEAD` on the resolver returning `X-Repo-Commit`, `Content-Length`, `ETag`, and the LFS `X-Linked-Size` / `X-Linked-Etag` semantics the client expects.
- `GET /api/whoami-v2` and token endpoints, permissive for anonymous public read.
- `/.well-known/openapi.json` for discoverability.

**Xet:** not required. The client falls back to plain LFS-style resolve when the endpoint is not Xet-enabled. Morcom's own content addressing (BT v2 Merkle or Iroh BLAKE3) is philosophically identical and is the Morcom-native fast path. A Xet-compatible CAS is a later optimisation.

## 6. Index and search stack

- **Postgres:** canonical registry (repos, revisions, manifest hashes, signatures, seeder/mirror lists, license metadata).
- **Meilisearch** for search: typo-tolerant, trivial to self-host, built on Tantivy so a Rust gateway can embed the engine directly if preferred. Typesense is the alternative.
- **Mirrorability** as a first-class requirement:
  1. A signed append-only transparency log of manifest registrations (Rekor-style) so nobody can silently rewrite history.
  2. Periodic signed snapshots distributed by git-style replication and torrent.
  3. TUF roles (root, snapshot, timestamp, targets) over the snapshots so a compromised mirror cannot serve stale or forged index state. Root keys are multi-party, held by distinct member orgs.

If Morcom's servers vanish, any mirror reconstitutes the full signed index. Precedents: PyPI mirroring with TUF (PEP 458), Debian mirror network, kernel.org mirrors.

## 7. Cost model

Monthly, order of magnitude.

| Line item | Estimate |
|---|---|
| Index API, Postgres, Meilisearch | Low thousands USD |
| Tracker + DHT bootstrap | Hundreds USD (metadata only, bandwidth-light) |
| Signing/verification (Sigstore public infra, Rekor) | Near zero |
| Metadata mirrors and snapshots | Trivial |
| Fallback pinning, ~300 TB | Backblaze B2 about $1,800 (free egress via Cloudflare Bandwidth Alliance); Cloudflare R2 about $4,500 with $0 egress |

**Same 300 TB on S3 + CloudFront:** egress alone at roughly one full read per month would run $15,000 to $27,000. That is the economics Morcom is built to escape.

**Contrast:** HF's own docs cite 45 PB on the legacy LFS/S3 backend serving 3M+ models to 18M+ developers, an egress-dominated CDN operation. Morcom externalises byte serving to seeders and pays only for the long tail.

## Technical caveats

- Iroh throughput figure comes from a single GitHub issue, not an official benchmark.
- libtorrent large-swarm numbers should be measured, not assumed.
- `picklescan` and safetensors do not fully solve malware. Provenance is the defence.
- HF does not publish absolute storage or CDN costs; the contrast is inferred from published pricing and scale figures.