CIN API Reference (HTTP/JSON, WebSocket, gRPC)

Status: Planned for v0 / v0.1 Last updated: YYYY‑MM‑DD

0) Overview

The CIN (Crumpet Indexer Node) API exposes:

HTTP/JSON (public): search, article detail, trending, snapshot metadata, health/version.
WebSocket (public): live events stream (score updates, disputes, snapshot rotation).
gRPC (peer/control): snapshot announcements, metadata fetch, attestations, health.

A single node serves its own view of the corpus. Quorum/consensus across nodes is implemented in clients/SDKs by comparing responses (see headers in §1.5).

1) Conventions

1.1 Base URL & TLS

Public HTTP: https://<host>:<port>/v1/* (HTTP/2; HTTP/1.1 compatible)
WebSocket: wss://<host>:<port>/v1/ws/events
gRPC: grpcs://<host>:8081 (HTTP/2) — peer/control plane

1.2 Formats & Encodings

JSON; UTF‑8; timestamps are UNIX seconds.
CID in responses is string CIDv1; query params may accept either CIDv1 or 32‑byte digest hex (0x…).

1.3 Pagination & Limits

Pagination: from (offset, default 0), size (default 20, max 50).
Errors on size > 50 with 400.

1.4 Language & Ranking

lang uses BCP‑47 (e.g., en, en-US, zh-Hans).
sort / ranker: top | trending | new.
- trending uses default half‑life 36h (node reports in /v1/version).

1.5 Response Headers (introspection)

Node returns metadata to help multi‑indexer clients:

Header

Meaning

x-crumpet-node

Node self‑name (operator‑defined)

x-crumpet-params-hash

Current Config parameter hash used at index time

x-crumpet-snapshot-root

Merkle root of the language segment used (for list endpoints)

x-crumpet-snapshot-to-time

Snapshot logical upper time (seconds)

etag

Fingerprint of the response payload for caching

cache-control

public, max-age=30, stale-while-revalidate=120 (typical)

Clients may include x-crumpet-client: <name>/<version> for telemetry attribution (optional).

1.6 Errors

Uniform error body:

jsonCopyEdit{ "error": { "code": "INVALID_PARAM", "message": "size must be <= 50", "details": { "field": "size", "max": 50 } } }

Common codes: INVALID_PARAM, NOT_FOUND, TOO_MANY, RATE_LIMITED, INTERNAL.

1.7 Rate Limits

Operators may apply per‑IP token buckets. Standard headers:

x-rate-limit-limit, x-rate-limit-remaining, x-rate-limit-reset (epoch seconds). 429 on excess.

2) HTTP/JSON — Public Endpoints

2.1 Search

GET /v1/search

Query params

q (string, 1–256 chars, required)
lang (string, optional; filter)
sort (top|trending|new, default top)
from (int ≥0, default 0)
size (int 1–50, default 20)
Filters: author=0x… (20‑byte hex), tag=foo (repeatable), status=normal|under_dispute|action_taken, fromTime, toTime (unix seconds)

200 Response

jsonCopyEdit{
  "ranker": "top",
  "paramsHash": "0x34ab…",
  "total": 1234,
  "results": [
    {
      "cid": "bafybeigd…",
      "title": "How L2 Based Rollups Work",
      "subtitle": "A practical guide",
      "snippet": "…based rollups use L1 sequencing…",
      "tags": ["l2","rollups"],
      "author": "0xAbc123…",
      "createdAt": 1735600000,
      "version": 2,
      "lang": "en",
      "status": "normal",    // "under_dispute" | "action_taken"
      "score": 12.34,        // rank score (float), not on-chain
      "scoreNet": 115        // on-chain net (int)
    }
  ]
}

Errors

400 INVALID_PARAM (empty q, invalid lang, size>50)
200 total:0 for no matches

2.2 Article Detail

GET /v1/article/{cid}

Path

{cid}: CIDv1 (string) or digest hex 0x… (32 bytes)

200 Response

jsonCopyEdit{
  "cid": "bafybeigd…",
  "manifest": {
    "schema": "crumpet.aom/1",
    "article": {
      "title": "…",
      "subtitle": "…",
      "lang": "en",
      "tags": ["…"],
      "license": "CC‑BY‑4.0",
      "canonicalUrl": "https://…",
      "version": 2,
      "previousCid": "bafybeih…",
      "author": { "address": "0xAbc…", "display": "0xAbc…", "kns": null }
    }
  },
  "metrics": { "wordCount": 1240, "estReadingMins": 6 },
  "lineage": { "previousCid": "bafy…", "nextCid": "bafy…", "version": 2 },
  "score": { "upWeight": 190, "downWeight": 75, "net": 115 },
  "dispute": { "status": "none" },               // "under_dispute" | "action_taken"
  "verifiedPreview": true,                       // preview.html matched renderer hash
  "paramsHash": "0x34ab…"
}

Errors

404 NOT_FOUND (unknown CID)
502 when dependent subsystems unavailable (rare)

GET /v1/trending

Query params

lang (string; required when server indexes multiple langs)
window (6h|12h|24h|48h|7d, default 24h)
size (1–50)

200 Response

jsonCopyEdit{ "ranker": "trending", "halfLifeHours": 36, "results": [ /* same shape as search.results[] */ ] }

2.4 Snapshot Metadata (per language)

GET /v1/snapshot/{lang}/latest

200 Response

jsonCopyEdit{
  "lang": "en",
  "root": "0xe7a1…",          // Merkle root (bytes32 hex)
  "metaHash": "0x9cd4…",      // hash(renderer|analyzer|policy|range)
  "ipfsCid": "bafybeia…",     // snapshot bundle
  "fromTime": 1735500000,
  "toTime": 1735600000,
  "rendererVersion": "1.0.0+abcd",
  "sanitizerVersion": "1.0.0+abcd",
  "policyHash": "0x…",
  "analyzerVersion": "icu-74:cjk-bigram",
  "paramsHash": "0x34ab…",
  "signatures": [
    { "node": "0xNodeA", "sig": "0x…" },
    { "node": "0xNodeB", "sig": "0x…" }
  ]
}

Errors

404 if language not served.

2.5 Health

GET /v1/health

jsonCopyEdit{
  "status": "ok",               // "ok" | "degraded"
  "time": 1735601234,
  "ingestLagSeconds": 8,
  "snapshotFreshnessSeconds": { "en": 240, "de": 600 },
  "peers": 21,
  "wsClients": 112
}

2.6 Version

GET /v1/version

jsonCopyEdit{
  "nodeVersion": "0.1.0+abcd",
  "rendererVersion": "1.0.0+abcd",
  "sanitizerVersion": "1.0.0+abcd",
  "analyzerVersion": { "en": "icu-74", "zh-Hans": "cjk-bigram" },
  "rankers": { "trendingHalfLifeHours": 36 },
  "paramsHash": "0x34ab…"
}

3) WebSocket — Live Events

GET /v1/ws/events → JSON Lines (one JSON object per message). Clients should auto‑reconnect with backoff and resume from last seq if provided.

Common envelope

jsonCopyEdit{ "seq": 10234, "ts": 1735601200, "type": "<event>", "data": { … } }

Event types & payloads

article_published

jsonCopyEdit{ "type":"article_published", "data": { "cid":"bafy…", "author":"0xAbc…", "createdAt":17356… } }

score_updated

jsonCopyEdit{ "type":"score_updated", "data": { "cid":"bafy…", "upWeight":190, "downWeight":75, "scoreNet":115 } }

dispute_opened

jsonCopyEdit{ "type":"dispute_opened", "data": { "cid":"bafy…", "caseId":42 } }

dispute_resolved

jsonCopyEdit{ "type":"dispute_resolved", "data": { "cid":"bafy…", "caseId":42, "actionTaken":true } }

snapshot_rotated

jsonCopyEdit{ "type":"snapshot_rotated", "data": { "lang":"en", "root":"0xe7a1…", "toTime":1735600000 } }

Close codes

1000 normal; 1013 try again later (maintenance); 1011 server error.

4) gRPC — Control & Federation

4.1 Service Definition (excerpt, `proto/cin.proto`)

protoCopyEditsyntax = "proto3";
package crumpet.v1;

message Empty {}

message HealthRequest {}
message HealthResponse {
  string status = 1; // "ok" | "degraded"
  uint64 time = 2;
  uint32 peers = 3;
}

message AnnounceReq {
  string lang = 1;       // e.g., "en"
  bytes  merkle_root = 2; // 32 bytes
  bytes  meta_hash = 3;   // 32 bytes
  string ipfs_cid = 4;
  bytes  node_pubkey = 5; // ed25519
  bytes  signature = 6;   // sig over (lang|root|meta|cid)
}

message AnnounceAck { bool accepted = 1; string reason = 2; }

message SnapshotMetaReq { string lang = 1; }
message SnapshotMeta {
  string lang = 1;
  bytes  merkle_root = 2;
  bytes  meta_hash = 3;
  string ipfs_cid = 4;
  uint64 from_time = 5;
  uint64 to_time = 6;
  string renderer_version = 7;
  string analyzer_version = 8;
  bytes  params_hash = 9;
}

message AttestReq { string lang = 1; bytes merkle_root = 2; bytes attestor = 3; bytes signature = 4; }
message AttestAck { bool recorded = 1; string reason = 2; }

service CrumpetControl {
  rpc Health(HealthRequest) returns (HealthResponse);
  rpc AnnounceSnapshot(AnnounceReq) returns (AnnounceAck);
  rpc GetSnapshotMeta(SnapshotMetaReq) returns (SnapshotMeta);
  rpc AttestSnapshot(AttestReq) returns (AttestAck);
}

Transport

HTTP/2 + TLS by default; optional mTLS between known peers.
Timeouts: client default 3s; retry with backoff on UNAVAILABLE.

Semantics

AnnounceSnapshot: producer nodes notify peers; peers may pull bundle (HTTP/3/IPFS) and verify.
AttestSnapshot: watchers/peers post acceptance; node records for local federation decisions (on‑chain attest handled separately by Watchers → NodeRewards).

5) Object Schemas (JSON)

5.1 `ArticleSummary`

jsonCopyEdit{
  "cid": "string",
  "title": "string",
  "subtitle": "string|null",
  "tags": ["string"],
  "author": "0x…",
  "createdAt": 0,
  "version": 1,
  "lang": "en",
  "status": "normal|under_dispute|action_taken",
  "score": 0.0,
  "scoreNet": 0
}

5.2 `ArticleDetail`

jsonCopyEdit{
  "cid": "string",
  "manifest": { "schema":"crumpet.aom/1", "article": { /* safe manifest fields */ } },
  "metrics": { "wordCount": 0, "estReadingMins": 0 },
  "lineage": { "previousCid":"string|null", "nextCid":"string|null", "version": 1 },
  "score": { "upWeight": 0, "downWeight": 0, "net": 0 },
  "dispute": { "status":"none|under_dispute|action_taken", "caseId": 0, "notes": ["cid?"] },
  "verifiedPreview": true,
  "paramsHash": "0x…"
}

5.3 `SnapshotMeta`

See §2.4.

5.4 `Error`

See §1.6.

6) Examples

6.1 cURL — Search

bashCopyEditcurl -s 'https://node.example/v1/search?q=rollups&lang=en&sort=trending&size=10' \
  -H 'accept: application/json'

6.2 cURL — Article

bashCopyEditcurl -s 'https://node.example/v1/article/bafybeigd…'

6.3 WebSocket — NodeJS

jsCopyEditconst ws = new WebSocket('wss://node.example/v1/ws/events');
ws.onmessage = (ev) => console.log(JSON.parse(ev.data));
ws.onclose = (ev) => setTimeout(() => reconnect(), 1000);

6.4 gRPC — `grpcurl`

bashCopyEditgrpcurl -d '{}' -insecure node.example:8081 crumpet.v1.CrumpetControl/Health

7) Caching & Freshness

List endpoints (/search, /trending) are cacheable for short TTL (e.g., 30s); include etag.
/article/{cid} may be cached longer (immutable CID content), but score and dispute fields change; TTL ≤ 60s.
Cache-Control and ETag present; clients may use If-None-Match.

8) Security & CORS

TLS required; HSTS recommended.
CORS: node operators may restrict origins. Default: allow GET from any origin; disallow credentials.
DoS protection: pagination caps; q length cap; server timeouts (2s per request, 10s hard).

9) Status & Error Codes (HTTP)

Code

When

200

Success

304

If-None-Match hit

400

Validation failed (INVALID_PARAM)

404

Not found (article/lang)

408

Upstream timeout (rare)

429

Rate limited

500

Internal error

503

Maintenance / overload

10) Compatibility & Versioning

Path version /v1/ for breaking changes.
Additive fields are non‑breaking.
x-crumpet-params-hash signals changes in economic parameters that might affect ranking/scoring; clients may reconcile across nodes.

11) SDK Mapping (JS)

cin.search({ … }) → /v1/search
cin.getArticle(cid) → /v1/article/{cid}
cin.trending({ … }) → /v1/trending
cin.getLatestSnapshot(lang) → /v1/snapshot/{lang}/latest
cin.events() → WebSocket consumer
cin.headers(resp) → parse x-crumpet-* for multi‑indexer quorum.

12) Operational Notes for Operators

Keep /v1/health cheap; don’t block on external calls.
Ensure HTTP/2 at edge for better multiplexing; enable WebSocket upgrade.
Expose Prometheus at /metrics (scrape path; not public on the same origin unless protected).
If you run multiple languages, snapshot rotations may occur independently; headers are per language for list endpoints.

13) Acceptance Criteria

All listed endpoints implemented with shapes above.
Headers present; ETag consistent; cache directives set.
WebSocket emits events promptly on on‑chain changes and snapshot rotations.
gRPC control plane compiles from proto/cin.proto and passes interop tests.
SDK smoke tests pass against a fresh node.

Appendix A — Query Grammar & Normalization

q is passed to analyzers per language; unsupported tokens ignored.
Phrase queries: quoted "exact phrase" supported.
Field filters (optional v0.1): tag:foo author:0xabc recognized if present; otherwise treated as text.
Stopwords per language follow ICU defaults.

Appendix B — Fingerprint for Quorum (client‑side)

Compute SHA‑256 over canonical JSON:
```
cssCopyEdit{
  paramsHash,
  ranker,
  items: [ {cid, status, scoreNetBucket} * topK ]
}
```
where scoreNetBucket = floor(scoreNet / 5) to absorb tiny deltas.

PreviousRepo Skeleton & CONTRIBUTING.md

Last updated 6 months ago

hashtag0) Overview

hashtag1) Conventions

hashtag1.1 Base URL & TLS

hashtag1.2 Formats & Encodings

hashtag1.3 Pagination & Limits

hashtag1.4 Language & Ranking

hashtag1.5 Response Headers (introspection)

hashtag1.6 Errors

hashtag1.7 Rate Limits

hashtag2) HTTP/JSON — Public Endpoints

hashtag2.1 Search

hashtag2.2 Article Detail

hashtag2.3 Trending

hashtag2.4 Snapshot Metadata (per language)

hashtag2.5 Health

hashtag2.6 Version

hashtag3) WebSocket — Live Events

hashtag4) gRPC — Control & Federation

hashtag4.1 Service Definition (excerpt, proto/cin.proto)

hashtag5) Object Schemas (JSON)

hashtag5.1 ArticleSummary

hashtag5.2 ArticleDetail

hashtag5.3 SnapshotMeta

hashtag5.4 Error

hashtag6) Examples

hashtag6.1 cURL — Search

hashtag6.2 cURL — Article

hashtag6.3 WebSocket — NodeJS

hashtag6.4 gRPC — grpcurl

hashtag7) Caching & Freshness

hashtag8) Security & CORS

hashtag9) Status & Error Codes (HTTP)

hashtag10) Compatibility & Versioning

hashtag11) SDK Mapping (JS)

hashtag12) Operational Notes for Operators

hashtag13) Acceptance Criteria

0) Overview

1) Conventions

1.1 Base URL & TLS

1.2 Formats & Encodings

1.3 Pagination & Limits

1.4 Language & Ranking

1.5 Response Headers (introspection)

1.6 Errors

1.7 Rate Limits

2) HTTP/JSON — Public Endpoints

2.1 Search

2.2 Article Detail

2.3 Trending

2.4 Snapshot Metadata (per language)

2.5 Health

2.6 Version

3) WebSocket — Live Events

4) gRPC — Control & Federation

4.1 Service Definition (excerpt, `proto/cin.proto`)

5) Object Schemas (JSON)

5.1 `ArticleSummary`

5.2 `ArticleDetail`

5.3 `SnapshotMeta`

5.4 `Error`

6) Examples

6.1 cURL — Search

6.2 cURL — Article

6.3 WebSocket — NodeJS

6.4 gRPC — `grpcurl`

7) Caching & Freshness

8) Security & CORS

9) Status & Error Codes (HTTP)

10) Compatibility & Versioning

11) SDK Mapping (JS)

12) Operational Notes for Operators

13) Acceptance Criteria