URL to markdown

URL to markdown fetches one public web page and returns its main content as clean markdown. One page is one paid call of 1,000 zat.

An agent that needs to read the web does not have to parse HTML. It sends a URL and gets headings, links, code blocks and lists, without the navigation, scripts and ads around them.

At a glance

How it works

1. Unpaid attempt. The service checks the URL, fetches the page through its address guard, converts it and caches the result. Any failure is returned here as an error.
2. 402. If the page worked, the service answers with the price.
3. Paid retry. The agent sends the same request with a voucher?A message the agent signs for each call. It states the agent's running total, so the facilitator can check it against the deposit.. The service verifies it and answers from the cache, so it does not fetch the page a second time. You get the version the unpaid attempt saw, even if the page changed in between.

The service holds a page for your paid retry for as long as MARKDOWN_CACHE_MS (60 seconds, 120 under Docker, since a first deposit can take tens of seconds). Other clients' pages cannot push it out of the cache. A client holds at most four pages at a time, and its oldest hold goes first. A paid retry that arrives after its hold has run out finds no page and fetches again, so it can fail like an unpaid attempt, with the voucher already signed. The stock client counts that call as paid.

The route

The answer:

curl -si 'http://127.0.0.1:4032/v1/markdown?url=https%3A%2F%2Fexample.com'

HTTP/1.1 402 Payment Required
Content-Type: application/json; charset=utf-8
PAYMENT-REQUIRED: eyJ4NDAyVmVyc2lvbiI6MiwiZXJyb3IiOiJQYXltZW50IHJlcXVpcmVkIiw...
Cache-Control: no-store

{}

{
  "x402Version": 2,
  "error": "Payment required",
  "resource": {
    "url": "http://localhost:4032/v1/markdown",
    "description": "Fetches one public web page and returns its main content as clean markdown.",
    "mimeType": "application/json",
    "serviceName": "URL to markdown",
    "tags": ["web", "markdown", "scrape"]
  },
  "accepts": [
    {
      "scheme": "prepaid-voucher",
      "network": "zcash:regtest",
      "amount": "1000",
      "asset": "ZEC",
      "payTo": "uregtest1...",
      "maxTimeoutSeconds": 60,
      "extra": {
        "windowZat": "100000",
        "creditRatioBps": 2500,
        "creditCapZat": "200000",
        "zeroConfMinExpiryBlocks": 20,
        "termsVersion": "cb7f148af0fda19a"
      }
    }
  ],
  "extensions": {
    "z402": {
      "unit": "page",
      "note": "The site sees this server's IP, not yours."
    }
  }
}

{
  "url": "https://example.com/",
  "finalUrl": "https://example.com/",
  "title": "Example Domain",
  "byline": null,
  "markdown": "This domain is for use in documentation examples without needing permission. Avoid use in operations.\n\n[Learn more](https://iana.org/domains/example)",
  "length": 149,
  "truncated": false
}

The url in the query string is not part of the offer. The 402 names the route, so the facilitator never sees which page you asked for.

Price and billing

A page costs 1,000 zat whatever its size. A page longer than 1 MiB of markdown is cut and marked truncated, at the same price.

The rate limit counts fetches, not requests. Only a cache miss takes a token, so an unpaid attempt and its paid retry cost one token together, and a page already in the cache is never refused for the limit. A miss that joins a fetch of the same URL already running takes none.

Limits and refusals

Every refusal below happens on the unpaid first attempt, before any voucher exists, so a refused page is never charged.

Error bodies hold the code and, for a fetch that failed upstream, that status. They never name an address the service resolved.

The address guard

A service that fetches any URL a stranger sends is an SSRF?Server-side request forgery: tricking a server into fetching an address inside its own network. target. This one checks each request and each redirect before it connects.

Refused addresses are private, loopback, link-local (including 169.254.0.0/16, where cloud metadata lives), carrier-grade NAT, unique-local, multicast, reserved, unspecified, broadcast, documentation and benchmarking ranges. IPv4-mapped IPv6 addresses are judged as the IPv4 address they map, and IPv4-compatible, NAT64, 6to4 and Teredo addresses are refused.

The conversion

1. jsdom parses the page at its final URL. Scripts never run and nothing else is loaded.
2. Readability finds the main content. When it finds none, or the document is more than 64 levels deep or has more than 40,000 elements, the whole body is used.
3. Links and images are made absolute. A link keeps only an http, https or mailto target and an image only an http or https one. Any other link keeps its text without a target, and any other image is removed.
4. Turndown writes markdown with # headings, fenced code and - bullets. Scripts, styles, noscript, iframes and forms are removed.

The charset comes from a byte order mark, then Content-Type, then a <meta> tag in the first 1 KiB, and is UTF-8 otherwise. text/plain is returned as it is.

Conversions run on worker threads (MARKDOWN_CONVERT_WORKERS, one by default), each with a heap of MARKDOWN_WORKER_HEAP_MB (192) and the same deadline. A page that runs out of either ends only its worker and is refused as too_large, so the server itself never stalls or runs out of memory. The heap also sets the page budget: (heap minus 64) times 300 start tags, which is 38,400 by default. Nesting deeper than 512 elements is refused. Long Wikipedia articles have about 22,000 tags.

Make a paid call

This is the service that runs as its own provider, so your first call to it also makes a deposit to its address, one more window of 100,000 zat, even if you already paid the operator's services. It needs a funded wallet and a running service: Make a paid call names the stack that paid-fetch.ts reaches, and Pay from an agent explains the setup.

import { close, paidFetch } from "./paid-fetch.js";

interface Page {
  url: string;
  finalUrl: string;
  title: string | null;
  byline: string | null;
  markdown: string;
  length: number;
  truncated: boolean;
}

const request = new URL("http://127.0.0.1:4032/v1/markdown");
request.searchParams.set("url", "https://example.com");

try {
  const res = await paidFetch(request);
  if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);
  const page = (await res.json()) as Page;
  console.log(`${page.title} (${page.length} characters)`);
  console.log(page.markdown);
} finally {
  await close();
}

import { wrapFetchWithPayment, x402Client } from "@x402/fetch";
import { LightwalletdTxStatus, PrepaidVoucherScheme, ZalletRpcWallet } from "@z402-net/client";
import { createLightwalletdClient } from "@z402-net/lightwalletd";
import { NETWORK_REGTEST } from "@z402-net/protocol";

// Zallet pays the deposit and lightwalletd (Zaino) reports its status.
// These are the Z3 regtest defaults.
const wallet = new ZalletRpcWallet({
  url: "http://127.0.0.1:50232",
  user: "zebra",
  password: process.env.ZALLET_PASSWORD ?? "zebra",
});
const lwd = createLightwalletdClient({ url: "http://127.0.0.1:28137" });
const scheme = new PrepaidVoucherScheme({ wallet, chain: new LightwalletdTxStatus(lwd) });

// The one line that adds Zcash to the stock x402 client
export const paidFetch = wrapFetchWithPayment(fetch, new x402Client().register(NETWORK_REGTEST, scheme));

/** Lets a deposit in flight finish, then closes the wallet and the chain client. */
export async function close(): Promise<void> {
  await scheme.close();
  lwd.close();
}

Privacy

• The site sees the service, not you. The page's server sees the service's IP address and a fixed User-Agent (z402-markdown/0.1). It gets no cookies and nothing from your request. That also means pages behind a login cannot be fetched.
• The operator sees the URL and the page. It has to, to fetch and convert it. The URL travels in the query string, so keep secrets out of it.
• The facilitator sees the payment, not the page. The offer names the route and not the query, so the URL never reaches the facilitator.
• The payment goes to a separate provider. Its address and API key are not the operator's.

See Privacy model for what the payment itself reveals.

Run it yourself

cp services/markdown/.env.example services/markdown/.env   # set the facilitator URL, key and Z402_PAY_TO
pnpm --filter @z402/service-markdown start

The ready line goes to stdout. Logs go to stderr.

markdown listening on http://127.0.0.1:4032 network=zcash:regtest

The service has no model and no model provider key. It fetches public pages itself. A bad value stops the start with a markdown: <VARIABLE>: ... line on stderr and exit code 1.

Running as provider 2

URL to markdown runs as an independent provider of the rail, paid to its own address and connecting with its own API key, as an outside provider would. The operator's other services use the operator's address and key. To run it the same way:

1. Make keys for the provider

   merchant-keygen writes a new account and prints its shielded-only address. The file holds the spending seed, so keep it under .secrets/. It needs the zdecrypt binary (pnpm --filter @z402/facilitator build:native).

   node scripts/merchant-keygen.mjs --network regtest --out "$PWD/.secrets/markdown-keys.json"

2. Register the view-only key

   POST /providers takes the address and its view-only key (UIVK). The script reads only the network, the address and the UIVK from the file, sends those three, and writes the API key the facilitator issues to a file without printing it.

   node scripts/register-provider.mjs \
     --keys .secrets/markdown-keys.json \
     --facilitator http://127.0.0.1:4022 \
     --out .secrets/markdown-api-key

3. Point the service at them

   Z402_FACILITATOR_URL=http://127.0.0.1:4022
   Z402_FACILITATOR_TOKEN_FILE=../../.secrets/markdown-api-key
   Z402_PAY_TO=uregtest1...

At start the service calls the facilitator's GET /providers/me to prove that its key and address belong together. If they do not, it stops and names Z402_FACILITATOR_TOKEN or Z402_PAY_TO. See Providers and API keys.

Settings

The variables every service shares (Z402_NETWORK, Z402_FACILITATOR_URL, Z402_FACILITATOR_TOKEN, Z402_PAY_TO, Z402_PRICE_ZAT, PUBLIC_URL, HOST, PORT) are listed under LLM inference. Here Z402_PRICE_ZAT defaults to 1000 and PORT to 4032. Its own:

Behind your own reverse proxy, set MARKDOWN_TRUST_PROXY so that the rate limit sees client IPs and not the proxy's. true is refused, because Express would then take the left-most X-Forwarded-For address, which the client writes itself.

Memory is about 330 MB idle with one warm worker, and a busy worker can add up to its heap. With the defaults the worst page brings that to about 520 MB. The Docker deployment gives it 768 MiB.

Next steps

References

• services/markdown/src/service.ts: the route, the checks before payment, the cache and the paid handler
• services/markdown/src/guard.ts: URL checks and the address policy
• services/markdown/src/fetch.ts: the pinned, redirect-by-hand fetch and body limits
• services/markdown/src/convert.ts: jsdom, Readability and turndown
• services/markdown/src/convert-pool.ts: conversion workers and their limits
• services/markdown/src/limits.ts: the rate limit and the page cache
• services/markdown/src/errors.ts: the error codes and their statuses
• services/markdown/src/config.ts: the settings and their limits
• services/markdown/README.md: refusals, the guard, conversion and provider 2
• services/markdown/.env.example: every variable with its default
• scripts/register-provider.mjs: registering a provider and saving its key
• docs/superpowers/specs/2026-10-27-z402-services-design.md: section 4.2, the service's contract
• docs/superpowers/specs/2026-10-28-z402-multi-provider-design.md: sections 4.1 and 4.6, providers and the start-up check
• OWASP: Server Side Request Forgery
• Mozilla Readability
• Turndown
• jsdom