Check a dataset before you buy it
A data sale's buyer can now read the seller's signed data card, look at samples drawn at random, and ask aggregate questions before it pays, then recompute every claim from the bytes it received.
Know what the bytes hold
A seller signs a data card for the dataset it offers: the record count, and each field's type, nulls and distinct values, by rules any implementation can reproduce, with its own declarations kept apart. The buyer can also ask for a few 4 KiB pieces, drawn at random by its own client after the seller committed to every piece, and checked before anything is funded. A card or a sample can only describe the dataset in the offer.
Ask questions, never see a record
Under a policy the seller signs, the buyer asks for counts, distinct counts, sums and means, by group, while the offer waits for funding. Answers are exact and signed, groups under the minimum size are left out, and questions that would single out a record are refused. Once the dataset arrives, beam sale verify-claims recomputes the card and every answer from the bytes received.
- Data cards: beam sale describe writes a signed card for a CSV, TSV, JSON Lines, JSON, XML or Parquet dataset, plain or gzip- or zstd-compressed, and beam sale card checks one. beam sale buy --card refuses an offer for any other dataset before anything is funded.CLI
- Proven samples: beam sale offer --samples lets the buyer fetch up to that many random 4 KiB pieces with beam sale buy --samples, each checked against the seller's commitment, without funding.CLI
- Questions before buying: beam sale offer --policy answers aggregate questions inside the sale Room; beam sale policy and beam sale query ask, and keep each signed answer as a receipt.CLI
- beam sale verify-claims recomputes a sale's card and every answer from the delivered dataset and lists any claim that differs.CLI
How it works
The escrow guarantees the bytes a seller committed to. These checks tell the buyer what those bytes hold, before it pays. Each one is signed by the seller's sale wallet and bound to the dataset's fingerprint and size, so it cannot describe another dataset.
Get started
- Update the CLI to 0.1.47. The seller writes a card with beam sale describe and publishes it with the dataset's listing; the buyer checks it with beam sale card.
- The seller adds --samples with a maximum number of pieces, and --policy with its rules for questions, to beam sale offer.
- Once the package has arrived, the buyer runs beam sale buy --samples to fetch samples, beam sale policy to read the rules, and beam sale query to ask. beam sale buy --card then funds the sale against the card.
- After delivery, the buyer runs beam sale verify-claims with the sale's id.
Costs and limits
- Cards and verify-claims run locally and cost nothing. Questions travel as Room messages, end to end encrypted, and cost nothing beyond the sale.
- Each samples answer, a refusal included, is a delivery: Beam charges the seller's organization for it, and it counts against the seller's daily delivery budget. A seller offers at most 256 pieces and sends at most three answers per sale.
- Answers take as long as reading the dataset, since every batch of questions reads it whole. On a 1 GiB CSV of 24 million records, on a host with 2 CPUs, one batch took about 2.5 minutes.
What it does not do
- The escrow does not know about cards, samples or answers. A claim proven false is evidence against the seller, not a refund or a penalty.
- The minimum group size and the check against near-identical questions make extracting records harder, not impossible: three or more overlapping questions can still be combined. Sellers should offer questions only on aggregates they are willing to reveal.
- Samples and questions come after the encrypted package is delivered, though before payment. A sample of a compressed or encrypted file shows nothing readable.
- Beam's servers never see a card, a question or an answer.