Streaming and snapshots

Bulk endpoints serve large datasets: a full snapshot pull is tens to hundreds of MB. Every /full request answers 200 and streams the body inline. There is no redirect and no separate download URL.

Pulling a snapshot

curl -s "$BASE_URL/v1/sanctions/full" \
  -H "Authorization: Bearer $API_KEY" \
  -o sanctions.json

• The default format is application/json, a single array. Pass ?format=jsonl for application/x-ndjson, one record per line, emitted as available. Pass ?format=csv for a flattened spreadsheet, which arrives as an attachment.
• Record order is not guaranteed between runs. If you need stable order, sort by id after ingestion.
• Responses are not compressed. Sending Accept-Encoding changes nothing.

Freshness and repeat pulls

Every /full response is derived from a snapshot prepared ahead of time, so the response starts streaming immediately. The snapshot is rebuilt once a day, and the X-Snapshot-As-Of header tells you which build you received.

Responses carry ETag and Cache-Control: private, max-age=3600. Send the ETag back as If-None-Match to skip an unchanged pull:

curl -s "$BASE_URL/v1/sanctions/full" \
  -H "Authorization: Bearer $API_KEY" \
  -H 'If-None-Match: "<etag-from-last-pull>"'

An unchanged snapshot answers 304 Not Modified with no body. See Rate limits and headers for when the ETag changes.

Client requirements

• Disable connection timeouts (or set them very high) for the pull; a full stream can take several minutes.
• Process records as they arrive; do not buffer the whole response before parsing.
• Always inspect the final record for an in-band error envelope. See Errors.

Recovering an interrupted stream

Streams are not resumable. Restart the pull from the beginning. Results across runs are consistent, though not byte-identical.