Creating and managing data

How do I create and upload data?

Complete author setup and MFA, create a private listing, complete the required declarations, accept the submission terms, and choose one file to upload.

• One image: upload one PNG or JPG file.
• Text data: upload CSV, JSON, JSONL, Parquet, or a text-only ZIP.

The uploaded file, including a TAR or ZIP, must be 100 MB or smaller. Save the listing, wait for processing to finish, and then submit it for review.

How do I upload one PNG or JPG image?

Upload the .png or .jpg file directly. You do not need a TAR for one image. DataMarket verifies the real file type and reads its pixel dimensions.

For multiple images, use a TAR and give every image a matching JSON file. Keep PNG and JPG images in separate TAR files. Approved listings can show up to three images without requiring a download.

Accepted: PNG or JPG
Maximum upload: 100 MB
Multiple images: use PNG-only or JPG-only TAR shards

How do I package multiple images in a TAR?

Give each media file one JSON file with exactly the same name before the extension. For example, img_001.png must be paired with img_001.json. A media file without its matching JSON file is rejected.

Rules for every TAR

• The complete TAR must be 100 MB or smaller.
• Keep files at the top level so names and pairs are easy to verify.
• Use regular files only. Links and unsafe paths are rejected.
• Use matching basenames for every media/JSON pair.
• For images, create a PNG-only TAR or a JPG-only TAR. Never mix PNG and JPG.
• Each media file must be 2 MB or smaller to appear in the browser preview.
• Large 1 GB WebDataset shards are not supported yet.

PNG-only image TAR

image_png_shard_00000.tar
|-- img_001.png
|-- img_001.json
|-- img_002.png
`-- img_002.json

JPG-only image TAR

image_jpg_shard_00000.tar
|-- img_001.jpg
|-- img_001.json
|-- img_002.jpg
`-- img_002.json

Image JSON example

{
  "image_id": "img_001",
  "caption": "A street intersection at dusk.",
  "description": "A wide view facing east.",
  "width": 1920,
  "height": 1080,
  "format": "png",
  "scene_tags": [
    "city",
    "traffic",
    "dusk"
  ],
  "license": "commercial"
}

Create the TAR

mkdir shard
# Copy PNG-only or JPG-only images and matching JSON files into shard/
tar -cf dataset_shard_00000.tar -C shard .

DataMarket checks media signatures, JSON structure, matching pairs, paths, entry count, and expanded size. If a check fails, the processing page explains what to fix.

What happens during processing?

DataMarket verifies the upload and checksum, scans the file, checks its format and archive safety, extracts text schema or media metadata, and creates a safe preview for review. If processing fails, open the processing page, follow the reported correction, and upload the fixed private file.

What provenance and policy information is required?

Identify the source, collection method, ownership or distribution authority, legal basis or consent, geography, collection period, sensitive fields, restrictions, and supporting evidence requested for the category. Never change a category or declaration merely to avoid a stricter policy.

How do pricing, fees, and payouts work?

You set the USD price. DataMarket displays its 10% marketplace commission and the estimated 90% author proceeds. Provider onboarding and charge and payout capabilities must be active before a draft can proceed to commerce. Provider and payout status appear in the author workspace.

How do I become an author?

Sign in to your User account, open Account, and select Register as an author at the bottom of the page. Then switch to your author workspace and create it. MFA is required before you can use author tools; payment onboarding and review are separate steps before any data can be sold.

How do I protect my author account?

Keep MFA enabled, use a unique password, protect recovery methods, and review unexpected account or payout changes immediately. DataMarket support will not ask for your password, MFA code, payment credentials, source records, or access tokens by email.