Importing Data Packages

Learn how to bulk-import multiple tables at once using Frictionless Data Packages.

Overview

Instead of creating tables one at a time, you can import an entire data schema using the Data Package format. This is perfect for:

• 🚀 Quick Setup: Create your entire database schema in one request
• 🔗 Relationships: Automatically handles foreign key dependencies
• 📦 Portability: Export schemas from one app and import to another
• 🎯 Consistency: Ensure all related tables are created together

What is a Data Package?

A Data Package is a standard format for describing datasets, defined by Frictionless Data. Think of it as a blueprint for your database that includes multiple related tables.

Basic Structure

{
  "name": "my-database-schema",
  "title": "My Application Database",
  "description": "Complete schema for my application",
  "resources": [
    {
      "name": "users",
      "schema": { /* table definition */ }
    },
    {
      "name": "posts",
      "schema": { /* table definition */ }
    }
  ]
}

Import Methods

Method 1: API Endpoint (Programmatic)

Use the API for automated imports, CI/CD pipelines, or scripting.

Endpoint:

POST /api/apps/{app-slug}/datatables/create_schema/

Example:

REST API

curl -X POST https://your-site.taruvi.cloud/api/apps/blog-app/datatables/create_schema/ \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d @schema.json

Python

import requests
import json

with open("schema.json") as f:
    schema = json.load(f)

response = requests.post(
    "https://your-site.taruvi.cloud/api/apps/blog-app/datatables/create_schema/",
    headers={"Authorization": "Bearer YOUR_TOKEN"},
    json=schema
)
result = response.json()

JavaScript

import fs from "fs";

const schema = JSON.parse(fs.readFileSync("schema.json", "utf-8"));

const response = await fetch(
  "https://your-site.taruvi.cloud/api/apps/blog-app/datatables/create_schema/",
  {
    method: "POST",
    headers: {
      "Authorization": "Bearer YOUR_TOKEN",
      "Content-Type": "application/json"
    },
    body: JSON.stringify(schema)
  }
);
const result = await response.json();

Creating a Data Package

Example: Blog Application

Let's create a complete blog schema with users, posts, and comments:

{
  "name": "blog-schema",
  "title": "Blog Application Schema",
  "description": "User-generated content platform",
  "resources": [
    {
      "name": "users",
      "title": "User Accounts",
      "schema": {
        "fields": [
          {
            "name": "id",
            "type": "integer",
            "constraints": {"required": true}
          },
          {
            "name": "username",
            "type": "string",
            "constraints": {"required": true, "maxLength": 50}
          },
          {
            "name": "email",
            "type": "string",
            "format": "email",
            "constraints": {"required": true}
          },
          {
            "name": "bio",
            "type": "string"
          },
          {
            "name": "created_at",
            "type": "datetime"
          }
        ],
        "primaryKey": ["id"]
      }
    },
    {
      "name": "posts",
      "title": "Blog Posts",
      "schema": {
        "fields": [
          {
            "name": "id",
            "type": "integer",
            "constraints": {"required": true}
          },
          {
            "name": "author_id",
            "type": "integer",
            "constraints": {"required": true}
          },
          {
            "name": "title",
            "type": "string",
            "constraints": {"required": true, "maxLength": 200}
          },
          {
            "name": "content",
            "type": "string"
          },
          {
            "name": "published_at",
            "type": "datetime"
          },
          {
            "name": "status",
            "type": "string",
            "constraints": {"enum": ["draft", "published", "archived"]}
          }
        ],
        "primaryKey": ["id"],
        "foreignKeys": [
          {
            "fields": ["author_id"],
            "reference": {
              "resource": "users",
              "fields": ["id"]
            }
          }
        ]
      }
    },
    {
      "name": "comments",
      "title": "Post Comments",
      "schema": {
        "fields": [
          {
            "name": "id",
            "type": "integer",
            "constraints": {"required": true}
          },
          {
            "name": "post_id",
            "type": "integer",
            "constraints": {"required": true}
          },
          {
            "name": "user_id",
            "type": "integer",
            "constraints": {"required": true}
          },
          {
            "name": "content",
            "type": "string",
            "constraints": {"required": true}
          },
          {
            "name": "created_at",
            "type": "datetime"
          }
        ],
        "primaryKey": ["id"],
        "foreignKeys": [
          {
            "fields": ["post_id"],
            "reference": {
              "resource": "posts",
              "fields": ["id"]
            }
          },
          {
            "fields": ["user_id"],
            "reference": {
              "resource": "users",
              "fields": ["id"]
            }
          }
        ]
      }
    }
  ]
}

Smart Features

Automatic Dependency Resolution

The system automatically detects foreign key relationships and creates tables in the correct order:

users (no dependencies)
  ↓
posts (depends on users)
  ↓
comments (depends on posts and users)

Creation Order: users → posts → comments

You don't need to order your resources manually—the system figures it out!

Optional Materialization

Without materialize=true (default):

• Creates DataTable metadata entries
• Does NOT create physical database tables yet
• You can review and materialize later

With materialize=true:

• Creates DataTable metadata entries
• Creates physical database tables immediately
• Handles dependencies automatically
• Ready to insert data right away

API Example:

REST API

# Create metadata only
curl -X POST https://your-site.taruvi.cloud/api/apps/blog-app/datatables/create_schema/ \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d @schema.json

# Create tables immediately
curl -X POST "https://your-site.taruvi.cloud/api/apps/blog-app/datatables/create_schema/?materialize=true" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d @schema.json

Python

import requests

headers = {"Authorization": "Bearer YOUR_TOKEN"}
url = "https://your-site.taruvi.cloud/api/apps/blog-app/datatables/create_schema/"

# Create metadata only
response = requests.post(url, headers=headers, json=schema)

# Create tables immediately
response = requests.post(url, headers=headers, json=schema, params={"materialize": "true"})

JavaScript

const headers = {
  "Authorization": "Bearer YOUR_TOKEN",
  "Content-Type": "application/json"
};

// Create metadata only
await fetch("https://your-site.taruvi.cloud/api/apps/blog-app/datatables/create_schema/", {
  method: "POST", headers, body: JSON.stringify(schema)
});

// Create tables immediately
await fetch("https://your-site.taruvi.cloud/api/apps/blog-app/datatables/create_schema/?materialize=true", {
  method: "POST", headers, body: JSON.stringify(schema)
});

Admin Example:

• Check the "Create physical database tables immediately" checkbox

Response Format

Successful Import

{
  "created_count": 3,
  "error_count": 0,
  "created_tables": [
    {
      "id": 1,
      "name": "users",
      "physical_table_name": "blog_app_users",
      "provider_type": "flat_table",
      "is_materialized": true,
      "field_count": 5
    },
    {
      "id": 2,
      "name": "posts",
      "physical_table_name": "blog_app_posts",
      "provider_type": "flat_table",
      "is_materialized": true,
      "field_count": 6
    },
    {
      "id": 3,
      "name": "comments",
      "physical_table_name": "blog_app_comments",
      "provider_type": "flat_table",
      "is_materialized": true,
      "field_count": 5
    }
  ],
  "errors": [],
  "materialized": true
}

Partial Success

If some tables fail (e.g., duplicate names), you get details:

{
  "created_count": 2,
  "error_count": 1,
  "created_tables": [
    {
      "id": 1,
      "name": "users",
      "physical_table_name": "blog_app_users",
      "is_materialized": true,
      "field_count": 5
    },
    {
      "id": 3,
      "name": "comments",
      "physical_table_name": "blog_app_comments",
      "is_materialized": true,
      "field_count": 5
    }
  ],
  "errors": [
    {
      "resource_index": 1,
      "table_name": "posts",
      "error": "DataTable with name 'posts' already exists for this app"
    }
  ],
  "materialized": true
}

Provider Types

You can specify different provider types for each resource:

{
  "resources": [
    {
      "name": "users",
      "provider_type": "flat_table",
      "schema": { /* structured data */ }
    },
    {
      "name": "user_metadata",
      "provider_type": "jsonb",
      "schema": { /* flexible data */ }
    }
  ]
}

Available Providers:

• flat_table (default): Traditional PostgreSQL tables
• jsonb: PostgreSQL JSONB storage for flexible schemas
• mongodb: MongoDB collections

Best Practices

1. Start Small

Test your schema with 1-2 tables before importing large packages:

{
  "resources": [
    {
      "name": "test_table",
      "schema": { /* minimal schema */ }
    }
  ]
}

2. Use Descriptive Names

Include titles and descriptions for documentation:

{
  "name": "users",
  "title": "User Accounts",
  "description": "Registered user accounts with authentication credentials",
  "schema": { /* ... */ }
}

3. Validate Before Import

Use a JSON validator to check syntax:

• JSONLint
• Frictionless Validator

4. Version Control

Store your Data Packages in version control (Git) for:

• Schema versioning
• Rollback capability
• Team collaboration
• Documentation

5. Foreign Key References

Always use the logical resource name (not physical table name):

{
  "foreignKeys": [
    {
      "fields": ["user_id"],
      "reference": {
        "resource": "users",  // ✅ Logical name
        "fields": ["id"]
      }
    }
  ]
}

Don't use:

{
  "reference": {
    "resource": "blog_app_users"  // ❌ Physical name
  }
}

The system automatically translates logical names to physical table names.

Common Use Cases

Use Case 1: New Application Setup

Create your entire schema on first deployment:

REST API

# In your CI/CD pipeline
curl -X POST "https://your-site.taruvi.cloud/api/apps/production/datatables/create_schema/?materialize=true" \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d @production-schema.json

Python

import requests
import json

with open("production-schema.json") as f:
    schema = json.load(f)

response = requests.post(
    "https://your-site.taruvi.cloud/api/apps/production/datatables/create_schema/",
    headers={"Authorization": f"Bearer {api_token}"},
    json=schema,
    params={"materialize": "true"}
)

JavaScript

import fs from "fs";

const schema = JSON.parse(fs.readFileSync("production-schema.json", "utf-8"));

const response = await fetch(
  "https://your-site.taruvi.cloud/api/apps/production/datatables/create_schema/?materialize=true",
  {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${apiToken}`,
      "Content-Type": "application/json"
    },
    body: JSON.stringify(schema)
  }
);

Use Case 2: Development to Production

Export schema from dev, import to production:

REST API

# Export from dev
curl -X GET https://your-site.taruvi.cloud/api/apps/dev-app/datatables/schemas/ \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -o dev-schema.json

# Import to production
curl -X POST "https://your-site.taruvi.cloud/api/apps/prod-app/datatables/create_schema/?materialize=true" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d @dev-schema.json

Python

import requests

headers = {"Authorization": "Bearer YOUR_TOKEN"}

# Export from dev
dev_schema = requests.get(
    "https://your-site.taruvi.cloud/api/apps/dev-app/datatables/schemas/",
    headers=headers
).json()

# Import to production
response = requests.post(
    "https://your-site.taruvi.cloud/api/apps/prod-app/datatables/create_schema/",
    headers=headers,
    json=dev_schema,
    params={"materialize": "true"}
)

JavaScript

const headers = {
  "Authorization": "Bearer YOUR_TOKEN",
  "Content-Type": "application/json"
};

// Export from dev
const devSchema = await fetch(
  "https://your-site.taruvi.cloud/api/apps/dev-app/datatables/schemas/",
  { headers }
).then(r => r.json());

// Import to production
const response = await fetch(
  "https://your-site.taruvi.cloud/api/apps/prod-app/datatables/create_schema/?materialize=true",
  {
    method: "POST",
    headers,
    body: JSON.stringify(devSchema)
  }
);

Use Case 3: Multi-Tenant Setup

Create the same schema for multiple tenants:

REST API

# Loop through tenants
for TENANT in tenant1 tenant2 tenant3; do
  curl -X POST "https://${TENANT}.yourapp.com/api/apps/main/datatables/create_schema/?materialize=true" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @app-schema.json
done

Python

import requests
import json

with open("app-schema.json") as f:
    schema = json.load(f)

for tenant in ["tenant1", "tenant2", "tenant3"]:
    response = requests.post(
        f"https://{tenant}.yourapp.com/api/apps/main/datatables/create_schema/",
        headers={"Authorization": f"Bearer {token}"},
        json=schema,
        params={"materialize": "true"}
    )
    print(f"{tenant}: {response.status_code}")

JavaScript

import fs from "fs";

const schema = JSON.parse(fs.readFileSync("app-schema.json", "utf-8"));

for (const tenant of ["tenant1", "tenant2", "tenant3"]) {
  const response = await fetch(
    `https://${tenant}.yourapp.com/api/apps/main/datatables/create_schema/?materialize=true`,
    {
      method: "POST",
      headers: {
        "Authorization": `Bearer ${token}`,
        "Content-Type": "application/json"
      },
      body: JSON.stringify(schema)
    }
  );
  console.log(`${tenant}: ${response.status}`);
}

Troubleshooting

Error: "Data Package must contain at least one resource"

Cause: The resources array is empty or missing.

Solution:

{
  "resources": [
    { /* at least one resource */ }
  ]
}

Error: "Resource must have a name"

Cause: A resource is missing the name field.

Solution:

{
  "resources": [
    {
      "name": "my_table",  // ✅ Add name
      "schema": { /* ... */ }
    }
  ]
}

Error: "DataTable with name 'X' already exists"

Cause: You're trying to import a table that already exists.

Solution:

• Delete the existing table first, or
• Remove it from your Data Package, or
• Use a different name

Error: "Invalid Frictionless schema"

Cause: The table schema doesn't follow Frictionless specifications.

Solution: Check:

• All fields have name and type
• Primary keys reference existing fields
• Foreign keys have valid structure

See the Schema Guide for valid schema format.

Next Steps

• Schema Migrations: Learn how to update schemas safely
• Query Data: Learn how to query your imported tables
• Relationships: Work with foreign key relationships
• Schema Guide: Detailed schema documentation
• API Reference: Complete endpoint documentation

Resources

• Frictionless Data Package Specification
• Frictionless Table Schema Specification
• Example Data Packages