Skip to main content

Importing Data Packages

Learn how to bulk-import multiple tables at once using Frictionless Data Packages.

Overview​

Instead of creating tables one at a time, you can import an entire data schema using the Data Package format. This is perfect for:

  • 🚀 Quick Setup: Create your entire database schema in one request
  • 🔗 Relationships: Automatically handles foreign key dependencies
  • 📦 Portability: Export schemas from one app and import to another
  • 🎯 Consistency: Ensure all related tables are created together

What is a Data Package?​

A Data Package is a standard format for describing datasets, defined by Frictionless Data. Think of it as a blueprint for your database that includes multiple related tables.

Basic Structure​

{
  "name": "my-database-schema",
  "title": "My Application Database",
  "description": "Complete schema for my application",
  "resources": [
    {
      "name": "users",
      "schema": { /* table definition */ }
    },
    {
      "name": "posts",
      "schema": { /* table definition */ }
    }
  ]
}

Import Methods​

Method 1: API Endpoint (Programmatic)​

Use the API for automated imports, CI/CD pipelines, or scripting.

Endpoint:

POST /api/apps/{app-slug}/datatables/create_schema/

Example:

curl -X POST https://your-site.taruvi.cloud/api/apps/blog-app/datatables/create_schema/ \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d @schema.json

Method 2: Admin Interface (Manual)​

Use the Django admin for one-time imports or manual schema management.

Steps:

  1. Navigate to Django Admin → Data tables
  2. Click "Import Data Package" button
  3. Select your app
  4. Paste or upload your Data Package JSON
  5. Optionally check "Create physical tables"
  6. Click "Import Data Package"

Creating a Data Package​

Example: Blog Application​

Let's create a complete blog schema with users, posts, and comments:

{
  "name": "blog-schema",
  "title": "Blog Application Schema",
  "description": "User-generated content platform",
  "resources": [
    {
      "name": "users",
      "title": "User Accounts",
      "schema": {
        "fields": [
          {
            "name": "id",
            "type": "integer",
            "constraints": {"required": true}
          },
          {
            "name": "username",
            "type": "string",
            "constraints": {"required": true, "maxLength": 50}
          },
          {
            "name": "email",
            "type": "string",
            "format": "email",
            "constraints": {"required": true}
          },
          {
            "name": "bio",
            "type": "string"
          },
          {
            "name": "created_at",
            "type": "datetime"
          }
        ],
        "primaryKey": ["id"]
      }
    },
    {
      "name": "posts",
      "title": "Blog Posts",
      "schema": {
        "fields": [
          {
            "name": "id",
            "type": "integer",
            "constraints": {"required": true}
          },
          {
            "name": "author_id",
            "type": "integer",
            "constraints": {"required": true}
          },
          {
            "name": "title",
            "type": "string",
            "constraints": {"required": true, "maxLength": 200}
          },
          {
            "name": "content",
            "type": "string"
          },
          {
            "name": "published_at",
            "type": "datetime"
          },
          {
            "name": "status",
            "type": "string",
            "constraints": {"enum": ["draft", "published", "archived"]}
          }
        ],
        "primaryKey": ["id"],
        "foreignKeys": [
          {
            "fields": ["author_id"],
            "reference": {
              "resource": "users",
              "fields": ["id"]
            }
          }
        ]
      }
    },
    {
      "name": "comments",
      "title": "Post Comments",
      "schema": {
        "fields": [
          {
            "name": "id",
            "type": "integer",
            "constraints": {"required": true}
          },
          {
            "name": "post_id",
            "type": "integer",
            "constraints": {"required": true}
          },
          {
            "name": "user_id",
            "type": "integer",
            "constraints": {"required": true}
          },
          {
            "name": "content",
            "type": "string",
            "constraints": {"required": true}
          },
          {
            "name": "created_at",
            "type": "datetime"
          }
        ],
        "primaryKey": ["id"],
        "foreignKeys": [
          {
            "fields": ["post_id"],
            "reference": {
              "resource": "posts",
              "fields": ["id"]
            }
          },
          {
            "fields": ["user_id"],
            "reference": {
              "resource": "users",
              "fields": ["id"]
            }
          }
        ]
      }
    }
  ]
}

Smart Features​

Automatic Dependency Resolution​

The system automatically detects foreign key relationships and creates tables in the correct order:

users (no dependencies)
  ↓
posts (depends on users)
  ↓
comments (depends on posts and users)

Creation Order: users → posts → comments

You don't need to order your resources manually—the system figures it out!

Optional Materialization​

Without materialize=true (default):

  • Creates DataTable metadata entries
  • Does NOT create physical database tables yet
  • You can review and materialize later

With materialize=true:

  • Creates DataTable metadata entries
  • Creates physical database tables immediately
  • Handles dependencies automatically
  • Ready to insert data right away

API Example:

# Create metadata only
curl -X POST https://your-site.taruvi.cloud/api/apps/blog-app/datatables/create_schema/ \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d @schema.json

# Create tables immediately
curl -X POST "https://your-site.taruvi.cloud/api/apps/blog-app/datatables/create_schema/?materialize=true" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d @schema.json

Admin Example:

  • Check the "Create physical database tables immediately" checkbox

Response Format​

Successful Import​

{
  "created_count": 3,
  "error_count": 0,
  "created_tables": [
    {
      "id": 1,
      "name": "users",
      "physical_table_name": "blog_app_users",
      "provider_type": "flat_table",
      "is_materialized": true,
      "field_count": 5
    },
    {
      "id": 2,
      "name": "posts",
      "physical_table_name": "blog_app_posts",
      "provider_type": "flat_table",
      "is_materialized": true,
      "field_count": 6
    },
    {
      "id": 3,
      "name": "comments",
      "physical_table_name": "blog_app_comments",
      "provider_type": "flat_table",
      "is_materialized": true,
      "field_count": 5
    }
  ],
  "errors": [],
  "materialized": true
}

Partial Success​

If some tables fail (e.g., duplicate names), you get details:

{
  "created_count": 2,
  "error_count": 1,
  "created_tables": [
    {
      "id": 1,
      "name": "users",
      "physical_table_name": "blog_app_users",
      "is_materialized": true,
      "field_count": 5
    },
    {
      "id": 3,
      "name": "comments",
      "physical_table_name": "blog_app_comments",
      "is_materialized": true,
      "field_count": 5
    }
  ],
  "errors": [
    {
      "resource_index": 1,
      "table_name": "posts",
      "error": "DataTable with name 'posts' already exists for this app"
    }
  ],
  "materialized": true
}

Provider Types​

You can specify different provider types for each resource:

{
  "resources": [
    {
      "name": "users",
      "provider_type": "flat_table",
      "schema": { /* structured data */ }
    },
    {
      "name": "user_metadata",
      "provider_type": "jsonb",
      "schema": { /* flexible data */ }
    }
  ]
}

Available Providers:

  • flat_table (default): Traditional PostgreSQL tables
  • jsonb: PostgreSQL JSONB storage for flexible schemas
  • mongodb: MongoDB collections

Best Practices​

1. Start Small​

Test your schema with 1-2 tables before importing large packages:

{
  "resources": [
    {
      "name": "test_table",
      "schema": { /* minimal schema */ }
    }
  ]
}

2. Use Descriptive Names​

Include titles and descriptions for documentation:

{
  "name": "users",
  "title": "User Accounts",
  "description": "Registered user accounts with authentication credentials",
  "schema": { /* ... */ }
}

3. Validate Before Import​

Use a JSON validator to check syntax:

4. Version Control​

Store your Data Packages in version control (Git) for:

  • Schema versioning
  • Rollback capability
  • Team collaboration
  • Documentation

5. Foreign Key References​

Always use the logical resource name (not physical table name):

{
  "foreignKeys": [
    {
      "fields": ["user_id"],
      "reference": {
        "resource": "users",  // ✅ Logical name
        "fields": ["id"]
      }
    }
  ]
}

Don't use:

{
  "reference": {
    "resource": "blog_app_users"  // ❌ Physical name
  }
}

The system automatically translates logical names to physical table names.

Common Use Cases​

Use Case 1: New Application Setup​

Create your entire schema on first deployment:

# In your CI/CD pipeline
curl -X POST "https://your-site.taruvi.cloud/api/apps/production/datatables/create_schema/?materialize=true" \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d @production-schema.json

Use Case 2: Development to Production​

Export schema from dev, import to production:

# Export from dev
curl -X GET https://your-site.taruvi.cloud/api/apps/dev-app/datatables/schemas/ \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -o dev-schema.json

# Import to production
curl -X POST "https://your-site.taruvi.cloud/api/apps/prod-app/datatables/create_schema/?materialize=true" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d @dev-schema.json

Use Case 3: Multi-Tenant Setup​

Create the same schema for multiple tenants:

# Loop through tenants
for TENANT in tenant1 tenant2 tenant3; do
  curl -X POST "https://${TENANT}.yourapp.com/api/apps/main/datatables/create_schema/?materialize=true" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d @app-schema.json
done

Troubleshooting​

Error: "Data Package must contain at least one resource"​

Cause: The resources array is empty or missing.

Solution:

{
  "resources": [
    { /* at least one resource */ }
  ]
}

Error: "Resource must have a name"​

Cause: A resource is missing the name field.

Solution:

{
  "resources": [
    {
      "name": "my_table",  // ✅ Add name
      "schema": { /* ... */ }
    }
  ]
}

Error: "DataTable with name 'X' already exists"​

Cause: You're trying to import a table that already exists.

Solution:

  • Delete the existing table first, or
  • Remove it from your Data Package, or
  • Use a different name

Error: "Invalid Frictionless schema"​

Cause: The table schema doesn't follow Frictionless specifications.

Solution: Check:

  • All fields have name and type
  • Primary keys reference existing fields
  • Foreign keys have valid structure

See the Schema Guide for valid schema format.

Next Steps​

Resources​