Data Connection in 10min

In this video, "Data Connection in 10min" by Ontologize, former Palantir engineers break down how data enters and exits Palantir Foundry [00:00].

Below is an end-to-end operational guide and use-case analysis based on the concepts, architecture, and realistic examples (like the Ontologize Theme Park) presented in the tutorial.

---

Architectural Process Map: Ingestion to Ontology Action

Understanding data movement through Foundry requires looking at Ingress (bringing data into datasets) vs. Egress (triggering operational webhooks):

                   PALANTIR FOUNDRY DATA CONNECTION
┌────────────────────────────────────────────────────────────────────────┐
│ [ External Data Sources ]                                             │
│   ├── Public Cloud / APIs (Snowflake, S3) ──> Direct Connection        │
│   └── On-Prem / VPC (Kafka, ERP)          ──> Agent-Based (Polled)     │
└──────────────────────────────────┬─────────────────────────────────────┘
                                   │
                                   ▼
┌────────────────────────────────────────────────────────────────────────┐
│ [ Sync Execution (Ingress) ]                                           │
│   ├── Files/Unstructured ──> Snapshot / Incremental ──> Dataset Extract│
│   ├── Tabular Data      ──> Append / Snapshot      ──> Dataset Extract│
│   └── Streaming         ──> Continuous Stream      ──> Real-Time Stream│
└──────────────────────────────────┬─────────────────────────────────────┘
                                   │
                                   ▼
┌────────────────────────────────────────────────────────────────────────┐
│ [ Data Pipeline & Transformation ]                                     │
│   └── Pipeline Builder / PySpark ──> Input Trigger ──> Clean Datasets  │
└──────────────────────────────────┬─────────────────────────────────────┘
                                   │
                                   ▼
┌────────────────────────────────────────────────────────────────────────┐
│ [ Ontology & Operational Egress ]                                      │
│   └── Backed Object Types ──> Object Monitor / Rule Alert              │
│                                  │                                     │
│                                  ▼                                     │
│                            Action / Webhook (Twilio / External API)    │
└────────────────────────────────────────────────────────────────────────┘

---

Step-by-Step Guidance: Setting Up Data Connection & Syncs

1. Define and Provision Data Sources: Choose between Direct or Agent-Based connection strategies.
2. Navigate to the Data Connection application in Foundry [00:00].
3. Click + New Source and choose the connection model based on your network architecture [01:06]:

• Direct Connection: Ideal for public internet endpoints (e.g., Snowflake, AWS S3, REST APIs) [01:14]. Follow the wizard or select generic connectors like JDBC [01:34].
• Agent-Based Connection: Required for private subnets, on-premises DBs, or VPCs [01:57]. Download the agent installer onto your infrastructure (e.g., an Azure VM monitoring a local Kafka broker) [02:22]. The agent polls Foundry outbound for instructions and pushes data back [02:14].

1. Configure credentials and define user/group access permissions directly on the source [00:45].

2. Configure Network Egress Policies: Ensure security rules permit inbound/outbound calls.

3. By default, Foundry restricts outbound calls to external hostnames [08:16].

4. During source setup or sync configuration, check for recommended egress domains suggested by the Data Connection wizard [08:37].

5. Add required target domains/IPs to your organization's Network Egress Policies before running syncs [08:33].

6. Configure and Run Syncs: Define data extraction frequency and write modes.

7. Select your source and click Explore Source to browse buckets, tables, or topics [03:28].

8. Choose the extraction strategy suited to your data shape [03:39]:

• File/JSON (e.g., AWS S3): Choose file/folder paths. Set snapshot frequency (e.g., daily at midnight) [03:45].
• Tabular (e.g., Snowflake): Select tables/views. Choose Append mode (adding new rows) and Incremental syncs to only ship changed data [04:04].
• Streaming (e.g., Kafka): Set up continuous streams for sensor telemetry [02:28, 04:22].

1. Execute the sync and monitor progress inside the Builds App (Job Tracker) [05:08].

2. Build Pipelines & Set Input Triggers: Connect raw sync extracts to downstream schedules.

3. Open Pipeline Builder or a Code Repository and pull in the dataset extract produced by the sync [05:51].

4. Transform raw data into cleansed operational datasets backing Ontology objects [05:32].

5. Configure dataset schedules using Input Triggers so downstream pipelines automatically re-run when new sync data lands [06:04].

6. Set Up Webhooks for Operational Egress: Configure write-back/actions to external systems.

7. Create a Source configured for external API calls (e.g., Twilio, SAP, or REST endpoints) [07:04].

8. Define a Webhook specifying the HTTP method, payload schema, and endpoint URL [07:04].

9. Attach the webhook to an Ontology Action triggered via Workshop buttons, Object Monitors, or Foundry Rules [07:12].

---

Realistic Use Case Analysis: The "Ontologize Theme Park"

The video illustrates these concepts using an operational theme park scenario.

1. Ingestion Strategy Breakdown

---

2. Best Practices & Optimization Highlights

Avoiding Compute Waste (Pipeline Schedule Design)

• Problem: Setting multiple sync datasets as Input Triggers for a single downstream pipeline causes the entire pipeline to re-run every time any single source updates [06:17].
• Impact: If Source A updates daily and Source B updates daily at different times, a dual-trigger pipeline runs twice per day, consuming 2x compute power unnecessarily [06:32].
• Solution: Designate a single primary dataset (e.g., the high-frequency guest ride log) as the sole input trigger, keeping static dimension datasets (like rides.json) as non-trigger inputs [06:45].

Webhook Egress vs. Bulk Sync

• Syncs: Designed for scheduled or continuous bulk data ingestion into Foundry datasets [08:08].
• Webhooks: On-demand, transactional event egress [08:08].
• Example: In the video, when an Object Monitor detects a dip in ridership across roller coasters, it triggers an Action that executes a Twilio Webhook to send an automated SMS alert to the park sector manager [07:31].

---