Primary DB
PostgreSQL
redwood-postgres-db-config
Secondary DB
Azure SQL
redwood-azure-sql-config
Hot Storage
Azure Blob
redwood-hot-storage-config
Cold Archive
Azure Blob
redwood-cold-storage-config
Secrets
Key Vault
airflowconnections
Bronze
Bronze Layer — Raw Ingest
5 data sources · 79 tables · 8 Airflow DAGs · PostgreSQL + Azure SQL
Schwab
File-based custodian feed via Azure Blob Storage. Fixed-width / CSV with SHA-256 dedup.
15Tables
12 PMUTC Daily
Full + IncSync mode
DailyArchive
View docs →
Data Pipeline ↗
Fidelity
Fixed-width COBOL-spec file feed via Azure Blob Storage. Keyed by customer code.
11Tables
10 AMUTC Daily
IncrementalSync mode
WeeklyArchive
View docs →
Data Pipeline ↗
Salesforce
CRM via REST / Bulk API with OAuth 2.0. 20 objects including Financial Services Cloud.
20Objects
5 AMUTC Daily
Smart syncSync mode
NoneArchive
View docs →
Data Pipeline ↗
Tamarac — PostgreSQL
Portfolio management API with PFX cert auth. Writes to PostgreSQL via async bulk insert.
17Tables
2:30 PMUTC Daily
PostgreSQLTarget
NoneArchive
View docs →
Data Pipeline ↗
Tamarac — Azure SQL
Portfolio management API with PFX cert auth. Writes to Azure SQL via pyodbc with blob staging.
16Tables
1:00 PMUTC Daily
Azure SQLTarget
BlobStaging
View docs →
Data Pipeline ↗
Bronze Layer — Full Database Documentation
Complete SQL schema reference for all 79 bronze tables across PostgreSQL and Azure SQL — column definitions, data types, primary keys, constraints.
Schwab: 15 tables
Fidelity: 11 tables
Salesforce: 20 tables
Tamarac PG: 17 tables
Tamarac SQL: 16 tables
Open reference →
Silver
Silver Layer — Curated Views & Indices
16 business-ready SQL views joining bronze tables · optimised with targeted indices
Gold
Gold Layer — Aggregations & Reporting
Materialised aggregations consumed by dashboards, client statements, and executive reporting
Gold Layer
Coming Soon
Aggregated, KPI-ready tables and materialised views built on top of the Silver layer — optimised for dashboard queries, client reporting, advisor performance scorecards, and executive summaries.
AUM by household / advisor
Monthly performance summary
Cash flow trends
Advisor activity scorecard
Client health dashboard
Fee revenue by period
Sales pipeline KPIs
Schedule
Daily Pipeline Schedule (UTC)
All Bronze ingest jobs — click any DAG name to open in Airflow
Daytime Sync Jobs
Monday–Sunday · all times UTC
→
→
→
→
→
Off-hours Archive Jobs
→
Archive DAG Summary
| DAG | Schedule | UTC Time | Scope | Retention |
|---|---|---|---|---|
redwood_database_archive |
0 2 * * * | Daily 2:00 AM | PostgreSQL table dumps → cold storage ZIP | Permanent |
redwood_schwab_file_archiver |
0 14 * * * | Daily 2:00 PM | Schwab hot-tier files → cold storage ZIP | 7 days hot |
redwood_fidelity_archiver |
0 3 * * 0 | Weekly Sun 3:00 AM | Fidelity hot-tier files → cold storage ZIP | 7 days hot |
Azure Key Vault Secrets
| Secret Name | Used By | Content |
|---|---|---|
redwood-postgres-db-config | Schwab Fidelity Salesforce Tamarac PG | host, port, database, user, password |
redwood-azure-sql-config | Tamarac AzureSQL | Azure SQL connection string (pyodbc) |
redwood-tamarac-api-config | Tamarac | API base URL, credentials |
redwood-tamarac-prod-cert-pfx | Tamarac | Base64-encoded PFX certificate |
redwood-hot-storage-config | Schwab Fidelity Tamarac | Azure Blob hot tier account URL + key |
redwood-cold-storage-config | All archive DAGs | Azure Blob cold tier account URL + key |
redwood-salesforce-config | Salesforce | OAuth client_id, client_secret, instance_url |
All secrets in the airflowconnections Azure Key Vault, accessed via AzureKeyVaultHook.
Email Notification Recipients — All Redwood DAGs
| Role | |
|---|---|
prashant.surana@collation.ai | Pipeline owner |
monitor@collation.ai | Monitoring alias |
mschwartz@sequoia-financial.com | Client contact |
handerson@sequoia-financial.com | Client contact |
vzivich@sequoia-financial.com | Client contact |
hpatel@sequoia-financial.com | Client contact |
karishni.mehta@collation.ai | Engineering |