🎯 Naming Strategy: Data Architecture

Date: 2025-01-16
Purpose: Establish consistent, semantic naming conventions for data architecture layers

📊 Current Situation Analysis

Your Existing Metaphors

1. database/ - Current top-level directory
2. SOURCE_OF_TRUTH/ - Proposed canonical data layer
3. output-styles/ - Generated/derived outputs
4. prompts/, storage/, knowledge/ - Operational layers

The Semantic Problem

• "database" implies a managed data storage system (PostgreSQL, Redis, etc.)
• But your directory contains:
  • ✅ Schemas & configs (canonical data)
  • ✅ Python modules (operational code)
  • ✅ Prompt templates (generation logic)
  • ✅ Examples & training data (reference materials)

This is not a "database"—it's a DATA PLATFORM.

🎨 Naming Options Analysis

Option 1: data/ ⭐⭐⭐

Industry Standard Name

data/
├── source/              # Canonical definitions
├── runtime/             # Operational modules
└── generated/           # Pipeline outputs

Pros:

• ✅ Universal convention in ML/AI
• ✅ Simple, clear, widely understood
• ✅ Works with existing tools/frameworks
• ✅ Low cognitive load

Cons:

• ❌ Too generic—doesn't convey sophistication
• ❌ Doesn't reflect multi-layer architecture
• ❌ Might imply "just data files"

Best For: Traditional ML projects, data science workflows

Option 2: data_layer/ ⭐⭐⭐⭐

Architectural Pattern Name

data_layer/
├── canonical/           # Source of truth
├── operational/         # Runtime services
└── materialized/        # Generated views

Pros:

• ✅ Communicates architectural thinking
• ✅ Implies separation of concerns
• ✅ Familiar to backend engineers
• ✅ Scales conceptually (presentation layer, logic layer, data layer)

Cons:

• ❌ Slightly longer name
• ❌ "Layer" might imply only one responsibility

Best For: Architecturally sophisticated systems with clear layer boundaries

Option 3: data_fabric/ ⭐⭐⭐⭐⭐

Modern Data Architecture Pattern

data_fabric/
├── definitions/         # Canonical schemas & configs
├── weave/              # Integration & transformation logic
└── views/              # Materialized outputs

Definition of Data Fabric:

"A data fabric is an architecture that facilitates the end-to-end integration of various data pipelines and cloud environments through the use of intelligent and automated systems." (IBM, 2024 (opens in a new tab))

Key Characteristics:

1. Unified Access: Single interface to heterogeneous data sources
2. Active Metadata: Intelligent understanding of data relationships
3. Knowledge Graph: Semantic connections between data entities
4. Automation: Self-service data access and governance

Pros:

• ✅ Perfect semantic match for your architecture
• ✅ You literally have:
  • Multiple storage backends (PostgreSQL, Redis, Vector DB)
  • Intelligent metadata (schemas, configs, examples)
  • Knowledge graph operations (embeddings, RAG)
  • Automated generation pipelines
• ✅ Modern, sophisticated terminology
• ✅ Communicates integration & orchestration
• ✅ Metaphorically rich ("fabric" = woven together)

Cons:

• ❌ Less universally known term
• ❌ Might require explanation for junior devs
• ❌ Could be seen as "buzzword-y"

Best For: Modern AI/ML platforms with:

• Multi-storage strategies
• Automated data pipelines
• Semantic understanding layers
• RAG/vector operations

Option 4: data_platform/ ⭐⭐⭐⭐

Product/Service Oriented Name

data_platform/
├── catalog/            # Data registry
├── services/           # Operational APIs
└── artifacts/          # Generated assets

Pros:

• ✅ Business-friendly terminology
• ✅ Implies productized capabilities
• ✅ Communicates value, not just structure
• ✅ Good for stakeholder communication

Cons:

• ❌ Might imply more infrastructure than exists
• ❌ "Platform" could be misleading at current scale

Best For: Data products, internal tooling, SaaS offerings

🏆 Recommendation: data_fabric/

Why Data Fabric Wins

Your system literally is a data fabric:

Your Implementation Matches Data Fabric Principles:

1. Unified Access Pattern ✅

   • Single directory structure
   • Consistent APIs across storage backends
   • Abstracted access patterns

2. Active Metadata Management ✅

   • JSON Schemas as active definitions
   • Auto-generated adapters (Pydantic, TypeScript, Drizzle)
   • Version-controlled configurations

3. Knowledge Graph Operations ✅

   • Vector embeddings in knowledge/
   • Semantic retrieval via RAG
   • Intent classification and routing

4. Automated Orchestration ✅

   • Config → Example generation
   • Schema → Adapter generation
   • Source → Runtime deployment

📐 Proposed Final Structure

data_fabric/                              # The unified data architecture
│
├── definitions/                          # Canonical source of truth
│   ├── schemas/                          # JSON Schema (canonical)
│   ├── configs/                          # Business rules & presets
│   ├── templates/                        # Prompt templates
│   └── examples/                         # Training examples (JSONL)
│
├── weave/                                # Integration & transformation
│   ├── knowledge/                        # Embeddings, RAG, retrieval
│   ├── storage/                          # Multi-backend abstractions
│   ├── prompts/                          # Dynamic prompt builders
│   └── generators/                       # Config → Example pipelines
│
├── views/                                # Materialized/generated outputs
│   ├── onboarding/                       # Pipeline outputs
│   ├── contracts/                        # Generated contracts
│   └── analytics/                        # Computed views
│
└── README.md                             # Architecture overview

Semantic Clarity

🔄 Alternative: Stick with database/ + Add Context

If changing the name is too disruptive, you could:

database/                                 # Keep existing name
├── _ARCHITECTURE.md                      # NEW: Explain it's a data fabric
├── canonical/                            # Rename: SOURCE_OF_TRUTH
├── operational/                          # Group: weave/ contents
└── materialized/                         # Group: views/ contents

Pros:

• ✅ No breaking changes
• ✅ Maintains git history
• ✅ Less migration work

Cons:

• ❌ Perpetuates semantic confusion
• ❌ Doesn't signal architectural sophistication
• ❌ New team members might misunderstand

🎯 Migration Path (If Choosing data_fabric/)

Phase 1: Rename Directory (Low Risk)

git mv database data_fabric
# Update all import paths
find . -type f -name "*.py" -exec sed -i 's/from database/from data_fabric/g' {} +

Phase 2: Restructure Internal Layout

cd data_fabric
mkdir -p definitions/schemas definitions/configs definitions/templates
mkdir -p weave/knowledge weave/storage weave/prompts
mkdir -p views/onboarding views/contracts
# Move existing files to new locations

Phase 3: Update Documentation

• Update all README files
• Regenerate architecture diagrams
• Update import statements in examples

Estimated Effort: 2-4 hours
Risk Level: Low (mostly file moves)
Breaking Changes: Import paths only

📊 Decision Matrix

🎤 Final Recommendation

Choose data_fabric/ If:

• ✅ You want to signal architectural sophistication
• ✅ Your system truly integrates multiple data sources
• ✅ You're building for scale and complexity
• ✅ Team is technically mature

Choose data_layer/ If:

• ✅ You want familiar, safe terminology
• ✅ You prioritize simplicity over precision
• ✅ Team includes junior developers
• ✅ You want broad, immediate recognition

Stick with database/ If:

• ✅ Migration effort is too high right now
• ✅ Git history preservation is critical
• ✅ External integrations reference this path
• ✅ You add clarifying documentation

📝 Subdirectory Naming (With data_fabric/)

Instead of SOURCE_OF_TRUTH/, use definitions/

Rationale:

• Shorter, more elegant
• Industry-standard term
• Pairs well with "data fabric" metaphor
• Implies "defining characteristics" of the data

Instead of Mixed Names, use Lifecycle Terms

definitions/   # What the data IS (canonical schemas, configs)
weave/         # How the data FLOWS (integration, transformation)
views/         # What the data BECOMES (materialized, generated)

Metaphor Consistency:

• Definitions = The thread (raw material)
• Weave = The loom (transformation process)
• Views = The fabric (finished product)

🔮 Future Considerations

If you adopt data_fabric/:

1. Next Addition: data_fabric/catalog/

   • Data lineage tracking
   • Data quality metrics
   • Schema registry interface

2. Next Addition: data_fabric/governance/

   • Access control policies
   • Data retention rules
   • Compliance documentation

3. Next Addition: data_fabric/observability/

   • Data flow monitoring
   • Quality dashboards
   • Performance metrics

This sets you up for true Data Fabric capabilities long-term.

🏁 TL;DR

Recommended: data_fabric/ with subdirectories:

• definitions/ (not SOURCE_OF_TRUTH/)
• weave/ (operational logic)
• views/ (materialized outputs)

Why: Your architecture literally is a data fabric—unified access, active metadata, knowledge graph operations, and automated orchestration across multiple storage backends.

Migration Effort: 2-4 hours (mostly imports)

Alternative: Keep database/ but add _ARCHITECTURE.md explaining it's a data fabric implementation.