Instance-Scoped Testing Architecture - Best Practices

Instance-Scoped Testing Architecture - Best Practices

Date: 2025-09-29
Version: 1.0
Status: 📋 RECOMMENDED ARCHITECTURE

🎯 Core Principle

Every testing scope should have its own isolated Quality Fabric instance with dedicated configuration, data, and context.

This follows the "Testing as a Service Instance" pattern where Quality Fabric acts as a reusable testing engine that can be instantiated multiple times with different configurations.

🏗️ Recommended Architecture

Current Structure Issues

Problem 1: Test data mixed with core code

quality-fabric/
├── data/
│   ├── test_history.json              # ❌ Mixed: QF self-test data
│   ├── maestro_frontend_history.json  # ❌ Mixed: Maestro frontend data
│   └── maestro_backend_history.json   # ❌ Mixed: Maestro backend data

Problem 2: Test documentation mixed with platform docs

quality-fabric/
├── MAESTRO_PREDICTIVE_TESTING.md      # ❌ Mixed: Project-specific
├── MAESTRO_REAL_TEST_RESULTS.md       # ❌ Mixed: Project-specific
├── MAESTRO_TEST_INSTANCES.md          # ❌ Mixed: Project-specific
└── DEEP_FRONTEND_TESTING_GUIDE.md     # ❌ Mixed: Project-specific

Problem 3: No clear instance isolation

• Each project shares the same Quality Fabric codebase
• Configuration not scoped per instance
• Risk of data contamination between projects

✅ Recommended Structure

Option 1: Instance Directory Pattern (RECOMMENDED)

quality-fabric/                           # Core platform (reusable engine)
├── services/                            # Core services (shared)
│   ├── ai/                              # Predictive testing engine
│   ├── api/                             # REST API endpoints
│   ├── core/                            # Test orchestration
│   └── adapters/                        # Framework adapters
│
├── configs/                             # Default configs (templates)
│   └── templates/
│       ├── predictive_testing.yaml
│       ├── frontend_testing.yaml
│       └── backend_testing.yaml
│
├── docs/                                # Platform documentation only
│   ├── PREDICTIVE_TEST_SELECTION.md
│   ├── SERVICE_VIRTUALIZATION_GUIDE.md
│   └── YAML_SERVICE_ENHANCEMENT.md
│
└── test-instances/                      # 🆕 Isolated test instances
    │
    ├── quality-fabric-self/             # 🆕 QF self-testing instance
    │   ├── .qf-instance.yaml            # Instance configuration
    │   ├── data/
    │   │   └── test_history.json        # Scoped ML data
    │   ├── configs/
    │   │   └── test_config.yaml         # Instance-specific config
    │   ├── docs/
    │   │   └── TEST_RESULTS.md          # Instance documentation
    │   └── scripts/
    │       └── run_tests.py             # Instance test runner
    │
    ├── maestro-frontend/                # 🆕 Maestro frontend instance
    │   ├── .qf-instance.yaml
    │   ├── data/
    │   │   ├── test_history.json        # Frontend ML data
    │   │   └── test_results/
    │   ├── configs/
    │   │   ├── vitest.config.yaml       # Frontend test config
    │   │   └── predictive_config.yaml
    │   ├── docs/
    │   │   ├── TESTING_GUIDE.md
    │   │   └── TEST_RESULTS.md
    │   ├── scripts/
    │   │   ├── analyze_pr.sh
    │   │   └── run_smart_run.sh
    │   └── tests/                       # Optional: test definitions
    │       ├── unit/
    │       ├── integration/
    │       └── e2e/
    │
    └── maestro-backend/                 # 🆕 Maestro backend instance
        ├── .qf-instance.yaml
        ├── data/
        │   ├── test_history.json        # Backend ML data
        │   ├── dependency_graph.json    # Cached dependency data
        │   └── test_results/
        ├── configs/
        │   ├── pytest.config.yaml       # Backend test config
        │   └── predictive_config.yaml
        ├── docs/
        │   ├── TESTING_GUIDE.md
        │   ├── TEST_RESULTS.md
        │   └── DEPENDENCY_ANALYSIS.md
        └── scripts/
            ├── analyze_pr.py
            └── run_smart_run.py

📋 Instance Configuration File (.qf-instance.yaml)

Each test instance has its own configuration:

# test-instances/maestro-frontend/.qf-instance.yaml

instance:
  name: "maestro-frontend"
  type: "frontend"
  version: "1.0.0"
  created: "2025-09-29"

  # Link to target repository
  repository:
    path: "maestro-frontend"
    type: "git"
    main_branch: "main"

  # Quality Fabric engine configuration
  quality_fabric:
    core_path: "quality-fabric"
    api_url: "local-service"
    instance_id: "maestro-frontend-prod"

  # Instance-specific data paths (relative to instance directory)
  data:
    test_history: "data/test_history.json"
    ml_models: "data/models/"
    test_results: "data/test_results/"
    cache: "data/cache/"

  # Testing configuration
  testing:
    framework: "vitest"
    test_command: "npm run test"
    coverage_threshold: 80
    parallel_execution: true
    max_workers: 4

  # Predictive testing configuration
  predictive:
    enabled: true
    confidence_threshold: 0.70
    risk_threshold: 60
    ml_training_enabled: true
    min_history_records: 50

  # Source code analysis
  analysis:
    source_dirs:
      - "src"
      - "components"
    test_dirs:
      - "src/__tests__"
      - "tests"
    exclude_patterns:
      - "node_modules"
      - "dist"
      - "build"

  # Integration settings
  integrations:
    ci_cd:
      platform: "github-actions"
      webhook_url: "https://api.github.com/repos/maestro/frontend/hooks"
    notifications:
      slack: true
      email: false

  # Environment-specific settings
  environments:
    development:
      api_url: "local-service"
      mock_services: true
    staging:
      api_url: "https://staging-qf.example.com"
      mock_services: false
    production:
      api_url: "https://qf.example.com"
      mock_services: false

🔧 Instance Initialization

Creating a New Test Instance

# Command-line interface
python -m quality_fabric.cli instance create \
  --name "maestro-frontend" \
  --type "frontend" \
  --repo "maestro-frontend" \
  --framework "vitest"

# This creates:
# test-instances/maestro-frontend/
#   ├── .qf-instance.yaml (from template)
#   ├── data/
#   ├── configs/
#   ├── docs/
#   └── scripts/

Instance Lifecycle Commands

# Initialize instance
qf instance init maestro-frontend

# Validate instance configuration
qf instance validate maestro-frontend

# Start instance (start API with instance context)
qf instance start maestro-frontend --port 8001

# Analyze repository
qf instance analyze maestro-frontend --commit HEAD

# Run tests
qf instance test maestro-frontend --smart-run

# Export instance data
qf instance export maestro-frontend --output ./backup.tar.gz

# Clone instance
qf instance clone maestro-frontend maestro-frontend-staging

# Destroy instance (archive and remove)
qf instance destroy maestro-frontend --archive

🎯 Benefits of Instance-Scoped Architecture

1. Complete Isolation ✅

• Each instance has its own data, config, and ML models
• No data contamination between projects
• Independent versioning and rollback

2. Multi-Project Support ✅

• Test multiple projects simultaneously
• Each project has dedicated resources
• Parallel execution without conflicts

3. Environment Parity ✅

• Same instance structure for dev/staging/prod
• Easy to replicate environments
• Consistent testing across environments

4. Scalability ✅

• Add new projects without modifying core
• Horizontal scaling (multiple instances)
• Distributed testing across instances

5. Maintenance ✅

• Update core without affecting instances
• Instance-specific tuning and optimization
• Clear separation of concerns

6. Portability ✅

• Export/import entire test instances
• Share instance configurations across teams
• Easy migration between environments

🔄 API Request Flow with Instance Context

Before (No Instance Context)

# All projects share same Quality Fabric
POST /api/predictive/analyze-pr
{
  "repo_path": "maestro-frontend",
  "base_ref": "main",
  "head_ref": "HEAD"
}
# ❌ Problem: Where is ML data stored? How to isolate?

After (With Instance Context)

# Instance-aware API
POST /api/instances/maestro-frontend/predictive/analyze-pr
{
  "base_ref": "main",
  "head_ref": "HEAD"
}
# ✅ Solution: Instance context provides all paths and config

Instance Context Loading

# services/api/testing_service_api.py

@app.post("/api/instances/{instance_id}/predictive/analyze-pr")
async def analyze_pr_instance(
    instance_id: str,
    request: PRAnalysisRequest
):
    # Load instance configuration
    instance = InstanceManager.load(instance_id)

    # Initialize selector with instance context
    selector = PredictiveTestSelector(
        repo_path=instance.repository.path,
        test_history_path=instance.data.test_history,
        config=instance.config
    )

    # Run analysis with instance-scoped data
    result = await selector.analyze_pr(
        base_ref=request.base_ref,
        head_ref=request.head_ref
    )

    # Save results to instance data directory
    await instance.save_results(result)

    return result

🏭 Production Deployment Pattern

Single Quality Fabric Core + Multiple Instances

┌─────────────────────────────────────────────────────────┐
│           Quality Fabric Core (Single Installation)     │
│                                                          │
│  ┌────────────┐  ┌────────────┐  ┌─────────────┐      │
│  │ API Server │  │ ML Engine  │  │  Orchestrator│      │
│  └────────────┘  └────────────┘  └─────────────┘      │
└────────────┬────────────┬────────────┬──────────────────┘
             │            │            │
    ┌────────┴─┐    ┌────┴────┐    ┌─┴────────┐
    │          │    │         │    │          │
┌───▼────┐ ┌──▼──────┐ ┌────▼──────┐ ┌──────▼─────┐
│Instance│ │ Instance│ │ Instance  │ │  Instance  │
│   A    │ │    B    │ │    C      │ │     D      │
└────────┘ └─────────┘ └───────────┘ └────────────┘
Frontend   Backend     Mobile App    Microservice
Testing    Testing     Testing       Testing

Key Points:

• One Quality Fabric installation serves multiple instances
• Each instance is lightweight (configs + data + docs)
• Core services are shared (efficiency)
• Instance data is isolated (security)

📝 Migration Plan from Current Structure

Phase 1: Create Instance Structure (Week 1)

# 1. Create test-instances directory
mkdir -p test-instances

# 2. Create quality-fabric-self instance
mkdir -p test-instances/quality-fabric-self/{data,configs,docs,scripts}
mv data/test_history.json test-instances/quality-fabric-self/data/

# 3. Create maestro-frontend instance
mkdir -p test-instances/maestro-frontend/{data,configs,docs,scripts}
mv data/maestro_frontend_history.json test-instances/maestro-frontend/data/test_history.json
mv MAESTRO_PREDICTIVE_TESTING.md test-instances/maestro-frontend/docs/
mv DEEP_FRONTEND_TESTING_GUIDE.md test-instances/maestro-frontend/docs/

# 4. Create maestro-backend instance
mkdir -p test-instances/maestro-backend/{data,configs,docs,scripts}
mv data/maestro_backend_history.json test-instances/maestro-backend/data/test_history.json
mv MAESTRO_REAL_TEST_RESULTS.md test-instances/maestro-backend/docs/

Phase 2: Update API to Support Instances (Week 1)

# services/api/instance_manager.py (NEW)
class InstanceManager:
    """Manages Quality Fabric test instances"""

    @staticmethod
    def load(instance_id: str) -> Instance:
        """Load instance configuration"""
        instance_path = f"test-instances/{instance_id}"
        config_path = f"{instance_path}/.qf-instance.yaml"

        with open(config_path, 'r') as f:
            config = yaml.safe_load(f)

        return Instance(
            id=instance_id,
            path=instance_path,
            config=config
        )

    @staticmethod
    def list() -> List[str]:
        """List all instances"""
        return [d for d in os.listdir("test-instances")
                if os.path.isdir(f"test-instances/{d}")]

    @staticmethod
    def create(name: str, config: dict) -> Instance:
        """Create new instance"""
        # Implementation

Phase 3: Update Existing Scripts (Week 1)

# scripts/test_maestro_real.py (UPDATED)

# Before
selector = PredictiveTestSelector(
    repo_path="maestro-frontend",
    test_history_path="data/maestro_frontend_history.json"
)

# After
from services.api.instance_manager import InstanceManager

instance = InstanceManager.load("maestro-frontend")
selector = PredictiveTestSelector(
    repo_path=instance.repository.path,
    test_history_path=instance.get_data_path("test_history.json"),
    config=instance.config
)

Phase 4: Documentation Updates (Week 2)

• Move project-specific docs to instance directories
• Update core docs to reference instance pattern
• Create instance creation guides
• Update API documentation

🎯 Comparison: Before vs After

🚀 Quick Start: Creating Your First Instance

Step 1: Create Instance Directory

cd quality-fabric

# Create instance structure
mkdir -p test-instances/my-project/{data,configs,docs,scripts}

Step 2: Create Instance Configuration

# Create .qf-instance.yaml
cat > test-instances/my-project/.qf-instance.yaml << 'EOF'
instance:
  name: "my-project"
  type: "backend"
  repository:
    path: "/path/to/my-project"
    type: "git"
  quality_fabric:
    core_path: "quality-fabric"
    api_url: "local-service"
  data:
    test_history: "data/test_history.json"
  testing:
    framework: "pytest"
  predictive:
    enabled: true
EOF

Step 3: Initialize Instance Data

# Create empty test history
echo '{"executions": [], "version": "1.0"}' > test-instances/my-project/data/test_history.json

# Create README
cat > test-instances/my-project/docs/README.md << 'EOF'
# My Project - Quality Fabric Test Instance

Quality Fabric instance for testing my-project.

## Quick Start

Analyze PR

curl -X POST local-service/api/instances/my-project/predictive/analyze-pr \
-H "Content-Type: application/json" \
-d '{"base_ref": "main", "head_ref": "HEAD"}'

EOF

Step 4: Use Instance

# Python script
from services.api.instance_manager import InstanceManager
from services.ai.predictive_test_selector import PredictiveTestSelector

# Load instance
instance = InstanceManager.load("my-project")

# Initialize selector with instance context
selector = PredictiveTestSelector(
    repo_path=instance.repository.path,
    test_history_path=instance.get_data_path("test_history.json"),
    config=instance.config.testing
)

# Run analysis
result = selector.analyze_commit("HEAD")
print(f"Selected {len(result.selected_tests)} tests")

📊 Success Criteria

✅ Isolation: Each instance has completely separate data and configuration
✅ Scalability: Can create unlimited instances without conflicts
✅ Portability: Instances can be exported, imported, and cloned
✅ Maintainability: Core updates don't affect instances
✅ Clarity: Clear separation between core platform and test instances
✅ Documentation: Platform docs vs instance-specific docs are separate

🎉 Summary

Key Recommendations

1. Adopt Instance-Scoped Architecture ✅
- Create test-instances/ directory
- Move project-specific data, configs, docs to instances
- Keep core platform clean and reusable

2. Use Instance Configuration Files ✅
- .qf-instance.yaml for each instance
- Defines repository, data paths, testing config
- Enables instance lifecycle management

3. Update API to be Instance-Aware ✅
- Endpoints like /api/instances/{id}/predictive/analyze-pr
- Automatic context loading from instance config
- No need to pass repo_path repeatedly

4. Implement Instance Manager ✅
- Central service to load/create/manage instances
- Provides instance lifecycle commands
- Enables multi-instance orchestration

5. Clear Separation of Concerns ✅
- Core platform = reusable engine
- Instances = project-specific testing contexts
- No mixing of data or documentation

Next Steps: Implement Phase 1 migration and create InstanceManager service.

Status: 📋 RECOMMENDED - Ready for Implementation
Date: 2025-09-29