Senior Full-Stack AI Architecture Project

AMIF v1.0 Complete Project Report

Autonomous Multi-Modal Intelligence Fabric is an end-to-end incident intelligence platform that fuses camera, sensor, audio, document, and API signals into evidence-backed investigations, safe action plans, dashboards, audit logs, and graph-based context.

Open PDF Open PPTX Architecture Page API Docs when app is running

Executive Summary

AMIF is a production-style MVP with professional architecture boundaries. It runs locally while preserving a clean path to real Redpanda workers, Qdrant semantic memory, Redis episodic memory, YOLO/Whisper inference, LangGraph agents, and enterprise integrations.

Project Files

Generated source, docs, infra, reports

Approx Lines

Backend, frontend, docs, infra

API Routers

Auth, events, agents, graph, audit

Services

Event, processor, agent, search, action

Connectors

Camera, audio, IoT, docs, email

Real-World Use Cases

Factory Safety

Detect forklifts or people in restricted zones and correlate with sensor anomalies.

Predictive Maintenance

Monitor overheating, vibration, and noise; retrieve manuals and recommend inspection steps.

Warehouse Operations

Monitor unsafe vehicle movement, PPE gaps, and zone breaches with supervisor alerts.

Security Operations

Correlate unauthorized access, camera evidence, and operator notes into reports.

Compliance Audit

Trace every event, alert, action, agent run, and human acknowledgement.

Smart Facilities

Combine HVAC sensors, cameras, alarms, access systems, and maintenance documents.

Architecture Overview

The architecture is event-driven, memory-backed, agentic, and governance-first.

Ingestion

CamerasSensorsAudioDocumentsWebhooks

Processing

ValidationDeduplicationEnrichmentCorrelation

Memory

PostgreSQLRedis-readyQdrant-readyKnowledge Graph

Agents

ObserverRetrieverInvestigatorPlannerGuard

Mermaid Diagrams

These Mermaid definitions can be copied into Mermaid Live Editor, GitHub Markdown, Notion, or project docs. Inline SVG renderings are also included so this report works offline.

High-Level Architecture

flowchart TB

flowchart TB
  DS[Data Sources
Cameras | Sensors | Audio | Documents | APIs] --> ING[Ingestion Layer
Connectors + FastAPI APIs]
  ING --> GW[API Gateway
FastAPI + JWT + RBAC]
  GW --> SVC[Domain Services
Event | Vision | Audio | Sensor | Document | Search | Agent]
  SVC --> BUS[(Redpanda-ready Event Bus
raw.events | incident.events | alert.events | agent.tasks)]
  BUS --> PROC[Event Processing
Validation | Dedup | Enrichment | Correlation]
  PROC --> MEM[Memory Fabric
Postgres | Redis-ready | Qdrant-ready | Knowledge Graph]
  MEM --> AG[Agent Runtime
Observer -> Retriever -> Investigator -> Planner -> Guard -> Executor]
  AG --> ACT[Action Engine
Alerts | Tickets | Webhooks | Audit]
  ACT --> UI[Operator Console
Dashboard | Evidence | Trace | Graph | Observability]

Incident Demo Sequence

sequenceDiagram

sequenceDiagram
  participant U as Operator
  participant API as FastAPI Gateway
  participant E as Event Service
  participant P as Event Processor
  participant M as Memory Fabric
  participant A as Agent Runtime
  participant UI as Console
  U->>API: Seed Demo / Emit Events
  API->>E: Canonical AMIFEvent
  E->>P: Process event
  P->>P: derive temperature_anomaly
  P->>P: correlate forklift + overheating
  P->>M: persist incident, alert, audit
  P->>A: trigger investigation
  A->>M: retrieve events + documents
  A->>A: investigate, plan, guard
  A->>M: store agent run
  UI->>API: fetch incidents, alerts, graph
  API-->>UI: evidence-backed response

Agent Runtime State Machine

stateDiagram-v2

stateDiagram-v2
  [*] --> Observer
  Observer --> Retriever: incident accepted
  Retriever --> Investigator: evidence + context
  Investigator --> Planner: hypotheses
  Planner --> SafetyGuard: action plan
  SafetyGuard --> Executor: approved actions
  SafetyGuard --> HumanApproval: risky actions
  Executor --> [*]
  HumanApproval --> [*]

Memory Fabric

flowchart LR

flowchart LR
  Incident[Incident Context] --> PG[(PostgreSQL
Historical Memory)]
  Incident --> Redis[(Redis-ready
Episodic Memory)]
  Incident --> Qdrant[(Qdrant-ready
Semantic Memory)]
  Incident --> Graph[(Knowledge Graph
Relationships)]
  PG --> Builder[Context Builder]
  Redis --> Builder
  Qdrant --> Builder
  Graph --> Builder
  Builder --> Agent[Agent Runtime]

Backend Implementation

Implemented API Areas

Authentication and JWT
RBAC-ready role model
Canonical event ingestion
Vision mock detection
Audio mock transcription
Sensor event emission
Incident update workflow
Agent investigation API
Document upload and search
Knowledge graph endpoint
Audit logs
System stats and metrics

Repository Structure

amif/
  backend/app/
    main.py
    routers/        auth, events, incidents, alerts, documents, audio, actions, audit, users, knowledge, system
    services/       event_service, event_processor, agent_service, search_service, document_service, action_service
    models/         SQLAlchemy operational models
    schemas/        Pydantic API/event contracts
    static/         advanced vanilla-JS AMIF Console
  connectors/       camera, audio, IoT, document, email connector stubs
  infra/            Docker Compose, Prometheus, Kubernetes starter manifests
  docs/             architecture, API, evaluation, frontend UX
  reports/          PDF, PPTX, HTML project reports

Core Endpoints

Endpoint	Purpose
POST /api/auth/login	Authenticate and receive JWT
POST /api/demo/seed	Run the complete incident demo
POST /api/events	Ingest canonical AMIF event
GET /api/incidents	List incident queue
POST /api/incidents/{id}/investigate	Run agent investigation
POST /api/documents/upload	Upload and index manuals
POST /api/search/query	Semantic retrieval / RAG search
GET /api/knowledge/graph	Return graph nodes and relationships

Frontend Operator Console

The frontend is an advanced dependency-free vanilla JavaScript SPA with polished UI/UX and fully integrated API functionality.

Dashboard UX

Animated KPI counters, overview cards, service health, timelines, alert cards.

Incident Command

Master/detail queue, filters, evidence panel, agent actions, resolve/acknowledge flow.

Documents & RAG

Upload documents, index semantic chunks, query retrieval results with scores.

Agent Trace

View agent output, evidence, guard decisions, and workflow trace.

Knowledge Graph

Interactive SVG graph with clickable nodes and relationship inspection.

Advanced JS

Command palette, live refresh, theme toggle, modals, toasts, export, charts.

AI Components and Integration Path

AI Area	Current MVP	Production Upgrade
Vision AI	Mock YOLO/RT-DETR-style event boundary	YOLOv8/YOLOv11, RT-DETR, Florence-2
Audio AI	Mock Whisper-style transcription event	Whisper or faster-whisper
Document AI	Text cleanup, chunking, deterministic embedding fallback	PyMuPDF, OCR, BGE-M3 embeddings
RAG	Local vector/hash + keyword scoring	Qdrant, reranking, metadata filters
Agentic AI	Deterministic Observer/Retriever/Investigator/Planner/Guard	LangGraph + Llama/Qwen/DeepSeek
Safety AI	Policy-based guard requiring human approval	Llama Guard / enterprise policy classifier

End-to-End Demo Flow

Manual uploaded

Machine A safety manual is chunked and indexed.

Camera event

Forklift detected in restricted zone.

Sensor event

Machine A temperature exceeds threshold.

Correlation

Processor links visual and sensor evidence in 10-minute window.

Incident + alert

High-severity incident, alert, and ticket stub are created.

Agent investigation

Retriever finds manual evidence; Investigator and Planner create summary.

Guardrails

Safety Guard approves alerts/tickets and requires human approval for risky actions.

Dashboard

Operator sees timeline, evidence, graph, logs, and agent trace.

Canonical Event Example

{
  "schema_version": "1.0",
  "event_id": "evt_demo_forklift_001",
  "source_id": "camera_warehouse_a_01",
  "source_type": "camera",
  "event_type": "forklift_detected",
  "severity": "medium",
  "tenant_id": "demo_tenant",
  "location": {
    "site": "plant_1",
    "zone": "restricted_zone"
  },
  "payload": {
    "object": "forklift",
    "confidence": 0.94,
    "bbox": [
      120,
      84,
      420,
      360
    ]
  },
  "trace": {
    "producer": "vision-service-mock"
  }
}

Logs, Audit Trail, and Agent Trace Visualization

AMIF is designed for auditable AI. Every important event, incident, action, and agent output can be traced.

10:30:00.120event.ingestedcamera_warehouse_a_01 forklift_detected severity=medium

10:30:01.004event.processedvalidated schema_version=1.0 dedupe=pass

10:31:10.451event.ingestedtemp_sensor_machine_a temperature_reading 91.5C

10:31:10.612anomaly.generatedtemperature_anomaly deviation=11.5C severity=high

10:31:11.002incident.createdforklift + overheating correlation window=10m

10:31:11.220alert.createdhigh-priority incident alert opened

10:31:12.140agent.startedObserver -> Retriever -> Investigator -> Planner

10:31:15.844rag.searchmanual chunk retrieved: machine_a_safety_manual.txt#0

10:31:17.400guard.decisionzone restriction requires human approval

10:31:17.900agent.completedrisk_score=0.86 summary stored

Agent Trace

Observer

Incident accepted for investigation.

Retriever

Fetched related events and semantic document evidence.

Investigator

Generated hypotheses around overheating and restricted-zone activity.

Safety Guard

Approved alerts and ticketing; blocked zone restrictions without human approval.

Knowledge Graph Visual

Observability and Metrics

Platform Metrics

Event Throughput

78%

Incident Creation

64%

Agent Activity

56%

Semantic Memory

42%

Deployment Stack

Docker ComposePostgresRedisQdrantRedpandaPrometheusGrafanaKubernetes manifests

Security and Governance

Authentication

JWT login flow with hashed passwords and API protection.

Authorization

Roles: Admin, Operator, Analyst, Viewer, ServiceAccount.

Auditability

Audit logs track event ingestion, incident creation, updates, actions, and agent runs.

Human Approval

Guardrails require human approval for risky physical or operational actions.

Limitations and Roadmap

Current MVP Limitations

AI models are deterministic/mock boundaries.
Event processing is in-process rather than a Redpanda worker.
Redis and Qdrant are infra-ready but not deeply integrated.
External Jira/Slack/ServiceNow integrations are action stubs.
Migrations and full tests are future work.

Next Production Steps

Add FastStream + Redpanda worker and DLQ.
Integrate YOLO, Whisper, BGE-M3, Qdrant.
Replace deterministic agents with LangGraph and model gateway.
Add OpenTelemetry, Alembic, tests, Helm, and load testing.
Add real Slack/Jira/ServiceNow/SAP integrations.