Congruence
AI pipeline & backend · clinical documentation
Multimodal AI that writes psychotherapy and insurance notes, and catches the emotional cues clinicians miss.
- Role
- AI pipeline & backend
- Timeline
- 2024 – Present
- Stack
- FastAPI, React Native, OpenCV, Wav2Vec2, GPT-4, PostgreSQL, Redis, Kubernetes
92%
less documentation time
4+
clinics
200+
patients
<200ms
inference latency
Overview
Psychiatrists spend 4+ hours a day on notes, which interrupts therapy and misses non-verbal cues. Existing EHR tools add admin work without clinical insight.
Congruence runs during the session, fusing facial microexpressions, voice stress, and the transcript to draft SOAP notes and insurance packets. Documentation went from 45 minutes to 3.5.
What I built
- Built a multimodal CNN + Wav2Vec2 + LLM pipeline reaching 76% clinical emotion accuracy.
- Designed triple-embedding incongruence detection with 89% sensitivity on masked emotion.
- Got real-time inference under 200ms with GPU video, CPU audio, and Redis fusion.
- Deployed HIPAA-compliant on Kubernetes at 99.8% uptime.
Architecture
- Client
- React Native
- API
- FastAPI
- Vision
- OpenCV + CNN (60 FPS)
- Audio
- Wav2Vec2 (16kHz)
- LLM
- GPT-4 note generation
- Data
- PostgreSQL + Redis
- Infra
- Docker + Kubernetes
Decisions
Fuse modalities in real time
Video runs on GPU, audio on CPU, and Redis fuses the streams so notes are ready when the session ends, with p99 under 200ms.
Detect incongruence, not just emotion
Comparing face, voice, and words separately surfaces masked affect that a single-modality model misses.
Compliance from day one
HIPAA-compliant infrastructure, an IRB-approved validation study, and SOC 2 Type II so clinics could adopt without a security review stall.
Screens
Next project
Hanger · fashion discovery