TaskPulse Analytics Platform

Business Problem

TaskPulse is built around a question every engineering manager eventually asks: which users are productive, which workspaces are falling behind, and where are tasks piling up? Answering that by querying a live operational database directly is a bad idea — it competes with production traffic and invites ad-hoc, unrepeatable analysis. TaskPulse solves it by generating realistic workspace activity (tasks, assignments, status changes) and running it through a layered pipeline — raw capture, cleaning, and analytics — so the numbers a dashboard would show come from a purpose-built, incrementally-updated store instead of a query bolted onto the live app.

Architecture

TaskPulse runs as a pnpm monorepo backed by three separate PostgreSQL 16 databases — raw, clean, and analytics — each in its own Docker container with its own Drizzle ORM schema and migration history. Because they're physically separate databases, not just separate schemas, there are no foreign keys across layers: each layer only owns what it's responsible for. A Node.js + Express service generates simulated workspace and task events and writes them into the raw database. Python workers built with pandas and psycopg2 pull from raw, validate and normalize the data, and load it into the clean database; a second set of workers aggregates clean data into analytics tables — user productivity, workspace efficiency, task completion, and project performance metrics.

Pipeline

Each pipeline stage — extractor, transformer, loader — lives in its own Python module, wired together by a pipeline entrypoint. Extraction is incremental by default: the raw-to-clean job reads a job_history table for the last successful run and only pulls rows created after that timestamp, then records its own run in the same table once it finishes. Pandas handles the transformation logic — grouping, aggregation, completion-rate math — before loading into typed, Drizzle-managed tables. The full stack — three Postgres instances, the generator, and the workers — comes up with a single docker compose up.