Whoa Jobs

Every job offer, one dashboard.

A full-stack platform that collects work-study and full-time job postings from several job boards, deduplicates them and lets you track and apply from a single place.

GitHub·Features· Architecture

About

Looking for a work-study position or a first job means checking LinkedIn, Indeed, HelloWork and others every day, finding the same offer three times and losing track of where you applied.

Whoa Jobs fixes that with one idea:

The job boards come to you, not the other way around.

A Python ingestion engine collects postings in the background, an API normalizes and serves them, and a dashboard lets candidates browse, filter and apply — while admins steer the whole engine live, without ever restarting it.

Live ops — queue depth, active vs. target workers, requests per minute, error rate, circuit-breaker state
Hot settings — enable the engine or a single source, "Run now", tune rate limits and scaling, all applied within seconds, without a restart
Cron jobs & logs — schedules and events backed by real engine data
Audit, impersonation & broadcast — every sensitive action traced, "view as" a user, notify everyone

🧱 Architecture

 Job boards ── polite HTTP ──►  Engine (Python)
 LinkedIn · Indeed · HelloWork    ├── Orchestrator : scheduler + autoscaling
                                  └── Workers × N : fetch → parse → upsert
                                          │
                              Redis (BullMQ queues)
                                          │
                                          ▼
                        Postgres (Supabase) ◄──── NOTIFY engine_control
                                 ▲                       │
                                 │                       │
                   API — NestJS + tRPC ──────────────────┘
                                 ▲
                                 │ tRPC / SSE
                    Dashboard — TanStack Start

The project is an Nx monorepo mixing TypeScript (Bun) and Python (uv):

apps/
  dashboard/   TanStack Start front end
  server/      NestJS API
  engine/      Python ingestion engine
  fumadocs/    User documentation
packages/
  api/         tRPC routers, middlewares, business logic
  auth/        Better Auth configuration
  db/          Prisma schema & migrations
  ui/          Shared shadcn/ui components

🕷️ The ingestion engine

The most interesting part of the project. It is a horizontally scalable collector designed to be polite by construction:

Task queues on Redis (BullMQ) — one task = one search or one posting. At-least-once delivery, idempotent upserts, deduplication keys: a replayed task never creates a duplicate.
Autoscaling — the orchestrator sizes the worker pool from queue depth, as local processes, Docker containers or fixed replicas.
One shared polite client — robots.txt, a global token bucket hard-capped at 2 req/s, per-domain concurrency, a circuit breaker shared by every worker, exponential backoff with jitter, honest User-Agent.
No evasion, ever — no proxy rotation, no fingerprint spoofing, no CAPTCHA solving. A blocked source is paused and surfaced in the dashboard, never worked around.
Raw pages kept — parsers can be re-run on everything already downloaded without a single network request.
Live configuration — runtime settings validated against a single JSON spec on both sides (API and engine), reloaded through Postgres NOTIFY.

🛠️ Tech stack

Layer	Technologies
Front end	TanStack Start, React 19, Tailwind CSS 4, shadcn/ui, Motion, Recharts
API	NestJS 11, tRPC 11, Zod 4
Auth	Better Auth (organizations, teams, admin, impersonation)
Data	PostgreSQL (Supabase), Prisma, Supabase Storage
Engine	Python 3.12, BullMQ, Redis, httpx, SQLAlchemy, BeautifulSoup, Typer
Tooling	Nx, Bun, uv, Varlock, Oxlint, Oxfmt, pytest, mypy, Docker

💡 What I learned

Designing a distributed system that stays polite. Scaling workers is easy; making sure ten workers still send fewer requests than one careful human is the real challenge — hence rate limits and a circuit breaker shared through Redis rather than per process.
Making a Python service and a TypeScript API share one source of truth. The database schema is owned by Prisma, the engine only reflects it; runtime settings are validated against the same JSON spec on both sides.
Changing a running system safely. Every setting change is validated, audited and applied live through LISTEN/NOTIFY, with a periodic fallback when a notification is lost.
Treating compliance as a feature. Respecting each site's terms shaped the architecture as much as performance did.