Why We Unified Our Onboarding Pipeline — And Deleted 700 Lines of Code

The Silent Concurrency Problem

Our onboarding flow is the most resource-intensive operation in Jiwa AI. It scrapes websites, calls AI models for brand analysis, generates image prompts, renders visuals, and saves everything to the database. A single onboarding takes three to five minutes of continuous processing.

For WhatsApp users, we had already solved this. The onboarding ran asynchronously in isolated Cloud Run Job executions, each getting their own container with dedicated CPU and memory. But for web dashboard users, the same pipeline ran inline — inside the HTTP request handler, competing for resources with every other request hitting the service.

Two Users Onboarding at Once? Good Luck.

The problem was invisible with low traffic. One user onboarding at a time worked fine. But the moment two users hit the onboard button simultaneously, both pipelines competed for the same container's CPU and memory. Image generation slowed to a crawl. AI calls queued up. Timeouts started appearing.

Worse, the web onboarding route had grown into a 765-line monolith that duplicated the entire pipeline logic already implemented in our shared onboarding engine. Every bug fix required changes in two places. Every new feature meant double the work.

One Pipeline, Two Channels

The fix was architecturally simple: make the web flow work exactly like the WhatsApp flow. Create a job record, trigger a Cloud Run Job execution, and return immediately. The dashboard polls for progress instead of holding an HTTP connection open for seven minutes.

We extended our job model to carry web-specific inputs — the source URL, uploaded images, user context, and authentication details. The job runner now inspects which channel initiated the request and adjusts its behavior accordingly. WhatsApp jobs send progress messages via chat. Web jobs simply update their wave counter in the database, which the dashboard picks up every three seconds.

The result: our web onboarding route went from 765 lines to about 80. All the scraping, analysis, generation, and delivery logic lives in one place. Both channels exercise the same code path, the same error handling, and the same concurrency controls.

Real Progress, Not Fake Timers

The old dashboard showed a progress bar driven by hardcoded timers — five seconds for "scraping," ten seconds for "analyzing," and so on. The steps advanced whether or not the server had actually reached that stage. Users watching a slow onboarding would see the progress bar complete long before their content was ready.

Now the progress bar reflects reality. The onboarding engine reports which processing wave it has completed, and the dashboard maps those waves to meaningful progress steps. When the AI is analyzing your brand, the progress bar says so because the server actually reported it.

Graceful Error Handling Across Channels

We also took the opportunity to improve error specificity. When Instagram returns a "profile not found" error — almost always because the user has a personal account rather than a Business or Creator account — both channels now explain exactly what went wrong and how to fix it. WhatsApp users receive step-by-step instructions in their preferred language. Web users see a dedicated error screen with the same guidance.

Previously, both channels showed a generic "something went wrong" message that left users guessing.

The Impact

Concurrent onboardings no longer interfere with each other. Each gets its own isolated container, its own CPU allocation, and its own memory budget. The web dashboard responds instantly instead of holding a connection open for minutes. And our engineering velocity doubled on the onboarding flow because there is now exactly one pipeline to maintain.

Sometimes the best feature you can ship is deleting seven hundred lines of code.