After about 5 months into my first job as a software engineer, I was tasked with building a retry mechanism for sending failed financial data notifications.
The Problem
Our system extracts financial data and notifies external clients that their data is ready to be fetched via an API. This notification is sent as an HTTP request to the client's server, and once they receive it, they can pull the data from our system.
Sometimes, the client's server is down.
When that happens, we receive responses like 500, 503, or 504. This means our request reached their server, but the notification was not accepted or processed. The requirement was to build a retry mechanism so the client eventually gets notified.
Going Straight Into Designing
The ticket already mentioned exponential backoff, so my first instinct was to implement retries directly in code. The idea was to retry after a few seconds, then wait a bit longer, and keep increasing the delay after each failure.
Something like 3s → 5s → 10s → 30s → 1 min → 2 min.
To make this work, I thought I could simply use an in-memory counter variable to track when the next notification should be sent.
System components so far: just one variable.
Then my team lead asked a simple question.
"What if the pod restarts?"
If the pod restarts, the retry counter is lost, and we no longer know when or how many times we retried.
Making the State Persistent
The retry state needed to be persistent. Me and my team lead decided that we should store the retry count and the next retry timestamp in the database.
While discussing this, we noticed another gap. We weren't even storing failed notifications earlier. To support retries properly, we now had to store failed notifications as well.
To process these stored retries, I planned to add a cron job that would continuously scan the database and send the next eligible notification back to the application.
System components now: code changes, database table changes, and a new cron job.
"Is a cron job continuously scanning the database really a good idea?"
Kafka
Instead of scanning the database repeatedly, we could push failed notifications to Kafka and let the application consume and retry them.
System components now: code changes and Kafka setup.
But Kafka is not meant to be used as long-term persistent storage. Messages can be deleted after a retention period, such as 7 days.
So I brought the database back again to store failed notifications safely.
System components now: code changes, Kafka setup, and database table changes.
When One Client Becomes a Bottleneck
Then another scenario hit me.
What if one client becomes a hotspot? What if their server is down for days instead of hours?
In that case, retries for that single client could keep piling up. Meanwhile, other clients—whose servers might recover within minutes or hours—could get blocked behind this backlog.
This introduced two new questions. First, after how many attempts should we stop retrying? Second, how do we make sure smaller clients are not affected when one big client keeps failing?
Adding Client-Wise Queues
To handle this, I designed an in-memory, client-wise queue inside the application. Each client would have its own queue with a fixed capacity, for example, 10 entries.
The logic was simple: consume Kafka events only when the client's queue is not full.
This way, hotspot clients would naturally slow down, while other clients would still be able to receive notifications without getting blocked.
System components now: code changes, in-memory client-wise queues, Kafka setup, and database table changes.
The overall architecture seemed capable of handling most worst-case scenarios I could think of.
When My Design Met Reality
Now it was time to present the design to the CTO and get approval before implementation.
During the meeting, I explained my entire thought process. I talked through every edge case I had considered and showed the complete system using flow and block diagrams. Kafka, database, queues, limits, and future scenarios—everything was covered.
CTO then asked me two questions: how many clients have complained so far, and how frequently does this happen?
I said there are clients facing this issue. He asked "what's the exact number?"
I had no answers.
Those flow diagrams, block diagrams, and code snippets on the presentation screen seemed like wasted effort.
What Was Missing: Data
Before designing anything, he told me to go and look at the data—specifically, OpenSearch logs, as far back as we could.
He asked me to extract four metrics by answering these questions:
- Which clients have actually failed in the past year?
- How often did each client fail?
- How long did their servers usually take to recover?
- Which HTTP status codes were most common per client?
Those metrics would be:
- number_of_failed_clients
- frequency_of_failure_per_client
- avg_time_to_recover_per_client
- http_status_distribution_per_client
Only after having this data, he said, could we meaningfully decide the architecture. Without data, debating designs was pointless.
The Realization
A few days later, I got assigned another task and this task was moved to the backlog.
I think in most cases, to over-engineer something or to under-engineer something would be a result of lack of data. Looking back, I think I was just afraid of being wrong and that's what made me go deep into all those worst-case scenarios which I prepared for.
Having started with the data, following where it leads, and I would've built exactly what was needed.