By the end of the last article we had something that felt remarkably polished.

• A customer sends a message.
• It appears in Slack.
• A support representative replies.
• The reply is persisted in our application.
• The customer’s browser updates almost instantly through Server-Sent Events.
• Everything works.

And that’s exactly where a lot of projects stop.

Unfortunately, production systems don’t.

One of the biggest mistakes developers make when building integrations is designing entirely around the happy path. It’s fun. Everything responds immediately. Networks behave themselves. APIs are available. Databases commit transactions. Nobody clicks the same button seventeen times because “it didn’t look like anything happened.”

Reality is considerably less cooperative. Servers restart. Databases deadlock. Networks disappear. Cloud providers have bad days. Someone accidentally deploys on a Friday afternoon.

Sometimes that’s even you.

The good news is that we’ve accidentally been preparing for this throughout the entire series. When we decided to verify webhooks before processing them... When we created an inbox table instead of processing events immediately... When we separated Slack from our own domain model...

...we weren’t just organizing code. We were building a system that can recover from failure. Making something work is an implementation problem. Keeping it working is an architectural problem.

Today we’re going to focus on the second one.

Failure Is a Feature

Imagine this sequence of events:

Customer
    │
    ▼
Slack Event
    │
    ▼
Webhook Received
    │
    ▼
Event Processor
    │
    ▼
Database Update

It looks straightforward. Now let’s replay exactly the same sequence.

Customer
    │
    ▼
Slack Event
    │
    ▼
Webhook Received
    │
    ▼
Event Processor
    │
    ✖
Database Deadlock

Nothing about your code changed. The customer still typed the same message. Slack still delivered the same event. The only difference is that one component in the middle had a bad day.

The real question isn’t whether failures happen. They will. The question is what your application does next. Does the message disappear forever? Does Slack retry? Does your processor retry? Does someone receive two copies? Does somebody have to manually fix the database? Or does your application quietly recover without anyone noticing?

That’s what separates a demo from a production system.

Success Is Easy

Developers love to test successful requests... "The Happy Path." We fill out the form correctly. We click Submit. Everything works. We nod approvingly and move on.

Failure testing requires a completely different mindset. Instead of asking, “Did the feature work?”start asking questions like:

• What happens if the database is offline?
• What happens if Slack delivers this event twice?
• What happens if my server crashes halfway through processing?
• What happens if the scheduled task stops running overnight?
• What happens if my API times out after I’ve already saved half the work?
• What happens if I deploy a new version while events are still queued?

Those questions usually reveal architectural weaknesses long before your customers do. One of my favourite sayings is this:

Every application is perfectly designed for the failures it eventually experiences.

If you’ve never considered a particular failure mode, odds are your application won’t handle it particularly gracefully.

Why Queues Change Everything

Earlier in the series we introduced the Slack Event Inbox. At the time it seemed like a convenient place to temporarily store webhook payloads. Now it becomes something much more valuable. It becomes a recovery point. Imagine if we processed every Slack event directly inside the webhook.

Slack
    │
    ▼
Webhook
    │
    ▼
Business Logic

If your server crashes halfway through processing, you’re in an awkward position. Did the database commit? Did Slack receive a response? Will Slack retry? Did you partially update the conversation?

You may not actually know. Now compare that to our architecture.

Slack
    │
    ▼
Webhook
    │
    ▼
Inbox Table
    │
    ▼
Background Processor

The webhook’s job is simply to record the event safely. Once the row exists in the inbox table, we’ve already won. Even if the server immediately catches fire, the event hasn’t been lost. (If the server literally catches fire, you’ve got larger concerns than your support chat, but let’s stay focused.)

The processor can resume later. Another server can process the queue. A scheduled task can retry it tomorrow. The event patiently waits until someone successfully processes it.

That’s a remarkably powerful property for what initially looked like a simple staging table.

Processing Should Be Idempotent

One word shows up repeatedly in event-driven architecture discussions. Idempotent.

It’s also one of those words that developers immediately pretend they’ve understood before quietly opening another browser tab. Fortunately, the concept is much simpler than the spelling. An operation is idempotent if running it twice produces the same result as running it once.

Suppose Slack delivers this event:

{
    "thread_ts": "1751032712.003900",
    "ts": "1751032894.001200",
    "text": "Can you try clearing your browser cache?"
}

Your processor saves the support reply. Moments later, Slack decides your acknowledgement took a little too long and sends the exact same event again. Without idempotency, your customer now sees this.

Support
Can you try clearing your browser cache?
Can you try clearing your browser cache?

That isn’t a Slack problem. That’s an application problem. Fortunately, we already have everything we need. Slack gives every message a unique timestamp. Earlier we chose to persist that timestamp alongside the support reply.

Now we simply ask a question before inserting anything.

if ( supportMessageService.existsBySlackTs( event.ts ) ) {
	return;
}

If we’ve already processed this Slack message, we’re done. No exception, no warning, no duplicate reply. The processor simply moves on to the next event. One tiny guard clause removes an entire category of production bugs.

That’s why idempotency is one of the most valuable habits you can develop when building integrations. Never assume an event will arrive once. Assume it will eventually arrive twice. Or three times. Or during a deployment. Your code should be equally happy either way.

Not Every Failure Should Be Retried

Once you accept that failures happen, the natural instinct is to retry everything. Don’t. Some failures are temporary. Some aren’t.

A temporary database deadlock? Retry.

A transient network timeout talking to Slack? Retry.

Your SQL Server restarting after patching? Retry.

But what about this?

var conversation = supportConversationService.getBySlackThread( event.thread_ts );

If the conversation doesn’t exist because the thread timestamp is invalid, retrying that code another fifty times isn’t suddenly going to make the conversation appear.

You’ve learned something important. This isn’t a temporary failure. It’s bad data.

Understanding the difference between transient failures and permanent failures is one of the most important responsibilities of an event processor. The trick isn’t retrying everything. The trick is retrying only the things that have a reasonable chance of succeeding next time.

Poison Messages Aren’t Going Away

Eventually you’re going to encounter an event that simply cannot be processed. Not because your database is unavailable. Not because Slack is offline. Not because of a transient network problem.

The event itself is bad.

Maybe someone manually deleted a conversation that should still exist. Maybe a bug generated an invalid thread timestamp. Maybe an unexpected Slack event slipped through your filters after Slack introduced a new message subtype.

Whatever the reason, retrying isn’t going to fix it. Let’s assume your processor looks something like this.

try {
	slackEventProcessor.process( event );
}
catch ( any e ) {
	retryLater( event );
}

If process() throws the same exception every single time, you’ve just built an infinite loop.

Attempt 1
    │
    ▼
Exception
    │
Retry
    │
    ▼
Attempt 2
    │
    ▼
Exception
    │
Retry
    │
    ▼
Attempt 3

Somewhere around Attempt 4,763 your CPU begins quietly wondering why it wasn't installed in a Pac Man machine. This is what’s commonly referred to as a poison message. It’s perfectly valid data to store. It’s perfectly valid data to investigate. It is not valid data to keep retrying forever.

Retry Fatigue

One of the simplest additions you can make to an inbox table is a retry counter. Every processing attempt increments the counter.

UPDATE slack_event_inbox
SET retry_count = retry_count + 1
WHERE slack_event_id = ?

Eventually you reach a limit. Maybe that’s three attempts, or five, or ten. The exact number doesn’t matter nearly as much as having one. Once the retry limit has been exceeded, stop pretending the next attempt is magically going to be different. Move the event somewhere else. Mark it as failed. Notify someone. Do something other than endlessly consuming resources.

Meet the Dead-Letter Queue

One of my favourite architectural patterns has an admittedly terrible name; The dead-letter queue. It sounds like a place where abandoned email goes to contemplate its life choices. In reality, it’s much simpler than that. It’s just a place where messages go after you’ve decided they’re not going to succeed automatically.

Slack Event
     │
     ▼
Inbox
     │
     ▼
Processor
     │
 ┌───┴─────────────┐
 │                 │
 ▼                 ▼
Success      Retry Needed
                   │
          Retry Count Exceeded
                   │
                   ▼
          Dead-Letter Queue

Notice what didn’t happen. The event wasn’t deleted. Deleting failed events removes your ability to investigate them later.

Instead, we’ve quarantined them. They’re no longer blocking normal processing, but they’re still available for inspection. Think of the dead-letter queue as the “Needs Adult Supervision” folder.

Your Dead-Letter Queue Doesn’t Need RabbitMQ

If you’ve spent much time reading about event-driven systems, you’ll eventually encounter Kafka, or RabbitMQ, or Azure Service Bus, or Amazon SQS. Those are all excellent tools. They’re also completely unnecessary for many ColdFusion applications. Your dead-letter queue can simply be another database table.

slack_event_inbox
        │
        ▼
slack_event_deadletter

Sometimes the boring solution is the best solution. Your application already knows how to query a database. Your administrators already know how to inspect database rows. Your backup strategy already includes your database. There’s nothing wrong with using infrastructure you already have.

If you eventually outgrow it, migrating to a dedicated message broker is much easier than introducing one before you’ve actually needed it.

Logging for Humans

Developers often confuse logging with dumping stack traces into a file. Those are different things. Imagine opening your log file at 2:13 a.m. because support just reported that customer replies aren’t appearing. Which of these helps more?

Null Pointer Exception

Or this?

Slack Event Processing Failed
Event ID: 64381
Conversation: 1829
Slack Thread: 1751032712.003900
Slack User: U094832
Attempt: 3 of 5
Error: Conversation not found.

One tells you something failed. The other tells you where to start looking. Good logs answer questions. Poor logs create new ones.

Log Context, Not Just Exceptions

Whenever possible, include enough context that someone can reproduce the problem without opening a debugger. Things like:

• Slack Event ID
• Thread Timestamp
• Conversation ID
• Slack User ID
• Retry Count
• Processing Duration
• Exception Message

Notice what isn’t on that list. The entire Slack payload. Logging every incoming payload might seem like a good idea until someone sends a ten megabyte attachment or your log files suddenly require their own SAN.

Log the information you’ll actually use. Everything else already exists in your inbox table.

Build Yourself an Operations Page

Earlier in the series we built pages for customers. Then pages for support representatives. Now it’s time to build a page for yourself.

It doesn’t need to be pretty. It doesn’t even need Bootstrap. Although, admittedly, everything looks better with Bootstrap.

Imagine opening an internal dashboard and immediately seeing this.

Slack Event Inbox
Pending: 12
Processing: 1
Failed: 2
Oldest Pending Event: 18 seconds
Oldest Failed Event: 2 hours
Retries Today: 31

That tiny page tells you more about the health of your integration than a thousand successful webhook requests. Operational visibility isn’t something you bolt onto an application after it launches. It’s part of the application. Because eventually someone is going to ask, “Is support chat broken?”

It’s much nicer being able to answer that question with data than with, “It seems fine on my machine.”

Sometimes the Queue Is the Canary

One interesting side effect of queue-based architectures is that the queue itself becomes an early warning system.

Suppose your application normally processes Slack events in under a second.

Suddenly your dashboard reports this.

Pending Events
9:30  0
9:35  1
9:40  3
9:45  17
9:50  61

The queue isn’t the problem. It’s telling you there’s a problem somewhere else. Maybe SQL Server is under heavy load, or Slack is responding slowly. Maybe somebody accidentally deployed an infinite loop. Whatever the cause, your queue is giving you an early indication that something downstream has stopped keeping up.

The nice part is that customers probably haven’t noticed yet. You’ve still got time to fix it before support starts receiving emails that begin with, “Just wondering if you got my message...”

Recovering From Outages

One of the nice side effects of designing for failure is that recovery usually becomes surprisingly straightforward. Imagine your application goes offline for twenty minutes while you deploy a new version. During that time, customers continue sending support requests. Support representatives continue replying in Slack. Slack continues delivering webhook events. Your web server isn’t there to receive them.

What happens?

Hopefully Slack retries. If it does, great. If not, you’ve still got a problem.

Now imagine a different outage. Your webhook is still running. The inbox table continues receiving events. But your scheduled task that processes those events accidentally stopped overnight. Your dashboard now looks something like this:

Pending Events
8:00  0
8:15  3
8:30  12
8:45  48
9:00  113

It isn’t pretty, but it’s recoverable. The scheduled task starts running again. It begins processing events. The queue shrinks. Eventually everything catches up. Nobody manually copies JSON payloads or asks Slack to resend anything or edits database rows by hand. Your application simply continues where it left off.

That’s one of the biggest advantages of queue-based systems. They’re naturally resumable.

Recovery Should Be Boring

When production has a problem, excitement is generally a bad sign. The best recovery procedure usually looks something like this.

Problem Found
      │
      ▼
Problem Fixed
      │
      ▼
Restart Processor
      │
      ▼
Queue Drains
      │
      ▼
Everything Returns to Normal

That’s it. No SQL updates. No emergency scripts. No copying data between tables. If your architecture requires heroics every time something goes wrong, your architecture probably needs another look.

The goal isn’t eliminating failures, it's making recovery routine.

Putting It All Together

Let’s step back and look at the complete picture.

                  Slack
                     │
                     ▼
           Verified Webhook
                     │
                     ▼
             Event Inbox Table
                     │
                     ▼
        slack_event_processor.cfc
                     │
      ┌──────────────┴──────────────┐
      │                             │
      ▼                             ▼
Transient Failure          Permanent Failure
      │                             │
      ▼                             ▼
 Retry Later               Dead-Letter Queue
      │
      ▼
Success
      │
      ▼
Persist Support Reply
      │
      ▼
Publish SSE Event
      │
      ▼
Customer Browser

The application never depends on everything working perfectly. Every stage assumes that the previous stage might occasionally fail. That’s what makes the pipeline resilient. Not because failures don’t happen, but because they were expected.

What We’ve Built

Over the last eight articles we’ve gone from a simple webhook endpoint to a production-ready messaging pipeline.

• Customers can initiate conversations without ever thinking about Slack.
• Support representatives can reply without ever opening your application.
• Messages are verified before they’re trusted.
• Events are queued before they’re processed.
• Replies become first-class domain objects.
• Browsers update in real time.
• Duplicate deliveries don’t create duplicate messages.
• Temporary failures are retried.
• Permanent failures are quarantined.
• Operations has visibility into what’s happening.

Perhaps most importantly, the system doesn’t panic when something goes wrong. It simply keeps working. Or waits patiently until it can.

The Biggest Lessons

If there’s one idea I hope sticks after this article, it’s this: Production software isn’t defined by how it behaves when everything works. It’s defined by how it behaves when everything doesn’t.

Along the way we’ve reinforced several ideas that apply well beyond Slack integrations.

• Separate receiving work from processing work.
• Make every operation safe to run more than once.
• Retry transient failures.
• Stop retrying permanent failures.
• Never delete data you haven’t investigated.
• Build operational tools before you need them.
• Design recovery procedures that are boring.

None of those ideas are particularly exciting. That’s precisely why they work. Reliable systems are usually built from lots of small, boring decisions rather than one brilliant architectural breakthrough.

Looking Back

When we started this series, our goal sounded deceptively simple. Build a support chat. Somewhere along the way we ended up discussing webhook signatures, asynchronous processing, background workers, domain modeling, Server-Sent Events, idempotency, retry strategies, dead-letter queues, and operational visibility.

That’s because support chat was never really the point. The support chat was simply the vehicle. The real goal was learning how to build integrations that behave like first-class citizens inside your application instead of a collection of HTTP requests glued together with hope. Those lessons transfer remarkably well.

Today’s Slack integration could easily become tomorrow’s Microsoft Teams integration, or Stripe webhook processor, or GitHub App, or Salesforce synchronization engine.

The technologies will change. The architectural principles won’t.

In the Final Article

Our Slack integration is now reliable. Messages survive failures. Replies recover gracefully. Operations has the tools needed to understand what’s happening.

At this point, we’ve built all of the important pieces. The problem is that they’re currently spread across eight articles, several components, a handful of database tables, and enough code fragments to make assembling the whole thing feel like an archaeological expedition.

So, for the final article in this series, we’re going to put everything together.

We’ll build a complete, self-contained Page-to-Slack support chat skeleton and publish it as a downloadable repository. It will include the customer-facing chat page, the Slack API integration, secure event verification, the event inbox, background processing, conversation mapping, support replies, Server-Sent Events, retry handling, and the database schema needed to make it all work.

The goal won’t be to create a finished support platform for every possible application. It will be to provide a clean, understandable starting point that you can download, configure with your own Slack credentials and channel, and run.

You’ll still need to adapt the authentication, styling, data model, and deployment details to fit your application. But you won’t need to reconstruct the architecture from eight separate articles while wondering which code fragment came before which database migration.

By the end, you’ll have a working beginning-to-end skeleton:

Customer Page
      │
      ▼
ColdFusion Application
      │
      ▼
Slack Support Thread
      │
      ▼
Support Reply
      │
      ▼
Customer Browser

Add your keys. Choose your channel. Run the SQL. Start the application. Then make it yours. Because after eight articles about how the pieces work, it’s time to hand you all of the pieces in one box.