Introducing the Event Gateway

At Hookdeck, we recently started thinking differently about our platform and the services it provides. Hookdeck began as a webhook management solution. Engineering teams dealing with high-volume webhook consumption repeatedly built similar infrastructures. Typically, they included a queuing system, observability, filtering, routing, and possibly payload transformation. Hookdeck addressed this issue by developing a platform that offered these features and more.

Looking at the infrastructure landscape, we could see technologies frequently being chained together to offer this similar functionality, such as Amazon EventBridge (Event Bus and Pipes) combined with Amazon API Gateway and AWS Lambda, or newer entrants such as QStash by Upstash.

When we began examining the broader context, including the competitive landscape, we also realized that Hookdeck's potential extended beyond just webhooks. Webhooks are a subset of asynchronous events. These events have increasingly become the preferred programming paradigm for signaling changes in a system, whether within a code base, between micro-services on the same platform, or across different platforms.

We’re calling this new product category an Event Gateway.

The Evolution of Asynchronous Communication

Over the past two decades, events have taken a central place in software development. They effectively convey changes and facilitate action. With the rise of system decoupling and the micro-services trend, it's logical that events have come to drive many facets of software architecture.

In recent years, SaaS platforms have become increasingly specialized. While they once excelled as generalists, SaaS platforms now concentrate on their strengths, delegating tasks outside their core competency to other specialized SaaS platforms. Consider the ever-expanding range of fulfillment SaaS integrations with Shopify or the numerous integrations available on Github and Slack. Slack exemplifies hyper-specialization in the communication domain. As a founder, if you're focusing on, say, development tools, would you create something akin to Slack, Mailgun, or Stripe for notifications? Or would you integrate with them and emphasize what adds value for your users?

This movement towards vertical hyper-specialization highlighted the need for efficient inter-platform communication. API polling proved slow and wasteful, leading to the advent — or at least a significant increase in the usage — of webhooks.

Given the simplicity of webhooks — basically, an HTTP request — it's unsurprising that they evolved in various directions without a standard format. Providers often differ in timeout settings, retry rules, formats, and even methods. One of Hookdeck's initial goals was to bring coherence to this space by centralizing and standardizing providers, ensuring consistent delivery to customers irrespective of the webhook’s source.

Introducing the Event Gateway

An Event Gateway is a platform that is deployed between external and internal services that supports the receiving, processing, and delivery of events between those services. An Event Gateway should be dependable — guaranteeing that events don't get lost — and, once events are securely ingested, are processed and accurately delivered at scale.

Use Cases for an Event Gateway

Event gateways are extremely versatile as one can use them for a variety of use cases. The following list covers the most common that we've seen.

• Inbound webhook infrastructure: While receiving a few webhooks is simple enough, ingesting them at scale is not. One needs to build, at minimum, a queuing infrastructure and retry logic to provide guarantees and controls around critical inbound webhooks. Most users initially used Hookdeck for this use case.
• Outbound webhook infrastructure: An Event Gateway manages outbound webhooks, ensuring scalability, retry logic, and visibility. Compared with the inbound webhooks use case, the focus is to ensure that the gateway manages connectivity, ingestion, etc. This ultimately offsets the burden of webhooks management away from critical workloads. Companies that want to offer webhook functionality to their customers use an Event Gateway in this way.
• Async APIs: An async API isn't expected to provide a response for normal operations (beyond a 200 or an error code). They are mainly used for ingestion. A good example is a system needing to send measurement data to another monitoring system. The advantage of an Event Gateway is that it abstracts the sender from the receiver, which protects both systems (in case the destination becomes unavailable, for instance). We increasingly observe asynchronous API endpoints used as destinations. This is also another good use case for an Event Gateway by companies wishing to offer their customers an Async API.
• 3rd party to 3rd party message routing: An event gateway aids in integrating distinct systems, transforming payloads, and monitoring traffic. For instance, if one needs to integrate a Continuous Integration platform with an unsupported instant messaging application, an Event Gateway avoids having to host a service only for that purpose.
• Messaging broker for serverless applications: If you’re building an event-driven serverless architecture, you are bought into pay-as-you-go managed services. An Event Gateway sits at the core of your serverless architecture as a central hub for all asynchronous communication.

Features of an Event Gateway

Understanding specific features is crucial as we delve deeper into asynchronous communication. Each of these features offers a distinct advantage, addressing particular challenges in event-driven architectures:

• Fanning out: Distribute data from a source to multiple destinations efficiently.
• Fanning-in: Consolidate data from various sources into one target
• Filtering: Ensure only relevant events are processed, conserving computational resources and promoting data relevance.
• Transformation: Ensure interoperability between different systems by altering event structures or formats.
• Routing: Determining the destination of the event based on a logical definition.
• Rate limiting: Prevent system overloads from sudden surges in event traffic and conditionally control processing speed.
• Observability: Obtain insights into traffic trends and receive proactive alerts about anomalies.
• Error recovery: Replay events, whether you're the sender or the recipient, to ensure no events are lost.
• Verification: Process only trustworthy events to maintain data integrity.
• Handshake negotiation: Ensure secure communication channels with services using established standards or via custom connectors with 3rd party providers. See our supported providers' list for some examples.

Event Gateway Benefits

• Reduce Event-Driven Architecture (EDA) complexity: An Event Gateway centralizes event management and asynchronous communications between internal and 3rd party services.
• End-to-end event observability: Full visibility of events from ingestion through filtering, transformation, retries, and delivery.
• Reliability & scalability: Handle traffic spikes, avoid accidental and intentional DoS (Denial of Service) or DDoS (Distributed Denial of Service) attacks, and guarantee event delivery by having an Event Gateway ingest, verify, queue, and distribute events to your internal services.
• Error recovery: Event Gateways can pause event delivery and configure every delivery retry logic during service maintenance or unexpected downtime, meaning you never miss an event.
• Simplified security: In-built support for generic and vendor-specific event handshaking and verification.

How does an Event Gateway compare to an API gateway?

API Gateways emerged to tackle problems akin to those faced today. They gained traction with the surge of APIs when centralization became a necessity. API gateways tackled multiple challenges like authentication, metering, transformation, fanning out, and rate limiting.

An event gateway addresses analogous challenges but for asynchronous communication. It offers a unified interface for managing event flows in and out of a system, centralized logging, payload transformation, origin verification, and more.

While both API and event gateways aim to resolve similar problems, their mechanisms differ.

The core distinction is the asynchronous, durable nature of communication in Event Gateways. Their primary role is to safeguard events, ensuring they aren't lost, for example, due to downtime on the recipient's end.

Event gateways can manage messages beyond just webhooks. Imagine Hookdeck sources like an Amazon SNS topic or a GCP PubSub subscription. Or the transport being a WebSocket.

They also introduce features like message filtering, payload transformation, and enrichment. For instance, an event gateway could easily split an event stream based on a message's topic and then distribute the segmented stream to systems with specialized tasks.

Conclusion

We're thrilled about the potential of Event Gateways! While event-based systems have existed for a while, we have seen the signals of a trend emphasizing standardization, organization, and monitoring. As this specialized domain expands and the product category becomes established, a robust Event Gateway's role becomes vital and unlocks vast potential.

We believe Hookdeck stands at the vanguard of this evolution. Our platform prioritizes reliability, flexibility, and scalability. Our mission isn't merely to adapt to trends but to spearhead the journey toward a harmonized future of systems. Join us on our journey toward becoming the world’s first fully realized Event Gateway!