Webhooks: The Good, the Bad, and the Ugly

Why Trust Techopedia

Webhooks are a pillar of modern application development. They notify us of that new commit, an incoming text message, an email delivery, and a processed payment. Our systems can’t function without webhooks sending data seamlessly and securely across the internet. But what happens if they’re not secure? What happens if your webhooks are intercepted, manipulated, or even replayed against your systems? What are the best ways for a provider to protect its systems and make sure its webhooks are secure? We’ll delve into some of the best and worst practices we found as we explored the 100+ implementations we included in webhooks.fyi to extract principles to protect our systems now and in the future.

Webhooks 101

At its most basic level, a webhook is a message that one system sends another system to let it know something in the first system has changed. That receiving or listening system processes the webhook and responds. Neither the sender nor the receiver has to have a common programming language, technology stack, or even hosting environment. The underlying concept was put forth by Jeff Lindsay who coined the term and popularized it starting in 2007. Lindsay linked it back to the early object-oriented principle of “message passing.” In message passing, objects communicate changes in their internal state while other objects in the system can listen and take action as necessary. The underlying concept still applies, but we’ve moved from an object-oriented in-memory system to a global network.  

Moving back to our modern implementation, the webhook itself is usually a JSON payload over HTTPS. The listening app receives it, processes it, and gives a response. Within the listener, it may log the event, initiate a workflow, or even do nothing at all. We don’t care, and the webhook mechanism doesn’t care either. All we care about is that the webhook service published a change in state, and the listening app picked up that change and sent a response. 

Why Developers Love Webhooks:

  • It all happens over the internet using HTTP.
  • They use a simple payload like JSON or XML.
  • They work with any tech stack so developers don’t have to rely on vendors. 
  • They allow us to share changes in state between systems.
  • They’re easy to test and mock (e.g., we can create things that look like it to test).

The Fatal Flaw

The fatal flaw with webhooks is that we have three independent components: the generating or broadcasting system, the listening app, and the communications channel between them. When considering how we’re securing webhooks, we have to think about all three components. If you’re building your own first-party apps, you have control over all three. Unfortunately, the vast majority of us only have control over two of the three. The third component is opaque from our perspective. 

Therefore, when we think about how we’re building webhooks, we need to take a defensive perspective. If you’re building the listening app, you need to protect yourself from attackers in the communications channel intercepting, adapting, or even replaying webhooks. If you’re the broadcasting system, you need to protect 

Risks of Using Webhooks

If you only have control over the generating service or the listening app, you can’t inherently trust the other app or the channel. You must proactively consider what risks exist and what you can attack them with. The risks include:

  • Intercepting or snooping on payloads
  • Impersonating the broadcasting app
  • Modifying or manipulating the payloads
  • Replaying requests — if the first request was legitimate, the receiver will assume that the same request is too
  • Missing requests
  • Supporting forward compatibility as security requirements change

Best Practices for Webhook Providers

Webhooks are a cornerstone of modern product development. If you’re providing webhooks, it’s your responsibility to make sure that you’re making them as secure as possible. Here are a few best practices.

  1. Use HTTPS— there’s no excuse not to. Organizations like Let’s Encrypt make certificate management a fast, easy process. Use it. 
  2. Provide documentation and examples to demonstrate the secure, proper approach to using your webhooks. Providers should show the end-to-end webhook flow, including all of the parameters, URLs, and options a user might see. If you want bonus points, show all of the payloads so that the user knows what to expect. And finally, demonstrate verification with sample code and embed that verification in your libraries.
  3. Sign your payload to protect it from being modified. Whether you use a shared secret, HMAC, ESA/ECDSA, JWT/JWK/OAuth, or mTLS, any of them are reasonable approaches of varying complexity, but HMAC has become the de facto standard.
    webhook authentication methods
  4. Mitigate reply attacks by using timestamps or request ID in the signature verification. By including the timestamp, listening systems can choose how long to accept a given request and reject old requests. Alternatively, a request ID can be used and tracked to ensure each is unique, rejecting requests it has seen before. Whichever approach you choose, ensure that it is included in the signature so it can’t be modified either.
  5. Use multi-version support. When you have a version change, create a new signature that clearly communicates it and includes it in the payload. If you have multiple signatures in the payload, the end user will understand that they can use any of those versions and you can transparently upgrade without breaking their systems.
  6. Rotate keys and include them in your signature for zero downtime.

Improving Webhook Security by Verifying Signatures

While webhooks give us power and flexibility, they also require a live connection to the public internet. Without good security controls, third parties could explore webhooks for malicious activities such as sending fake events to taint your systems, obtaining confidential information, corrupting your data, or spam users. Luckily, many webhook providers deliver capabilities to verify the sender, validate the data sent, and even mitigate more specific attacks.

At ngrok, we’ve developed a module to verify webhook signatures in the connection long before the traffic reaches your app. Not only does this save time from modifying your app, but you can also protect those legacy apps you’re afraid to touch. After exploring over 100 webhook providers and building support for more than 70, we collected the most common, interesting, and challenging patterns. Check out the findings from the research on the ngrok blog.


Related Reading

Related Terms

Keith Casey
Director of Product Marketing at ngrok
Keith Casey
Director of Product Marketing at ngrok

Keith Casey currently works in Product and Go To Market at ngrok, helping teams build and launch their systems. Previously, he served on the Product Team at Okta working on Identity and Authentication APIs for apps and APIs, he worked as an early Developer Evangelist at Twilio, and he worked on the Ultimate Geek Question at the Library of Congress. His underlying goal is to get good technology into the hands of good people to do great things. In his spare time, he helps build and support the Austin tech community and blogs at CaseySoftware.com. He is also a co-author…