Open-Source Message Queueing using C# and MongoDB
Published: Monday, 27 March 2023
📭 Learn how we built a simple and reliable message queueing system in C# for All Quiet, using MongoDB as the database. It's open-source and available as a NuGet package.
Hello, I'm Mads Quist, founder of All Quiet, and I'm excited to share some technical insights with you in this blog post. We've open sourced the message queueing part of All Quiet and I'm here to tell you why we did it and how it fits into our tech stack.
Why do we need message queueing?📨📨📨
In our experience, message queueing is a crucial part of implementing a reactive user experience while providing resilience and fault tolerance for your application. At All Quiet, we needed message queueing for several use cases, including:
- Sending a double-opt-in email asynchronously after a user registers
- Sending a reminder email 24 hours after registration
- Sending push notifications with Firebase Cloud Messaging (FCM), which can fail due to network or load problems. Since push notifications are crucial to our app, we need to retry sending them if there's a problem.
- Accepting emails from outside our integration and processing them into incidents. This process can fail, so we wanted to decouple it and process each email payload on a queue.
Our tech stack
To understand our specific requirements, it's important to have some information about our tech stack:
- We run a monolithic web application based on .NET Core 6.
- The .NET Core application is running in a Docker container.
- We run multiple containers in parallel.
- An HAProxy instance distributes HTTP requests equally to each container, creating a high available setup.
- We use MongoDB as our underlying database.
- All of the above components are hosted by AWS on generic EC2 VMs.
Why did we build our own queueing system?
We wanted a simple queueing mechanism that could run in multiple processes simultaneously while guaranteeing that each message was only processed once. We didn't need high performance or real-time message processing, and we didn't need the ability for a pub/sub pattern.
We didn't want to create a fancy distributed system based on CQRS / event sourcing, because you know, the first rule of distributed systems is to not distribute.
We wanted to keep things as simple as possible, following the philosophy of choosing "boring technology".
RabbitMQ or Kafka would have added too much complexity to our tech stack, and relying on cloud solutions like AWS SQS, Google Cloud Tasks, or Azure Queue Storage would have created vendor lock-in. We still aim to avoid cloud-native solutions as much as possible to reduce dependencies and avoid vendor lock-in, even though we are running our system in a cloud environment.
So, we decided to build a small message queueing system for .NET ourselves. Since the code is generic and business agnostic, we published it as an open-source project. You can learn how to use the package AllQuiet.MongoQueueing on our GitHub repository and download it from NuGet. We hope someone might find the package useful.
Wednesday, 05 April 2023
Integrate New Relic Alerts into All Quiet
🚀 In under 5 min: Discover how to connect New Relic Alerts into All Quiet with a step-by-step instruction for a seamless integration.
Thursday, 23 March 2023
Integrate UptimeRobot notifications
💡 Discover how to effortlessly integrate your UptimeRobot monitors into All Quiet through a simple webhook integration. Optimize your monitoring experience with ease.
Read all blog posts and learn about what's happening at All Quiet.
© 2023 All Quiet GmbH. All rights reserved.