What is NATS?
NATS is an open-source, message bus technology that powers modern distributed systems. Accordingly, it is responsible for addressing, discovering, and exchanging messages that drive the common patterns in distributed systems; asking and answering questions, aka services/microservices, and making and processing statements, or stream processing.
What makes NATS so unique for distributed, modern systems?
Effortless M:N connectivity: Firstly, NATS manages addressing and discovery based on subjects and not hostname and ports. Defaulting to M:N communications, which is a superset of 1:1, meaning it can do 1:1 but can also do so much more. Thus, if you have a 1:1 system that is successful in development, ask how many other moving parts are required for production to work around the assumption of 1:1? Things like load balancers, log systems, and network security models, as well as proxies and sidecars. If your production system requires all of these things just to get around the fact that the connective technology being used, e.g. HTTP or gRPC, is 1:1, it’s time to give NATS.io a look.
Deploy anywhere: Due to its versatility, NATS can be deployed nearly anywhere; on bare metal, in a VM, as a container, inside K8S, on a device, or in whichever environment you choose. Likewise, NATS runs well within deployment frameworks or without.
Secure: Similarly, NATS is secure by default and makes no requirements on network perimeter security models. When you start considering mobilizing your backend microservices and stream processors, many times the biggest roadblock is security.
- Pub/Sub (Close paradigm to RabbitMQ)
In brief, NATS implements a publish-subscribe message distribution model for one-to-many communication. Thus, a publisher sends a message on a subject and any active subscriber listening on that subject receives the message. Additionally, subscribers can also register interest in wildcard subjects that work a bit like a regular expression (but only a bit). Therefore this one-to-many pattern is sometimes called a fan-out.
Request-Reply is a common pattern in modern distributed systems. For example, a request is sent, and the application either waits on the response with a certain timeout or receives a response asynchronously.
- Streaming using Jetstream
Firstly, NATS has a built-in distributed persistence system called JetStream. Specifically, this system enables new functionalities and higher qualities of service on top of the base 'Core NATS' functionalities and qualities of service.
JetStream is built-in to nats-server and therefore you only need 1 (or 3 or 5 if you want fault-tolerance against 1 or 2 simultaneous NATS server failures) of your NATS server(s) to be JetStream enabled for it to be available to all the client applications.
Initially created in 2011, has more than 200M downloads, over 1K contributors, and 45+ client libraries. Therefore, NATS is trusted by the largest, most mission-critical organizations in the world like Mastercard, Form3, IBM, SUSE, and many more large enterprises that run NATS in production for years.
NATS VS Kafka
|Delivery Guarantees||At most once, at least once, and exactly once is available in JetStream.||At most once, at least once, and exactly once.|
|Multi-tenancy and Sharing||NATS supports true multi-tenancy and decentralized security through accounts and defining shared streams and services.||Multi-tenancy is not supported.|
|Authentication||NATS supports TLS, NATS credentials, NKEYS (NATS ED25519 keys), username and password, or simple token.||Supports Kerberos and TLS. Supports JAAS and an out-of-box authorizer implementation that uses ZooKeeper to store connection and subject.|
|Authorization||Account limits including number of connections, message size, number of imports and exports. As well as user-level publish and subscribe permissions, connection restrictions, CIDR address restrictions, and time of day restrictions.||Supports JAAS, ACLs for a rich set of Kafka resources including topics, clusters, groups, and others.|
|Message Retention and Persistence||Supports memory, file, and database persistence. Messages can be replayed by time, count, or sequence number, and durable subscriptions are supported. Thus, with NATS streaming, scripts can archive old log segments to cold storage.||Supports file-based persistence. Thus, messages can be replayed by specifying an offset, and durable subscriptions are supported. Log compaction is supported as well as KSQL.|
|High Availability and Fault Tolerance||Core NATS supports full mesh clustering with self-healing features to provide high availability to clients. Hence, NATS streaming has warm failover backup servers with two modes (FT and full clustering). JetStream supports horizontal scalability with built-in mirroring.||Fully replicated cluster members are coordinated via Zookeeper, now also KRaft.|
|Deployment||The NATS network element (server) is a small static binary that can be deployed anywhere from large instances in the cloud to resource constrained devices like a Raspberry PI. Thus, NATS supports the Adaptive Edge architecture which allows for large, flexible deployments. Therefore, single servers, leaf nodes, clusters, and superclusters (cluster of clusters) can be combined in any fashion for an extremely flexible deployment amenable to cloud, on-premise, edge and IoT. Moreover, clients are unaware of topology and can connect to any NATS server in a deployment.||Kafka supports clustering with mirroring to loosely coupled remote clusters. Therefore clients are tied to partitions defined within clusters. Furthermore, Kafka servers require a JVM, eight cores, 64 GB to128 GB of RAM, two or more 8-TB SAS/SSD disks, and a 10-Gig NIC. (1)__|
Memphis.dev forked and refactored NATS.IO from the ground up to create a fully-fledged event-processing platform for developers.
Furthermore, it enables developers and data engineers to achieve all other message brokers' benefits in a fraction of the time. In other words, it focuses on removing frictions, automatic optimization, inline schema management, inline processing, ready-to-use connectors, and observability & troubleshooting abilities. As a result, all under the same hood, based on the reliability and core of NATS.
How Memphis.dev differs from NATS.IO
On the whole, Memphis focuses on four main pillars - Performance, Scalability, Dev Experience, Troubleshooting
- Firstly, increase performance and scalability.
Due to replacing naive algorithms with heuristics-based with higher success rates, and RAFT improvements to better suit Kubernetes runtimes.
- Then, remove noise and complex business logic from the developer/consumers side -
- Easy-to-use, understand, and onboard management tools like UI, and CLI that provide end-to-end control over every aspect of the cluster for both the maintainers like DevOps and SRE teams, all the way to the developers and data engineers who develop around it.
- Lightweight SDKs with background configuration that lower the amount of code and design decisions to the minimum. You still have full control over each parameter, but the out-of-the-box configuration with automatic optimizations does the heavy lifting for you.
- Finally, troubleshooting - lags or unacknowledged messages are hard to trace and mitigate! Usually, it requires building a dedicated wrapper, establishing a monitoring stack, crossing information between different consumers/databases/log sinks, and if the issue turned out to live at the application level, consequently restoring the message for validating the event is a pain.
We are highly investing in equipping our users with out-of-the-box observability and debug tools so the entire cycle of locating the root cause of a problem to fixing it will close in minutes including recovery mode that will make sure no message ever gets lost during a consumer failure.
Complete Comparison Table
|Real-time message tracing||True||False|
|Automatic runtime optimization||True||False|
|Dedup||True. Modified bloom filter||False|
|Message retransmit||Per specific consumer||False|
|Webhook triggers||Great for kubeapi, and backend endpoints||False|
|Consumer internal communication||Experimental||False|