Task scheduling is essential in modern applications to maximize resource utilization and user experience (Non-blocking task fulfillment).
A queue is a powerful tool that allows your application to manage and prioritize tasks in a structured, persistent, and scalable way.
While there are multiple possible solutions, working with a queue (which is also the perfect data structure for that type of work), can ensure that tasks are completed in their creation order without the risk of forgetting, overlooking, or double-processing critical tasks.
A very interesting story on the need and evolvement as the scale grows can be found in one of DigitalOcean’s co-founder’s blog post From 15,000 database connections to under 100.
Multiple. Each with its own advantages and disadvantages.
Cron
You can use cron job schedulers to automate such tasks. The issue with cron is that the job and its execution time have to be written explicitly and before the actual execution, making your architecture highly static and not event-driven. Mainly suitable for a well-defined and known set of tasks that either way have to take place, not by a user action.
Database
A database can be a good and simple choice for a task storing place, and actually used for that in the early days of a product MVP,
but there are multiple issues with that approach, for example:
Often, for task scheduling, the chosen queue would probably be a pub/sub system like RabbitMQ.
Choosing RabbitMQ over a classic broker such as Kafka, for example, in the context of task scheduling does make sense as a more suitable tool for that type of task given the natural behavior of Kafka to retain records (or tasks) till a specific point in time, no matter if acknowledged or not.
The downside in choosing RabbitMQ would be the lack of scale, robustness, and performance, which in time become increasingly needed.
With that idea in mind, Memphis is a broker that presents scale, robustness, and high throughput alongside a type of retention that fully enables task scheduling over a message broker.
On v1.2, Memphis released its support for ACK-based retention through Memphis Cloud. Read more here.
Messages will be removed from a station only when acknowledged by all the connected consumer groups. For example:
We mentioned earlier the advantages and disadvantages of using traditional queues such as RabbitMQ in comparison to common brokers such as Kafka in the context of task scheduling. When comparing both tools to Memphis, it’s all about getting the best from both worlds.
A few of Memphis.dev advantages –
const { memphis } = require("memphis-dev");
(async function () {
let memphisConnection;
try {
memphisConnection = await memphis.connect({
host: "MEMPHIS_BROKER_HOSTNAME",
username: "CLIENT_TYPE_USERNAME",
password: "PASSWORD",
accountId: ACCOUNT_ID
});
const station = await memphis.station({
name: 'tasks',
retentionType: memphis.retentionTypes.ACK_BASED,
})
const producer = await memphisConnection.producer({
stationName: "tasks",
producerName: "producer-1",
});
const headers = memphis.headers();
headers.add("Some_KEY", "Some_VALUE");
await producer.produce({
message: {taskID: 123, task: "deploy a new instance"}, // you can also send JS object - {}
headers: headers,
});
memphisConnection.close();
} catch (ex) {
console.log(ex);
if (memphisConnection) memphisConnection.close();
}
})();
const { memphis } = require("memphis-dev");
(async function () {
let memphisConnection;
try {
memphisConnection = await memphis.connect({
host: "MEMPHIS_BROKER_HOSTNAME",
username: "APPLICATION_TYPE_USERNAME",
password: "PASSWORD",
accountId: ACCOUNT_ID
});
const station = await memphis.station({
name: 'tasks',
retentionType: memphis.retentionTypes.ACK_BASED,
})
const consumer = await memphisConnection.consumer({
stationName: "tasks",
consumerName: "worker1",
consumerGroup: "cg_workers",
});
consumer.setContext({ key: "value" });
consumer.on("message", (message, context) => {
console.log(message.getData().toString());
message.ack();
const headers = message.getHeaders();
});
consumer.on("error", (error) => {});
} catch (ex) {
console.log(ex);
if (memphisConnection) memphisConnection.close();
}
})();