2019 update
Before thinking about writing a package for anything, first look if there are existing packages that do what you need. In the Meteor world, this means looking on Atmosphere for “job/queue/task/worker management/scheduling” packages, then on npm for the same search terms. You also need to define your requirements more precisely:
- do you want persistence, or would an in-memory solution work?
- do you want to be able to distribute jobs to different machines?
Meteor-specific
- job-collection – reliable (I used it in 2014 in production at a startup), but currently in maintenance mode. Lets you schedule persistent jobs to be run anywhere (servers, clients).
- SteveJobs – actively maintained by Max Savin, the author of several powerful Meteor tools
- littledata:synced-cron – “A simple cron system for Meteor. It supports syncronizing jobs between multiple processes.”
Abandoned packages:
- artwells:queue – priorities, scheduling, logging, re-queuing. Queue backed by MongoDB. Last code commit: 2015-Oct.
- super basic cron packages: easycron. Last update: Dec 2015.
- differential:workers – Spawn headless worker meteor processes to work on async jobs. Last code commit: Jan 2015
- cron (since 2015)
- PowerQueue – abandoned since 2014. Queue async tasks, throttle resource usage, retry failed. Supports sub queues. No scheduling. No tests, but nifty demo. Not suitable for running for a long while due to using recursive calls.
Npm packages
Meteor has been able to use npm packages directly for several years now, so this question amounts to finding job/worker/queue management packages on NPM. If you don’t care about persistence:
- Async “provides around 70 functions that include the usual ‘functional’ suspects (
map,reduce,filter,each…) as well as some common patterns for asynchronous control flow (parallel,series,waterfall…)” - d3-queue – minimalistic, written by D3 author Mike Bostock
If you do want persistence, since Meteor uses MongoDB already, it may be advantageous to use a job scheduling package with persistence to MongoDb. The most powerful and popular seems to be Agenda, but unfortunately it hasn’t been maintained in months, and it has a significant backlog of issues.
If you’re willing to add a dependency backed by redis to your project, there are more choices:
- bull – the most full-featured job queue solution for Node, backed by Redis
- bee – simple, fast, robust. Does not suffer from a memory leak that Bull exhibits
- Kue – priority job queue for Node
Like MongoDB, Redis can also provide high-availability (via Redis Sentinel), and if you want to distribute jobs among multiple worker machines, you can point them all at the same Redis server.