cluster-computing

Starting Zookeeper Cluster. Error: Could not find or load main class org.apache.zookeeper.server.quorum.QuorumPeerMain

by

I got the Error: Could not find or load main class org.apache.zookeeper.server.quorum.QuorumPeerMain, because I had downloaded the apache-zookeeper-X.X.X.tar.gz file and not the apache-zookeeper-X.X.X.bin.tar.gz file. Downloading, untarring and using the bin.tar file fixed it for me. You can also build the binaries from the apache-zookeeper-X.X.X.tar.gz file; see the answer of @vincent.

how to specify error log file and output file in qsub

by

Typically error and output files are given as pbs directives in the qsub script or as command line options to the qsub script like so: #! /bin/bash #PBS -q queue_name #PBS -A account_name #PBS -l nodes=12:ppn=12 #PBS -l walltime=18:00:00 #PBS -e /mypath/error.txt #PBS -o /mypath/output.txt or as a command line option to qsub like so: … Read more

How to change memory per node for apache spark worker

by

When using 1.0.0+ and using spark-shell or spark-submit, use the –executor-memory option. E.g. spark-shell –executor-memory 8G … 0.9.0 and under: When you start a job or start the shell change the memory. We had to modify the spark-shell script so that it would carry command line arguments through as arguments for the underlying java application. … Read more

Error in SLURM cluster – Detected 1 oom-kill event(s): how to improve running jobs

by

The approved answer is correct but, to be more precise, error slurmstepd: error: Detected 1 oom-kill event(s) in step 1090990.batch cgroup. indicates that you are low on Linux’s CPU RAM memory. If you were, for instance, running some computation on GPU, requesting more GPU memory than what is available will result in an error like … Read more

NodeJS|Cluster: How to send data from master to all or single child/workers?

by

Because cluster.fork is implemented on top of child_process.fork, you can send messages from a master to the worker by using worker.send({ msg: ‘test’ }), and from a worker to a master by process.send({ msg: ‘test’ });. You receive the messages like so: worker.on(‘message’, callback) (from worker to master) and process.on(‘message’, callback); (from master to worker). … Read more

How to run Cron Job in Node.js application that uses cluster module?

by

If are using PM2, You can use an environment variable provided by PM2 itself called NODE_APP_INSTANCE which requires PM2 2.5 or greater. NODE_APP_INSTANCE environment variable can be used to determine difference between process, for example you may want to run a cronjob only on one process, you can just do this if(process.env.NODE_APP_INSTANCE == 0) { … Read more