If you ever had to run lots of small jobs on Compute Canada systems, you likely observed that the process is inefficient, time consuming, and inconvenient. The available tools for speeding up the workflow (the built-in Job Array feature of our scheduler SLURM, GNU Parallel, or GLOST) leave a lot to be desired. Here we introduce our own home-grown package META ( https://docs.computecanada.ca/wiki/META_package_for_serial_farming ) designed to overcome the shortcomings of the existing tools. META can significantly accelerate your workflow involving running lots of jobs (short or long, serial or parallel, CPU or GPU based). Features like the ability to resubmit all the jobs which either failed or never ran, using a single command, are likely unique to this package. In this webinar we will cover the main features of the package, and will also discuss the typical scenarios when META can become very useful or even essential.
_________________________________________________
This webinar was presented by Sergey Mashchenko (SHARCNET) on October 6th, 2021, as a part of a series of regular biweekly General Interest webinars ran by SHARCNET. The webinars cover different high performance computing (HPC) topics, are approximately 45 minutes in length, and are delivered by experts in the relevant fields. Further details can be found on this web page: https://helpwiki.sharcnet.ca/wiki/Online_Seminars . Subscribe to our twitter account (@SHARCNET) to stay updated about our upcoming webinars.
SHARCNET is a consortium of 19 Canadian academic institutions who share a network of high performance computers (http://www.sharcnet.ca). SHARCNET is a part of Compute Ontario (http://computeontario.ca/) and Compute Canada (https://computecanada.ca).
0 Comments