Parsl
Brief for “Parallel Scripting Library,” Parsl helps you to take computing jobs and break up them throughout a number of programs utilizing roughly the identical syntax as Python’s present Pool objects. It additionally helps you to sew collectively totally different computing duties into multi-step workflows, which might run in parallel, in sequence, or by way of map/scale back operations.
Parsl helps you to execute native Python purposes, but additionally run some other exterior utility by the use of instructions to the shell. Your Python code is written like regular Python code, save for a particular perform decorator that marks the entry level to your work. The job-submission system additionally offers you fine-grained management over how issues run on the targets—for instance, the variety of cores per employee, how a lot reminiscence per employee, CPU affinity controls, how usually to ballot for timeouts, and so forth.
One glorious function Parsl gives is a set of prebuilt templates to dispatch work to a wide range of high-end computing sources. This not solely contains staples like AWS or Kubernetes clusters, however supercomputing sources (assuming you may have entry) like Blue Waters, ASPIRE 1, Frontera, and so forth. (Parsl was co-developed with the help of lots of the establishments that constructed such {hardware}.)
