December 2020 Meetup of the NL-RSE Community. Logo Banner

Location

The call will be conducted with the ZOOM conferencing platform. Follow Joining the Meeting instructions.

Zoom link will be published soon.

Registration

The registration is free

Please register via this Eventbrite page.

Agenda

Time Responsible Type Subject
09:00 - 09:15     hello & coffee
09:15 - 09:20 Mateusz Kuzak   Welcome and purpose
09:20 - 09:25 Mateusz Kuzak   Agenda and ground rules
09:25 - 09:30 everyone   Introductions and announcements
09:30 - 12:30 Mattia Mancini tutorial From a bash script to CWL
12:30 - 12:35   discussion Parking Lot
12:35 - 12:40   discussion Evaluation

Abstracts

From a bash script to CWL

In modern experimental sciences, the increasing amount of data to process and the development of standardized procedures to analyze it, is driving the utilization of automatic processing pipelines. Such pipelines combine different software components to carry out the computation. A wide range of solutions, ranging from bash scripts to python, can be used to perform such orchestration.

In ASTRON for the European Science Cloud project, we chose to use the Common Workflow Language (CWL) to describe one of our standard processing pipelines.

In this demo and tutorial, we will show our experience in converting our pipeline into the CWL (v1.0) standard and our experience with the available tools to run it and visualize it. We will follow by showing you how to create a CWL step and a CWL pipeline to describe an example workflow. Finally, we will conclude the meeting with our take on why we choose CWL and why this proved to be fit to solve our problems, and we would be welcome your opinions about that.

Our use case comes from astronomy, but this tutorial is focused on transferable skills which can be applied in other domains.

For the tutorial, we will be using google colab please make sure that you can execute it in your browser.

If you want to run the software on your computer, you will require the following tools installed on your laptop:

  • cwltool (to execute the workflow) - we recommend to use it in a virtual environment
  • singularity/docker (in case of the software we use in the demo is not available for your system)

As there is not much time during the tutorial, we suggest that, if you want to run the software on your computer, you make sure that you can run this example and docker or singularity is available.

Call for Contributions

We are always looking for speakers and exciting topics, please contact the host Mateusz Kuzak if you would like to give a talk or are interested in learning about some specific topics.

Contact

If you have any questions, please contact the host Mateusz Kuzak