ScaDS.AI-Tutorial: Big Data Processing on HPC

Datum

26.10.2023

Zeit

10:00 - 15:00

Sprecher

Apurv Deepak Kulkarni, Pramod Baddam, Wenyu Zhang

Zugehörigkeit

ScaDS.AI Dresden/Leipzig

Serie

ScaDS.AI Training

Sprache

en

Hauptthema

Informatik

Beschreibung

Apache Spark and Apache Flink are two typical Big Data analytics frameworks. Their APIs allow the development and testing of an application on a local workstation and later, without changing the source code of the application, distribute work to many computers when the local workstation is not sufficient anymore due to limited resources. The course focuses on the step from a local workstation to an HPC environment and presents how the typical Big Data analysis workflow can be organized in an HPC environment. In this course participants will be introduced to running a data pipeline and data processing along with managing the configurations on the HPC environment, using Apache Flink and Apache Spark. Agenda - Introduction and access to the ZIH HPC system - Considerations about the hardware environment - Setup of the required software environment - Distributed computing with Big Data frameworks - Efficient configuration of Big Data frameworks - Hands-on session Pre-knowledge - Basic knowledge of Big Data frameworks (e.g. Apache Flink, Apache Spark) is recommended (but not required) - Basic HPC Knowledge is recommended (but not required) Course language: English Registration is required. Participation is free.

Links

• Registration

Letztmalig verändert: 31.01.2023, 08:20:39