Python on Supercomputers: Lessons from Scaling Bioinformatics Workflows

Modern DNA and RNA sequencing technologies generate datasets that can easily reach terabytes in size, making large-scale bioinformatics analyses impractical on a standard desktop computer. While Python is widely used throughout bioinformatics, moving research workflows from a laptop to a High-Performance Computing (HPC) system involves far more than submitting a job to a scheduler. Workflows must execute reliably in non-interactive environments, efficiently utilize shared computing resources, and remain reproducible across large-scale analyses.
In this talk, I will share practical lessons from adapting and optimizing Python-based bioinformatics workflows for the Centre for High Performance Computing (CHPC), South Africa. Using a transcriptomics analysis pipeline as a case study, I will demonstrate how an existing Python workflow was transformed into a robust HPC application by replacing interactive execution with queue-safe command-line interfaces, integrating PBS job scheduling, improving file and input validation, strengthening error handling, and automating large-scale analyses. I will also show how Python integrates with established bioinformatics tools and R-based analyses to create reproducible workflows for read mapping, feature counting, differential expression analysis, and downstream visualization.
Beyond this case study, I will discuss common challenges researchers encounter when running Python on HPC systems, including inefficient file I/O, excessive memory usage, poor resource allocation, and workflows that fail when moved from an interactive desktop environment to a batch scheduler. Attendees will learn practical techniques for improving the reliability, scalability, and reproducibility of existing Python applications without having to rewrite them from scratch.
Whether you are a Python developer interested in scientific computing, a student entering computational biology, or a researcher looking to scale data-intensive analyses, this talk will provide practical techniques that can be applied immediately to move Python workflows from a laptop to a production HPC environment.