Building a Serverless Computation Environment with Python

Speaker: Ric da Silva

Track: Infrastructure / DevOps

Type: Talk

This talk will be a tour and demo of a virtual machine, compiler, and DSL called Teal (https://condense9.com), and is for anyone who builds data processing systems (or is interested in language design!).

Teal is for when you need to run sequences of tasks on a serverless platform (e.g. AWS Lambda), but don't want the huge burden of all the glue-infrastructure. It's an alternative to AWS Step Functions or Apache Airflow for building workflows with many steps, branches and loops.

The goal is to be able to take existing Python code and get it running with as little complexity as possible, and without any long-running infrastructure. It’s easy to deploy one Lambda function (e.g. with the well known Serverless Framework), but it’s much harder to write several functions and pass data between them. The Teal project started as an experiment to see whether it’d be feasible to simplify this by using familiar programming constructs (async/await concurrency, variables, functions, etc) to describe the workflows.

There will be a live demo (via screen-share) of using Teal to build, test and deploy a non-trivial data pipeline, taking just a few minutes from start to finish. This might have taken several hours before, or longer if you didn’t have orchestrator or task-runner infrastructure set up already. Follow along with the code on GitHub: https://github.com/condense9/teal-demos.

Prior knowledge of serverless is useful, but not essential - we'll briefly cover the necessary concepts.