Leveraging Python and ClickHouse to put 430TiB at your application's finger tips

Aruba-UXI is an IoT platform that collects and delivers beautifully expressive, dynamic, and blazing-fast network diagnostic metrics to its customers. With over 100,000 sensors publishing around 20,000 measurements per second, the platform has its work cut out for it. The ClickHouse cluster is now the backbone of three different product features, hosting 430 TiB of uncompressed data and serving 150 requests per second — all being wrangled together with simple FastAPI web servers and orjson to help improve serialization performance.

Leveraging Python's flexible and dynamic nature provides a good balance between development speed and application speed, so long as data processing is performed at the appropriate levels. As data moves further from the source, it can be easier to manipulate, but the cost of manipulation increases. In this talk, we will discuss how UXI finds that balance and leverages the features available within the toolchain to optimize big data applications.