Accelerate your pandas workload using FireDucks at zero manual effort

Speaker: Sourav Saha

Track: Data Science

Type: Remote Talk

Room: Central Room (Seminar Room 2)

Time: Oct 03 (Thu): 14:00

Duration: 0:45

In general, a Data Scientist spends significant efforts in transforming the raw data into a more digestible format before training an AI model or creating visualizations. Traditional tools such as pandas have long been the linchpin in this process, offering powerful capabilities but not without limitations. With numerous possible ways to write the same thing in pandas, often a user ends up selecting the uneconomical, inefficient ones, leading to large computational costs with the growth in data size. We introduce a couple of frequently occurring intricate performance issues in pandas, along with a compiler-accelerated high-performance library named FireDucks to auto-detect and optimize those issues without any manual effort. We will also demonstrate how FireDucks can outperform the existing high-performance pandas alternatives.

The growth of data sizes and the increase in performance cost create the demand for high-performance DataFrame libraries. However, the existing pandas alternatives often compel a user to learn completely new APIs (incurring migration cost), whereas some of the others demand a more efficient computational system (incurring hardware cost). To address the same, we at NEC R&D Lab Japan, have developed FireDucks, a solution that’s been crafted for the contemporary data professional who loves flexible user APIs in pandas and wants to enhance the performance of their application without any extra hardware cost when dealing with voluminous and complex data on a regular basis. It is released on pypi.org under the 3-Clause BSD License and can be simply installed using pip.

Here is the outline of the talk:

1. Current challenges of large-scale data analysis using pandas and other libraries. (5 mins)
2. Introduction to FireDucks (developed at NEC R&D Lab) and its offerings. (10 mins)
3. Demo on automatic acceleration of pandas workload using FireDucks. (5 mins)
4. Tricks and Tips on writing better code related to large-scale data analysis. (5 mins)
5. Performance comparison of FireDucks and other data analysis Python libraries. (5 mins)

The key takeaways from the session would be as follows:

1. How the choice and execution order of API calls in writing an application (not limited to pandas) impacts its performance.
2. How compiler technologies can be useful to auto-detect and optimize existing performance issues in a pandas-like application, enabling a user to write efficient code.
3. How FireDucks can help to focus more on in-depth data analysis instead of spending hours optimizing an existing code at the production level.

Target Audience: Whoever uses pandas or related libraries for large-data analysis shall find this session useful.