Links

▼
Wednesday, October 1, 2025

Why I’m Loving Spark 4’s Python Data Source (with Direct Arrow Batches)

›
TL;DR:  Apache Spark 4 lets you build  first-class  data sources in pure Python. If your reader yields  Arrow  RecordBatch  objects, Spark i...
Monday, September 8, 2025

Troubleshoot I/O & Wait Latency with OraLatencyMap and PyLatencyMap

›
I recently chased an Oracle performance issue where most reads were sub-millisecond (cache) , but a thin band around ~10 ms (spindles) domi...
Monday, March 17, 2025

ATLAS DCS Analysis with Apache Spark and Jupyter Notebooks

›
The ATLAS Detector Control System (DCS) at CERN is essential for ensuring optimal detector performance. Each year, the system generates t...
Thursday, February 27, 2025

Kepler’s Mars Orbit Analysis with Python Notebooks & AI-Assisted Coding

›
Johannes Kepler ’s analysis of Mars’ orbit stands as one of the greatest achievements in scientific history, revealing the elliptical natur...
Friday, April 26, 2024

Building an Apache Spark Performance Lab: Tools and Techniques for Spark Optimization

›
Apache Spark is renowned for its speed and efficiency in handling large-scale data processing. However, optimizing Spark to achieve maximum ...
Tuesday, January 30, 2024

Enhancing Apache Spark and Parquet Efficiency: A Deep Dive into Column Indexes and Bloom Filters

›
In the ever-evolving landscape of big data, Apache Spark and Apache Parquet continue to introduce game-changing features. Their latest updat...