James Westover
Distributed Systems Engineer — Kafka, Cassandra, Storm, Real-Time Pipelines, High-Throughput & Fault-Tolerant Systems
Profile
Distributed systems engineer with a decade of production experience designing and operating high-throughput, fault-tolerant infrastructure at Uber, Postmates, and HERE Technologies. Deep hands-on expertise across the core distributed systems stack: Kafka for durable event streaming, Cassandra tuned to 2–10M QPS with explicit consistency model selection (QUORUM, LOCAL_QUORUM, ONE), Apache Storm for real-time stream processing, Redis for low-latency shared state, and Elasticsearch for indexing at scale. At Uber, owned architecture for a globally distributed Cassandra cluster supporting location data ingestion for millions of drivers — driving global data reliability from ~80% to 97%+ and saving $30M/year via spoofing detection built on top of that pipeline. At HERE Technologies, designed a lambda architecture using Storm, Kafka, Elasticsearch, and PostGIS to ingest and process 250M+ GPS events per day from a global mapping fleet. Comfortable with the full systems spectrum: consistency models, replication strategies, capacity planning, load testing, failure modes, and cross-datacenter coordination. Currently based in Berlin, available for fully remote senior/staff distributed systems roles.
Core Distributed Systems Competencies:
- Apache Kafka: Production event streaming at Uber (location data pipeline), HERE (fleet data ingestion), Sonobi (ad impression analytics), and URBN (product catalog event workflows). Confluent Platform experience. Designed Kafka-backed "Weatherman" orchestration system for dynamic map reprocessing workflows at HERE
- Apache Cassandra (2–10M QPS): Owned globally distributed Cassandra infrastructure at Uber's Location Platform — explicit tuning of consistency models (QUORUM, LOCAL_QUORUM, ONE) for read/write performance and availability tradeoffs. Multi-region deployment, tombstone behavior management, capacity planning, schema design for high-write workloads
- Apache Storm (Stream Processing): Architected Python bolt-based Storm topology at HERE Technologies for real-time fleet telemetry — processing sensor, GPS, and IMU streams from mapping vehicles globally. Designed batch and streaming paths in lambda architecture with shared Kafka backbone
- Redis: Used for shared distributed state in probabilistic overlap estimation (HyperLogLog + MinHash sketches for 250M+ daily impression user groups at Sonobi), session state, and distributed cache layers. Production Redis experience across multiple orgs
- Elasticsearch: Production indexing at HERE Technologies for geospatial fleet analytics with PostGIS. Designed index mappings and query patterns for real-time fleet state search and historical analytics
- Consistency Models & Replication: Extensive hands-on experience with CAP tradeoffs in production — Cassandra consistency level selection under real traffic, multi-region replication strategy design, read/write quorum sizing for latency vs. durability. Understood and applied these in high-stakes contexts (GPS safety data, financial fraud detection)
- Fault Tolerance & Reliability: Drove Uber Location Platform ingestion reliability from ~80% to 97%+ globally. Led incident response for GPS millisecond clock offset bug affecting millions of devices. On-call rotations at Postmates peak-traffic war room calls. CPU Guardian service for Pi cluster stability. Deep experience with failure mode analysis and postmortem-driven improvements
- Capacity Planning & Load Testing: Scaled Cassandra cluster to handle 2–10M QPS. Built Python simulation harness at Uber scaling from 500 to 10,000 trips per test run. Delivered 4x CI build time improvement at Postmates through distributed test sharding across Kubernetes pods
- Geospatial Distributed Systems: Deep expertise in S2 and H3 cell indexing (Uber), PostGIS (HERE), real-time geohashing, multipath GPS correction (ShadowMaps at Uber), off-route detection with sensor fusion. Designed systems handling location data at global fleet scale
- Data Serialization & RPC: gRPC, Apache Thrift, Protobuf, Apache Avro, Apache Parquet at production scale. Primary contributor to tchannel-python (Tornado-based Thrift RPC transport) — Uber's internal RPC fabric used by thousands of services
- Big Data & Batch Processing: Apache Spark, Hadoop MapReduce, Hive, HBase, Druid, Presto, ClickHouse, Amazon EMR, Athena, Google Bigtable, Amazon Redshift. Designed ETL pipelines for 250M+ daily events at Sonobi
- Service Mesh & Observability: Envoy service mesh, OpenTelemetry distributed tracing, Prometheus, Grafana, Datadog, ELK Stack. Structured logging design for distributed system observability
Apache Kafka
Cassandra
Apache Storm
Redis
Elasticsearch
Zookeeper
Python
Go
Java
gRPC
Thrift
Protobuf
Avro
Kubernetes
Docker
PostgreSQL / PostGIS
HBase
Druid
Spark
Flink
Hadoop
Envoy
OpenTelemetry
Prometheus
S2 / H3
AWS (ECS, EMR, Kinesis)
GCP Bigtable
Ansible
GitHub Actions
Selected Experience
Uber — San Francisco, CA / Remote
Staff Software Engineer — Navigation Platform · February 2021 – October 2022
- Senior technical contributor on Uber's navigation platform — one of the company's largest distributed systems, operating at approximately 10,000 nodes per data center and serving all routing and mapping globally across 30M+ driver devices.
- Introduced off-route cross-street link support to the route object, reducing p95 off-route detection latency from ~5 minutes to ~2 minutes — a critical safety and UX metric affecting millions of active trips daily.
- Developed Python-based multiprocessing simulation harness scaling trip simulation capacity from 500 to 10,000 trips per run, enabling rapid regression testing across route deviation detection algorithms.
- Led critical incident response for GPS millisecond clock offset bug affecting data integrity across millions of devices globally — diagnosed root cause, coordinated multi-team remediation across sensor fusion, location, and navigation systems.
- Maintained J2ObjC transpilation system enabling Java routing logic to be shared directly with iOS clients — one of the few examples of cross-platform code sharing at this scale in the industry.
- Led Postmates acquisition integration: maintained business continuity for legacy distributed systems while designing compliant data deletion strategies across Uber's Schemaless distributed datastore using Go-based microservices.
Senior Software Engineer — Python Platform (Office of CTO) · January 2019 – June 2020
- Founding member of Uber's Python Platform team — mapped the entire Python dependency graph across hundreds of microservices, targeting the top 200 repositories and achieving 93% service conversion to Python 3 through purpose-built automation tooling.
- Major contributor to tchannel-python — the Tornado-based Thrift RPC transport library forming the backbone of inter-service communication across Uber's distributed system. Led Python 2→3 migration of the core transport layer.
- Became de facto Python infrastructure support for Uber's data science organization — maintained distributed notebook infrastructure, supported distributed query system, debugged Airflow fork used for ML pipeline orchestration.
Software Engineering Manager — Location Platform · February 2018 – January 2019
- Founded and built Uber's Location Platform team from scratch — designed architecture to centralize GPS and sensor data ingestion from the global driver and mapping fleet.
- Drove platform reliability from ~80% to 97%+ global data ingestion success — improving data quality for Uber's RideCheck safety initiative (crash detection, trip anomaly identification).
- Owned architecture strategy for globally distributed Cassandra cluster at 2–10M QPS: multi-region deployment design, consistency model selection (QUORUM vs. LOCAL_QUORUM vs. ONE) based on read/write priority, tombstone management for high-churn location data, and capacity planning for global scale.
- Directed anti-location-fraud program — built spoofing detection pipeline on top of location ingestion infrastructure, delivering $30M+ in annual savings by identifying and blocking GPS coordinate manipulation by drivers.
- Launched ShadowMaps initiative for multipath-aware GPS correction in urban canyons — real-time geospatial correction using H3/S2 cell indexing and shadow volume computation. (Engineering blog)
- Ensured GDPR compliance for location data collection from millions of drivers across EU markets — designed data minimization, consent propagation, and deletion workflows through the distributed pipeline.
- Made key engineering hires who went on to senior and staff roles; built team culture of high reliability and operational rigor from day one.
Postmates Inc. — San Francisco, CA
Senior Software Engineer 2 — Core Infrastructure · June 2020 – February 2021
- Led monorepo-wide Python 2→3 migration across 1M+ lines of code serving 100 engineers — including 3 coexisting Django versions with deep bytes-vs-unicode incompatibilities in high-throughput, latency-sensitive code paths.
- Delivered 4x CI build time improvement by orchestrating distributed test execution across Kubernetes pods — transforming 40–50 minute feedback loops into 10–15 minutes through intelligent test sharding and parallelization.
- Participated in weekly war room calls during peak-traffic scaling incidents — real-time incident response with infrastructure leadership during Postmates' highest-demand periods.
- Partnered with infrastructure team on Envoy service mesh adoption and OpenTelemetry distributed tracing integration — improving observability across the microservice fleet.
- Contributed to database reliability improvements using PostgreSQL with pgpool and PgBouncer connection pooling for high-concurrency marketplace traffic.
HERE Technologies — Berlin, Germany
Software Engineering Manager → Senior Software Architect · April 2015 – February 2018
- Architected a lambda data architecture for global mapping fleet telemetry: Apache Storm (Python bolts) for real-time stream processing, Kafka for durable event ingestion, Elasticsearch and PostgreSQL/PostGIS for storage and geospatial query — processing 250M+ GPS events per day from HERE's worldwide fleet of capture vehicles.
- Designed and built "Weatherman" — a Kafka-based orchestration system for dynamic map reprocessing workflows and QA pipeline management. Event-driven coordination of batch reprocessing jobs triggered by real-time quality signals from the streaming path.
- Designed Tornado-based backend APIs and React frontends for the Fleetvision fleet management platform — real-time GPS, IMU, and camera telemetry visualization including a 16-display video wall at HERE's Berlin HQ.
- Integrated JPEG-compressed sensor streams from FPGA-based encoder boards in the HT4 next-generation mapping rig — designed software abstraction layer for stream handling, metadata tagging, and reliable cloud ingestion coordination.
- Built and scaled engineering organization from 2 to 15 engineers across backend, frontend, mobile, DevOps, and embedded subteams — delivering 8+ concurrent programs including ADAS prototyping and corporate technology partnerships.
- Led ISO 9001 certification for safety-critical software governing mapping vehicles operating on public roads — authored QMS documentation and led external auditor engagement.
- Managed technology partnerships with Intel, Nvidia, Audi, and Mercedes-Benz for sensor-to-cloud ADAS and autonomous driving data infrastructure.
- Developed ADAS prototyping pipelines using ROS (Robot Operating System) for rapid sensor fusion and localization prototyping on real mapping vehicle data.
Sonobi — New York, NY
Software Engineer · December 2013 – January 2015
- Built RabbitMQ-based distributed job processing system in Python using multiprocessing and multithreading — processing analytics across 250M+ daily ad impressions.
- Explored log analytics pipeline using Kafka, Logstash, and Hadoop for impression-level event aggregation and reporting.
- Designed probabilistic user group overlap estimation using HyperLogLog and MinHash sketches stored in Redis — efficient intersection approximation for large user segments without storing full sets.
- Built ETL data warehousing pipeline using S3, Amazon Redshift, and Python — aggregating impression-level data for campaign analytics and optimization.
URBN — Philadelphia, PA (Remote)
Software Engineer · September 2024 – January 2025
- Maintained Django application managing URBN's internal product catalog across multiple retail brands.
- Contributed to asynchronous task processing systems using Celery and Kafka for event-driven workflow orchestration across the platform.
- Improved build systems and developer tooling for enhanced engineering velocity.
Westover Labs / Neher Data Services — Berlin, Germany
Founder (Westover Labs) & CTO (Neher Data Services) · 2021 – Present
- Architect and operate production distributed infrastructure: PostgreSQL with pgvector for semantic search (knowledge graph service), FastAPI inference services with structured logging and health monitoring, Ansible-managed deployment across bare-metal and cloud nodes.
- Build event-driven automation systems with webhook-based coordination, message queue integration, and resilient retry logic — applying distributed systems design patterns to agentic AI workflows.
- Design Bayesian sports modelling and DFS optimization system using PyMC5, Polars, and DuckDB — probabilistic inference pipelines with structured uncertainty quantification.
- Pioneer AI-augmented development methodology: orchestrate 12 specialized AI agent personas with intent routing, persistent knowledge graph context, and Jira automation — achieving team-level throughput as a solo engineer.
University of Rochester — Rochester, NY
Assistant Systems Administrator · October 2006 – October 2012
- Built and maintained Beowulf-style compute clusters using ROCKS with MPICH2 and OpenMPI for distributed HPC workloads — Monte Carlo simulations, molecular dynamics (LAMMPS), and electronic structure calculations (VASP).
- Constructed private cloud cluster using Eucalyptus (open-source EC2), later migrated to VMware — early hands-on experience with distributed virtualization and elastic resource management.
- Designed and implemented microsecond-precision timing systems for laser cooling in Bose-Einstein condensate experiments — real-time hardware interfacing under strict latency requirements.
Technical Depth: Distributed Systems Stack
- Messaging & Streaming: Apache Kafka (Confluent Platform), Apache Storm (Python bolts, real-time topologies), RabbitMQ, Amazon Kinesis, Celery. Designed both streaming and batch paths in lambda architectures.
- Distributed Databases: Cassandra (multi-region, 2–10M QPS, QUORUM/LOCAL_QUORUM/ONE consistency tuning, tombstone management, capacity planning), Google Bigtable, HBase, Amazon Redshift, DuckDB, PostgreSQL (PostGIS, pgpool, PgBouncer), MySQL, Oracle.
- Caching & In-Memory: Redis (HyperLogLog, MinHash, session state, distributed cache), Memcached, Varnish.
- Search & Analytics: Elasticsearch, Apache Solr, Druid, Presto, ClickHouse, Hive, Athena, S3 Select.
- RPC & Serialization: Apache Thrift, gRPC, Protobuf, Apache Avro, Apache Parquet. Primary contributor to tchannel-python (Uber's Tornado-based Thrift transport).
- Coordination: Zookeeper for distributed coordination; Kubernetes for container orchestration and distributed test execution.
- Observability: Envoy service mesh, OpenTelemetry distributed tracing, Prometheus, Grafana, Datadog, ELK Stack, structured logging.
- Geospatial Distributed Systems: S2 cell indexing, H3 hexagonal indexing, PostGIS, real-time geohashing, multipath GPS correction, sensor fusion pipelines at global fleet scale.
- Cloud Infrastructure: AWS (ECS, EC2, S3, EMR, Kinesis, Lambda, SageMaker, CloudFormation), GCP (Bigtable, Compute Engine), Azure. On-prem (XenServer, Eucalyptus, Mesosphere/DCOS, SaltStack).
- Languages: Python (primary — 2.4 through 3.13), Go (Postmates/Uber integration), Java (navigation platform), C++, Bash/Zsh.
- Consistency & Replication: Deep experience with CAP theorem tradeoffs in production — explicit consistency level selection in Cassandra, multi-region replication strategy design, quorum sizing for latency vs. durability under real traffic.
Education
MBA, Business Administration — Quantic School of Business and Technology, 2017
PhD coursework, Physics (Quantum Chemistry, Solid-State Physics, Statistical Mechanics) — University of Central Florida, 2009–2013
MS, Physics — University of Central Florida, 2010
BS, Physics & Applied Mathematics — University of Rochester, 2009
Publications