Skip to content
vfernandezg.devProjectsContact
All projects

Airbnb market analytics

Batch pipeline on Apache Spark and HDFS that processes Inside Airbnb open data for 12 cities, trains a price model and publishes the results to a web dashboard.

Home page of the dashboard, titled Short-term rentals in 12 cities. Four indicators, with 453,343 listings analyzed, 12 cities, 233.9 thousand hosts and 4.8 million reviews in 12 months, and below them a world map with one bubble per cityHome page of the dashboard, titled Short-term rentals in 12 cities. Four indicators, with 453,343 listings analyzed, 12 cities, 233.9 thousand hosts and 4.8 million reviews in 12 months, and below them a world map with one bubble per city
Role
University team project with five students in 2025; complete redesign and sole author of the current version in 2026
Period
Sep 2025 — Nov 2026
Status
Completed
Stack
  • Apache Spark
  • HDFS
  • Spark ML
  • Python
  • PostgreSQL
  • Next.js
  • Docker

The project started in 2025 as a team assignment for the Big Data course at Universidad Privada Antenor Orrego: a Kafka and Spark pipeline over the listings of Santiago de Chile. In November 2026 I rebuilt it from scratch on my own. The current version downloads the open data of Inside Airbnb for 12 cities, processes it with Apache Spark on HDFS, trains a price model and publishes the results to a web dashboard in English and Spanish.

A full run takes 453,343 listings, 165.5 million calendar rows and 17.9 million reviews from the source files to the site, in about 16 minutes on a laptop.

Architecture

Architecture diagram. The Inside Airbnb files enter a data lake on HDFS with four zones, Raw, Bronze, Silver and Gold, processed by a local Apache Spark cluster. Spark ML trains the model on Silver and writes metrics and predictions to Gold. The publish stage loads the marts into PostgreSQL, where the Next.js site reads them. Below, the six stages of a run with their duration: ingest, bronze, silver, gold, train and publish Architecture diagram. The Inside Airbnb files enter a data lake on HDFS with four zones, Raw, Bronze, Silver and Gold, processed by a local Apache Spark cluster. Spark ML trains the model on Silver and writes metrics and predictions to Gold. The publish stage loads the marts into PostgreSQL, where the Next.js site reads them. Below, the six stages of a run with their duration: ingest, bronze, silver, gold, train and publish

Each layer of the lake has a single responsibility:

  • Raw. The source file untouched, with its size and sha256. Inside Airbnb only publishes the latest snapshot of each city, so this zone is also the historical archive.
  • Bronze. The same content in Parquet, with every column as text. Only the format changes, and each file is fitted to a column contract: a schema change at the source does not shift data.
  • Silver. Typed tables, with prices converted to US dollars, no duplicates and quality flags.
  • Gold. Small aggregates, one for each question the dashboard asks. The site never touches raw data.

Everything runs on a local cluster defined with Docker Compose: HDFS, Spark in standalone mode with a master and two workers, and a PostgreSQL database that mirrors the production schema. The root of the lake is a Hadoop URI, so the same code works on hdfs://, s3a:// or abfss://.

Engineering decisions

  • Incremental and idempotent. Each stage processes only the city and snapshot partitions that are missing, and running it again duplicates nothing. When the source publishes a new snapshot, it is stored next to the previous ones and the marts move to the most recent.
  • The CSV is read according to RFC 4180. The Santiago file holds 18,904 records in 32,585 lines, because there are line breaks inside fields. The original version read it line by line and the columns shifted. That case now has a regression test.
  • Doubtful rows are flagged, not deleted. An outlier price, coordinates outside the city or an invalid room type stay as flags in Silver, and each consumer decides what to filter.
  • Validations that stop the publication. Duplicate listings, invalid room types, cities without data or repeated calendar dates halt the pipeline before anything reaches the site.
  • Performance without Python UDFs. Every transformation is a native Spark expression. The calendar is reduced to listing by month on its way to Silver, before any join, and Bronze unions the compressed files, which cannot be split, into a single plan so they are processed in parallel.
  • A new city is one configuration entry. The source catalog has 124 cities; adding one to the cities.yaml file and running again processes only that city.

The pipeline running

What follows is the record of a full run on the local cluster, from 15 November 2026. The Spark master, with its two workers and the ten cores taken by the application:

Spark master interface. Two workers in the ALIVE state with their five cores in use each, 10 of 10 cores and 4.9 of 6 GiB of memory taken, and the airbnb-lake:run application in the RUNNING state Spark master interface. Two workers in the ALIVE state with their five cores in use each, 10 of 10 cores and 4.9 of 6 GiB of memory taken, and the airbnb-lake:run application in the RUNNING state

In bronze, each compressed file from the source is one task, and the tasks are spread over the two executors:

Stages tab of Spark during bronze. One active stage, compressed CSV to Parquet, with 9 of 12 tasks finished and 3 running, and below it the stages already completed Stages tab of Spark during bronze. One active stage, compressed CSV to Parquet, with 9 of 12 tasks finished and 3 running, and below it the stages already completed

In silver the shuffles of the deduplication show up, with their read and write columns:

Stages tab of Spark during silver. The active stage, for typing, prices in USD, deduplication and quality, with several tasks running, and the earlier stages with their shuffle columns Stages tab of Spark during silver. The active stage, for typing, prices in USD, deduplication and quality, with several tasks running, and the earlier stages with their shuffle columns

The plan of the listings query starts with the read from Bronze and two broadcast joins against two small tables, cities and exchange rates. Further down, outside the capture, come the windows that remove duplicates and the write:

Plan of the Silver query in the SQL tab of Spark, completed in 48 seconds. Three scans, of 453,343 listings, 12 cities and 30 exchange rates, joined by two BroadcastHashJoin nodes before a sort Plan of the Silver query in the SQL tab of Spark, completed in 48 seconds. Three scans, of 453,343 listings, 12 cities and 30 exchange rates, joined by two BroadcastHashJoin nodes before a sort

In train, fitting the model launches hundreds of short jobs, 937 in this run, with the ten cores busy:

Jobs tab of Spark during train. Two active jobs and 968 completed, all with the description Train, GBT price model and evaluation Jobs tab of Spark during train. Two active jobs and 968 completed, all with the description Train, GBT price model and evaluation

Executors tab of Spark during train. Ten active tasks, five on each of the two executors, and more than 13,000 completed tasks Executors tab of Spark during train. Ten active tasks, five on each of the two executors, and more than 13,000 completed tasks

When it finishes, each zone of the lake takes this much on HDFS:

/lake/raw      2.7 GB   source files, compressed
/lake/bronze   2.0 GB   Parquet, all text
/lake/silver   349 MB   typed Parquet; calendar by listing and month
/lake/gold       7 MB   17 data marts

Those 17 Gold marts, here in the NameNode file browser, are all that reaches the site:

File browser of the HDFS NameNode in the /lake/gold folder, with 17 folders, one per data mart, among them city_kpis, map_grid, model_metrics and price_histogram

The price model

A gradient-boosted trees model (Spark ML’s GBT) predicts the logarithm of the nightly price from the city, location, size, room type, amenities, host and reviews. On the 20% of listings held out for testing, 69,100 listings, it reaches an R² of 0.83 and a median relative error of 20.1%. The baseline rule, which predicts the median of each city and room type, stays at 0.54 and 34.5%.

Price model page with four indicators: explained variance of 0.83, median relative error of 20.1%, mean absolute error of 50 dollars and 69,100 test listings. Below, the error by city against the baseline rule and the fifteen features that matter most, led by the city and the property type Price model page with four indicators: explained variance of 0.83, median relative error of 20.1%, mean absolute error of 50 dollars and 69,100 test listings. Below, the error by city against the baseline rule and the fifteen features that matter most, led by the city and the property type

No feature is derived from the price: there is an explicit list of forbidden columns and a test that checks it. The split between training and test is deterministic, by a hash of the identifier, so a listing always falls in the same set.

The model is not equally accurate across the range. It overshoots on the cheapest listings and falls short on the most expensive ones, and the site shows it with a chart of bias by price bracket. In addition, a K-Means groups the neighborhoods of all cities by relative price, review activity, availability and type of supply; the number of groups is chosen with the silhouette coefficient.

The dashboard

The site is built with Next.js and TypeScript. Pages are pre-rendered from the marts and served from cache, so a normal visit runs no query. When it finishes publishing, the pipeline calls a revalidation endpoint and the pages are regenerated on the next visit. The site connects to the database with a read-only role.

Each city has its own page, with a price map on MapLibre GL that switches between a heat map and the price per cell of about 500 meters:

Santiago page. Indicators for a median nightly price of 56 dollars, 18,904 listings, an estimated occupancy of 21% and an average rating of 4.79, and below them a heat map that concentrates the highest prices in the north-east of the city Santiago page. Indicators for a median nightly price of 56 dollars, 18,904 listings, an estimated occupancy of 21% and an average rating of 4.79, and below them a heat map that concentrates the highest prices in the north-east of the city

The charts use Apache ECharts, loaded separately so it stays out of the initial JavaScript of every page. Each chart has an equivalent table, so no value depends only on color or on hovering, and the palette is validated for color blindness in light and dark mode.

Seasonality heat map with twelve cities in rows and the months in columns. Red marks the high season and blue the low season; Lisbon is the most seasonal and Bogotá the steadiest Seasonality heat map with twelve cities in rows and the months in columns. Red marks the high season and blue the low season; Lisbon is the most seasonal and Bogotá the steadiest

The language is part of the URL, with translated routes, and is not stored in cookies.

Quality and delivery

  • Tests. 94 tests with pytest and Spark in local mode, including a full run over a temporary lake and the publication to PostgreSQL, which checks that the load is atomic and that the read-only role has the right permissions.
  • Static analysis. ruff and strict mypy on the pipeline; ESLint and TypeScript on the site.
  • CI. GitHub Actions runs lint, types and tests for the pipeline, and lint, types and build for the dashboard, on every pull request.
  • Smoke tests. Playwright walks the site on desktop, in dark mode and on mobile, and the same tests can run against the deployed site.
  • Infrastructure. The database on Neon, the project on Vercel and the DNS are defined with Terraform in a separate repository.

From the original version to the current one

  • From one city with an 18-column summary CSV to 12 cities with the detailed 90-column files, the calendar and the reviews.
  • From loading each CSV into memory with pandas and resending it row by row through Kafka to a streaming download straight into the lake.
  • From no Silver layer and no partitioning, with duplicated data on every rerun, to persisted, partitioned and idempotent layers.
  • From a model with an R² close to zero to one at 0.83 with no target leakage.
  • From opening a Parquet file by hand in Power BI to a website with pre-aggregated data and caching.
  • From no tests to 94 tests, strict types and CI.

The code of the first version is kept in the legacy/ folder of the repository, with its authorship.

Limits

  • Prices are the advertised ones, and many listings do not publish a price (in Paris, 44%). The price statistics and the model use only the ones that do.
  • The conversion to US dollars uses a fixed exchange rate, so that repeating a run gives the same result.
  • Occupancy is an estimate by Inside Airbnb based on reviews.
  • The local cluster has a single DataNode, with no replicas: it is good for developing and demonstrating, not for keeping data.

The data comes from Inside Airbnb and is used under the CC BY 4.0 license. The site publishes only aggregates, with no host names or review text. The project is independent and is not affiliated with Airbnb or Inside Airbnb.