Younes El HamriData engineering · MoroccoAvailable 5 days/week

New property listings in front of sales in under five minutes

Client
Real-estate client (anonymised)
Sector
Real estate
Delivered
Delivered through Carré Agency
Period
2023
Stack
Playwright, Docker, Azure VMs, Redis, Pydantic, Postgres
New listing to saleshoursunder 5 minutes
Listing collectionmanual, 2 portalsautomated
In production2+ years

The code that changed

scraper/parse_listing.pya sketch of the change, not the client's code
− first version
price = page.locator(".listing-price span").inner_text()
rooms = page.locator("ul.features li:nth-child(2)").inner_text()
city = page.locator("div.location > a").first.inner_text()
+ what replaced it
raw = page.locator('script[type="application/ld+json"]').first.inner_text()

# Typed at the boundary: a portal renaming a field fails here, loudly,
# not three steps downstream in Postgres.
listing = Listing.model_validate(json.loads(raw))

CSS selectors broke on every redesign; the JSON-LD the portals publish for search engines barely moves. Two years at a 99.2% success rate.

Problem

The sales team copied listings off two portals by hand. By the time a listing reached their sheet, a competing broker had usually already called the owner, and in this market the first caller wins.

What I did

Five Playwright workers in Docker on Azure VMs poll only the latest-ads pages every five minutes. Redis deduplicates on URL, Pydantic normalises each listing into a typed record, and Postgres stores it for the sales queue.

The decision that made it last

The first version parsed CSS selectors, which is how most scrapers are written and why most scrapers break: one redesign and every rule is wrong.

Switching to the Schema.org JSON-LD already embedded in the listing pages changed the economics of the whole thing. That metadata is structured for search engines, and it barely moves when the visual layout does. Two years in production at a 99.2% success rate is that decision, not the crawler.

The rest of the design

Poll narrowly. Only the latest-ads pages, every five minutes. There is no reason to crawl a whole portal to find what was posted since the last run.

Deduplicate in Redis. URL-level, so a listing that appears on both portals reaches the sales queue once.

Validate at the boundary. Pydantic turns each listing into a typed record before it is allowed into Postgres, so a portal changing a field breaks loudly at the edge rather than quietly downstream.

What it changed commercially

Manual entry disappeared, and being first to call an owner roughly tripled conversion on hot leads. The engineering is unremarkable; the timing is the product.