Hanger

Retailer Scraping with Puppeteer

Used Puppeteer to crawl retailer category pages and product pages with normalized field extraction and per-retailer parsing modules so one site change didn't break the entire pipeline.

• Extracted normalized fields: title, brand, price, sale_price, currency, images, sizes, color, material, category, product_url, retailer, SKU/variant IDs
• Anti-breakage patterns: retry logic, exponential backoff, selector fallbacks
• Per-retailer parsing modules isolated failures to individual crawlers

Scraping at Scale

• Distributed scrape jobs batched by retailer/category with controlled concurrency to avoid rate limits
• Stored raw HTML snapshots + parsed payloads for debugging and diffing when retailers changed page structure
• Predictable run durations with worker pool limiting and per-domain throttles

Admin Interface + Cron-Controlled Refresh

Built an admin dashboard for full operational control over the ingestion pipeline.

• Enable/disable retailers, categories, or individual crawlers on the fly
• Set scrape frequency per crawler (e.g., 1×/day or more for fast-moving inventory)
• Trigger manual re-runs and view job health: success %, failures, last run, duration
• Upserts on each run; missing items marked 'inactive' instead of hard-deleted for churn tracking + recovery

Embeddings + Vector Indexing (CLIP + Pinecone)

For every product we precomputed embeddings at ingestion time so nothing ran on the user query path.

• Generated CLIP image embedding + text embedding per product at ingestion
• Stored in Pinecone: vector = CLIP embedding, metadata = retailer, price range, category, size availability, gender, color
• Precomputing kept search fast and latency predictable — no embedding generation at query time

Query → Retrieval → Rerank

• Parse intent + constraints from user query (e.g. 'black mini dress under $120' → color=black, category=dress, price<120)
• Retrieve candidates from Pinecone via semantic vector search (CLIP text embedding of query, or image embedding for inspo images) + hard metadata filters (price, retailer, category, in-stock sizes)
• Re-rank with blended score: vector similarity distance + inventory confidence (in-stock at last refresh) + preference boosts (brands saved, liked styles)
• Return paginated results with stable sorting so items don't shuffle between pages

Signals

• Implicit: clicks, dwell time, add-to-collection, 'more like this'
• Explicit: likes/dislikes, brands to follow/avoid, price comfort range, preferred categories

Recommendation Flow

• Maintain a user preference profile: embedding centroid from liked items, negative centroid from disliked items, structured constraints (price, categories)
• Periodically query Pinecone with the user's preference embedding + apply metadata filters
• Diversify results using clustering + similarity thresholds to avoid 20 near-identical black tops
• Delivered as: 'For You' feed, 'Because you liked X', 'New in your style' (items scraped in last 24h prioritized)

React + Server-Driven Pagination

• Consistent product card component + skeleton loaders across all views
• Cursor-based pagination preferred over offset — prevents duplicates when inventory changes mid-scroll
• API returns items + nextCursor; frontend requests next page on scroll or 'Load more'
• Cached results per query so back/forward navigation is instant
• Debounced query input to avoid firing on every keystroke
• Optimistic UI for likes/saves with rollback on failure
• Search filters wired directly to backend metadata filters — no expensive client-side re-processing

Why This Mattered

The catalog is large and constantly changing — stock and pricing update daily. Stable pagination + caching + consistent ranking prevented duplicated items across pages, missing items when new inventory arrived mid-scroll, and jarring reshuffles when toggling filters.