# Product enrichment and NEAH price/SKU crawl

Two separate CLIs:

1. **Price/SKU crawl** (authenticated store pages) → writes a CSV with `sku`, `price`, `currency`
2. **Enrichment** (public web search + OpenAI) → reads that CSV and partial-upserts Elasticsearch, including file-known price/SKU

**Input format: CSV only** (export/save Excel as CSV manually).

## Setup

```bash
npm install
cp .env.example .env
# fill ELASTIC_*, OPENAI_API_KEY, ENRICH_WEBSITE_ID
# for NEAH crawl: set NEAH_COOKIE or use --cookie-file
```

## 1) Crawl price + SKU + page metadata from NE Animal Health

Save products as CSV with headers:

```csv
#,Product Name,Product URL,Category
```

Authenticated crawl extracts: `sku`, `price`, `currency`, `brand`, `imageUrl`, `productImages`, `tags`, `category`, plus WooCommerce attributes (`customAttributes` like Weight / Dimensions, and structured `weight` / `weightUnit` / `dimensions`).

Enrichment also extracts attributes from web research when stated. ES docs get:

- `additionalFields.customAttributes`: `[{ "name": "Weight", "value": "2 lb" }, …]`
- `customFields`: `{ "Weight": "2 lb", "Dimensions": "6 × 6 × 6 in" }`
- optional top-level `weight`, `weightUnit`, `dimensions`

## Enrichment mode (default: OpenAI web search research + write)

OpenAI **searches the web itself** (Responses API `web_search` tool), then writes a full researched product description into structured JSON.

```env
ENRICH_MODE=openai_web
ENRICH_MODEL=gpt-4o
SEARCH_PROVIDER=openai
OPENAI_API_KEY=...
ENRICH_DESC_FALLBACK=none
```

Flow per product:

1. OpenAI `web_search` researches manufacturer/distributor pages  
2. OpenAI converts research notes → structured enrichment JSON  
3. Upsert to Elasticsearch immediately + checkpoint  

No Bright Data needed. Optional: `ENRICH_MODE=brightdata` for the older SERP+fetch path.

```bash
npm run crawl:neah-price-sku -- --input ./neanimalhealth_products.csv --concurrency 2 --force
```

## 2) Validate enrichment for 2 products (JSON file, no ES write)

Clear sticky empty OpenAI cache first if you previously got empty descriptions:

```bash
rm -rf .cache/product-enrichment
```

```bash
npm run enrich -- --input ./neanimalhealth_products.with-price-sku.csv --limit 2 --force --preview-json ./enrich-preview-2.json
```

This writes `./enrich-preview-2.json` with `validation` + mapped `document` for each product. Review before full ES run.

## 3) Enrich into Elasticsearch

```bash
npm run enrich -- --input ./neanimalhealth_products.with-price-sku.csv --concurrency 5 --force
```

Invalid docs (e.g. missing description) are **skipped** for ES insert and logged as `validation_failed_skip_es`.

### Price / SKU rules

| Source | Written to ES? |
|---|---|
| `sku` / `price` / `currency` columns in the input CSV | Yes |
| OpenAI or public web extraction | Never |

### Sample CSV

```csv
product name,product url,category,sku,price,currency
Example Product,https://example.com/product,Example Category,SKU-1,19.99,USD
```

## Tests

```bash
npm run test:enrich
```

## Run with PM2 (background)

Deploy [`ecosystem.product-enrich.config.cjs`](../../ecosystem.product-enrich.config.cjs), ensure `.env` and CSVs are in place, then:

```bash
# price/SKU crawl (writes CSV, then exits)
pm2 start ecosystem.product-enrich.config.cjs --only neah-price-sku-crawl
pm2 logs neah-price-sku-crawl

# enrichment (writes to ES, then exits) — run after crawl output exists
pm2 start ecosystem.product-enrich.config.cjs --only product-enrich
pm2 logs product-enrich

pm2 status
pm2 stop product-enrich
pm2 delete product-enrich
```

`autorestart` is off so the process does not loop after finishing.

**Per-product ES writes:** each validated product is upserted to Elasticsearch immediately and checkpointed. If Bright Data credits run out mid-run, already-finished products stay in ES. Re-run without `--force` to resume remaining rows (skips `status: completed`).

## Dependencies

`zod`, `csv-parse`, `p-limit` (plus existing `openai`, `@elastic/elasticsearch`, `cheerio`, etc.).