menuvision

Build beautiful HTML photo menus from restaurant URLs, PDFs, or photos using Gemini Vision and AI image generation

# MenuVision - Restaurant Menu Builder

Build a beautiful HTML photo menu for any restaurant from URLs, PDFs, or photos.

## When to Use

When the user wants to create a digital menu for a restaurant. Triggers: "build a menu", "create restaurant menu", "menu from PDF", "menu from photos", "digital menu", "menuvision".

## Quick Start

```
1. Extract:  URL/PDF/photo  →  menu_data.json     (Gemini Vision)
2. Generate: menu_data.json →  images/*.jpg        (Gemini Image)
3. Build:    menu_data.json + images → Menu.html   (CSS/JS inline, images relative)
```

### Example usage (ask the AI):
- "Build a menu for https://www.shoyu.at/menus"
- "Create a photo menu from this PDF" (attach file)
- "Make a digital menu from these photos of a restaurant menu"

## Pipeline Components

The AI agent creates these scripts:

| Script | Purpose |
|--------|---------|
| `extract_menu.py` | Extract menu data from URL/PDF/photo → structured JSON |
| `generate_images.py` | Generate food photos via Gemini Image |
| `build_menu.py` | Build HTML menu from JSON + images (CSS/JS inline, images as relative paths) |
| `publish_menu.py` | (Optional) Publish HTML to GitHub Pages |

---

## DATA CONTRACT (Critical)

All three pipeline stages share this exact JSON schema. The AI agent MUST use these field names — any deviation breaks the pipeline.

### menu_data.json Schema

```json
{
  "restaurant": {
    "name": "Restaurant Name (if visible)",
    "cuisine": "cuisine type (Chinese, Indian, Austrian, Japanese, etc.)",
    "tagline": "any subtitle or tagline"
  },
  "sections": [
    {
      "title": "Section Name (in primary language)",
      "title_secondary": "Section name in secondary language (if present, else empty string)",
      "category": "food or drink",
      "note": "Any section note (e.g. 'served with rice', 'Mon-Fri 11-15h')",
      "items": [
        {
          "code": "M1",
          "name": "Dish Name (primary language)",
          "name_secondary": "Name in secondary language (if present)",
          "description": "Brief description (primary language)",
          "description_secondary": "Description in secondary language (if present)",
          "price": "12,90",
          "price_prefix": "",
          "allergens": "A C F",
          "dietary": ["vegan", "spicy"],
          "variants": []
        }
      ]
    }
  ],
  "allergen_legend": {
    "A": "Gluten",
    "B": "Crustaceans"
  },
  "metadata": {
    "languages": ["German", "English"],
    "currency": "EUR"
  }
}
```

### Field Reference

| Field | Type | Required | Notes |
|-------|------|----------|-------|
| `restaurant.name` | string | Yes | Display name in HTML header |
| `restaurant.cuisine` | string | Yes | Passed to `build_food_prompt()` as cuisine context |
| `restaurant.tagline` | string | No | Subtitle line in HTML header |
| `sections[].title` | string | Yes | Section heading in primary language |
| `sections[].title_secondary` | string | No | Section heading in secondary language |
| `sections[].category` | `"food"` or `"drink"` | Yes | Drives food grid vs drink list layout. Only `"food"` items get generated images. |
| `sections[].note` | string | No | Section-level note (e.g. "served with rice", "Mon-Fri 11-15h") |
| `items[].code` | string | Yes | Unique per item. Links to image filename. Use existing codes (M1, K2) or generate (A1, A2) |
| `items[].name` | string | Yes | Primary language. For CJK menus, this is the CJK name |
| `items[].name_secondary` | string | No | Secondary language. For CJK menus, this is the English/Latin name |
| `items[].description` | string | No | Brief description. Fed to `build_food_prompt()` for image generation |
| `items[].description_secondary` | string | No | Description in secondary language |
| `items[].price` | string | Yes | Preserve original format ("12,90" not "12.90") |
| `items[].price_prefix` | string | No | e.g. "ab" (starting from), "ca." |
| `items[].variants` | array | No | `[{"label": "6 Stk", "price": "8,90"}, ...]` — set main price to smallest variant |
| `items[].allergens` | string | No | Space-separated codes exactly as printed: "A C F" |
| `items[].dietary` | array | No | `["vegan", "vegetarian", "spicy", "gluten-free", "halal", "kosher"]` |
| `allergen_legend` | object | No | Map of allergen codes to display names: `{"A": "Gluten", ...}` |
| `metadata.currency` | string | Yes | ISO code: "EUR", "USD", "JPY", "CNY", "THB", etc. |
| `metadata.languages` | array | No | Languages detected in the menu: `["German", "English"]` |

---

## EXTRACTION PROMPT

Send this exact prompt to Gemini. It defines the schema AND the extraction rules. Do not paraphrase it.

```
You are a restaurant menu data extractor. Analyze this menu content and extract ALL items into structured JSON.

Return this exact JSON structure:
{
  "restaurant": {
    "name": "Restaurant Name (if visible)",
    "cuisine": "cuisine type (Chinese, Indian, Austrian, Japanese, etc.)",
    "tagline": "any subtitle or tagline"
  },
  "sections": [
    {
      "title": "Section Name (in primary language)",
      "title_secondary": "Section name in secondary language (if present, else empty string)",
      "category": "food or drink",
      "note": "Any section note (e.g. 'served with rice', 'Mon-Fri 11-15h')",
      "items": [
        {
          "code": "M1",
          "name": "Dish Name (primary language)",
          "name_secondary": "Name in secondary language (if present)",
          "description": "Brief description (primary language)",
          "description_secondary": "Description in secondary language (if present)",
          "price": "12,90",
          "price_prefix": "",
          "allergens": "A C F",
          "dietary": ["vegan", "spicy"],
          "variants": []
        }
      ]
    }
  ],
  "allergen_legend": {
    "A": "Gluten",
    "B": "Crustaceans"
  },
  "metadata": {
    "languages": ["German", "English"],
    "currency": "EUR"
  }
}

CRITICAL RULES:
1. Extract EVERY item. Do not skip ANY dish, drink, or menu entry.
2. Preserve original item codes/numbers if present (M1, K2, S3, etc.). If none exist, generate sequential codes per section (e.g. A1, A2 for appetizers, M1, M2 for mains).
3. Extract prices EXACTLY as written (preserve comma/period format).
4. If an item has a price prefix like "ab" (starting from), capture it in "price_prefix".
5. If an item has multiple size/quantity variants (e.g. 6 Stk / 12 Stk / 18 Stk at different prices), use the "variants" array:
   [{"label": "6 Stk", "price": "8,90"}, {"label": "12 Stk", "price": "15,90"}]
   In this case, set the main "price" to the smallest variant's price.
6. Capture allergen codes exactly as shown (letters, numbers, or symbols).
7. If an allergen legend is visible anywhere, include it in "allergen_legend".
8. Identify dietary flags from descriptions/icons: vegan, vegetarian, spicy, gluten-free, halal, kosher.
9. If the menu is bilingual, capture BOTH languages. Put the primary/dominant language in name/description and the secondary in name_secondary/description_secondary.
10. For set menus or lunch specials with a fixed price covering multiple choices, create a section with note explaining the format, and list each choice as an item.
11. Classify each section as "food" or "drink".
12. For drinks, still extract name, price, and any size variants.

Return ONLY valid JSON. No markdown fences, no explanatory text.
```

### Vision Prompt Variant
For image-based inputs (screenshots, PDF pages, photos), prepend a context line before the base prompt:

```python
EXTRACTION_PROMPT_VISION = (
    "You are a restaurant menu data extractor. "
    "This is a photo/scan of a restaurant menu page.\n\n"
    "Return this exact JSON structure:"
    + EXTRACTION_PROMPT.split("Return this exact JSON structure:")[1]
)
```

Then each input type adds its own prefix:

| Input Type | Prefix prepended to `EXTRACTION_PROMPT_VISION` |
|---|---|
| Screenshot | `"This is a screenshot of a restaurant menu webpage at {url}. Extract A