AI Vision Diagnosis

Instructions

Pipeline Architecture

Implement the diagnosis flow as a multi-step pipeline where each step enriches the output of the previous step:

Image Upload → Preprocessing → Vision API Analysis → Context Injection →
Diagnosis Card Generation → Treatment Recommendations → User Response

Step 1: Image Upload and Preprocessing

1. Accept images via camera capture or file upload
2. Validate image quality:

• Minimum resolution: 640×480 pixels
• Reject blurry images (if client-side analysis available, check Laplacian variance)
• Warn if image is overexposed or underexposed
• Maximum file size: 20MB (Claude Vision limit)

1. Resize for API submission:

• Resize to max 1568px on the longest edge (Claude Vision optimal)
• Convert to JPEG at quality 85 if PNG is unnecessarily large
• Preserve EXIF orientation; strip other EXIF data for privacy

1. Collect metadata at capture time:

• GPS coordinates (if user permits)
• Capture timestamp
• Plant type (user selection or prior context)
• Affected plant part (leaf, stem, fruit, root, whole plant)

Step 2: Vision API Analysis

1. Construct the Vision API request:

   message = client.messages.create(
       model="claude-sonnet-4-20250514",
       max_tokens=2048,
       messages=[{
           "role": "user",
           "content": [
               {
                   "type": "image",
                   "source": {
                       "type": "base64",
                       "media_type": "image/jpeg",
                       "data": base64_image
                   }
               },
               {
                   "type": "text",
                   "text": structured_prompt
               }
           ]
       }]
   )

1. Structured prompt for plant diagnosis:

   Analyze this plant image for signs of disease, pest damage, or nutrient deficiency.

   Plant: {plant_type}
   Part shown: {plant_part}
   Location: {region}
   Date: {capture_date}

   Provide your analysis as JSON:
   {
     "observations": [list of visual observations],
     "primary_diagnosis": {
       "condition": "name",
       "confidence": "high|medium|low",
       "visual_evidence": [specific features that support this diagnosis]
     },
     "differential_diagnoses": [
       { "condition": "name", "confidence": "high|medium|low", "distinguishing_features": "what to look for" }
     ],
     "image_quality_notes": "any issues affecting diagnosis accuracy",
     "additional_images_needed": "what other angles/parts would help"
   }

1. Parse the Vision API response as structured JSON
2. Validate confidence levels: if primary diagnosis confidence is “low”, flag for human review

Step 3: Context Injection

After receiving the Vision API analysis, enrich the diagnosis with contextual data:

1. Weather context:

• Fetch 14-day weather history for the location
• Run disease risk models (see weather-disease-modeling skill)
• If weather conditions support the diagnosed disease, increase confidence
• If weather conditions contradict the diagnosis, flag for review

1. Regional pest/disease database:

• Query regional pest reports (e.g., state extension service alerts)
• Check if the diagnosed condition is currently active in the region
• Include regional prevalence data in the diagnosis card

1. Seasonal context:

• Map the diagnosis against typical timing (e.g., late blight peaks in humid summer, not in dry winter)
• Flag off-season diagnoses for additional scrutiny

1. Plant growth stage:

• Determine approximate growth stage from GDD accumulation or planting date
• Certain diseases are stage-specific (e.g., blossom end rot during fruiting)

Step 4: Diagnosis Card Generation

Produce a structured diagnosis card for the user:

{
  "diagnosis_id": "uuid",
  "timestamp": "ISO-8601",
  "plant": { "type": "tomato", "variety": "if known", "growth_stage": "fruiting" },
  "primary_diagnosis": {
    "condition": "Early Blight (Alternaria solani)",
    "confidence": "high",
    "confidence_factors": [
      "Visual: concentric ring pattern on lower leaves",
      "Weather: 10 days of warm humid conditions",
      "Regional: early blight reported in county this month"
    ]
  },
  "differential_diagnoses": [...],
  "severity": "moderate",
  "image_url": "reference to uploaded image",
  "treatment_plan": { ... }
}

Step 5: Treatment Recommendations

1. Generate treatment plan based on the confirmed diagnosis:

• Immediate actions: what to do today
• Cultural controls: pruning, sanitation, spacing changes
• Organic treatments: copper fungicide, neem oil, biological controls
• Conventional treatments: specific fungicide/pesticide names with application rates
• Prevention: steps to prevent recurrence

1. Treatment safety:

• Always include pre-harvest interval (PHI) for any chemical treatment
• Note protective equipment required
• Flag treatments inappropriate for organic gardens

1. Follow-up schedule:

• When to re-inspect
• When to re-apply treatment
• What improvement to expect and by when

Error Handling and Safety

1. Never present AI diagnosis as definitive — always frame as “likely” or “consistent with”
2. Confidence calibration: use three levels and explain what each means to the user:

• High: visual evidence strongly matches, supported by context
• Medium: visual evidence is suggestive but not conclusive
• Low: multiple conditions could explain the symptoms

1. Escalation path: provide contact information for local extension services when confidence is low
2. Image storage: retain images only with user consent; delete after diagnosis if not consented

Inputs Required

• Plant image (JPEG/PNG, minimum 640×480)
• Plant type and affected part
• Geographic location (for weather and regional data)
• Planting date or growth stage (if known)
• User’s garden type (organic, conventional) for treatment filtering
• Weather data API access for context enrichment

Output Format

Diagnosis Response

{
  "diagnosis_card": {
    "diagnosis_id": "string",
    "primary_diagnosis": {
      "condition": "string",
      "confidence": "high|medium|low",
      "confidence_factors": ["string"],
      "severity": "mild|moderate|severe"
    },
    "differential_diagnoses": [
      { "condition": "string", "likelihood": "string" }
    ]
  },
  "treatment_plan": {
    "immediate_actions": ["string"],
    "cultural_controls": ["string"],
    "organic_treatments": [{ "product": "string", "application": "string", "phi_days": 0 }],
    "conventional_treatments": [{ "product": "string", "application": "string", "phi_days": 7 }],
    "prevention": ["string"],
    "followup_date": "ISO-8601"
  },
  "context_data": {
    "weather_risk_score": 45,
    "regional_prevalence": "active in area",
    "seasonal_fit": "typical timing"
  },
  "disclaimers": [
    "AI-assisted diagnosis — confirm with local extension service for critical decisions"
  ]
}

Anti-Patterns

• Skipping the context injection step — Vision API analysis alone is insufficient; weather and regional data dramatically improve accuracy
• Presenting AI diagnosis as definitive — always include confidence levels and disclaimers
• Using a single-shot prompt — the multi-step pipeline with structured JSON output produces far more reliable results than asking for everything in one prompt
• Ignoring image quality — garbage in, garbage out; validate and reject unusable images upfront
• Hardcoding treatment recommendations — treatments vary by region, organic certification status, and product availability; keep them data-driven
• Storing user images without consent — garden photos may contain location metadata and property views; handle as PII
• Recommending chemical treatments without safety data — always include PHI, PPE requirements, and organic compatibility notes