Queen Detection - Technical Documentation

🎯 Overview

AI-powered queen bee detection system using machine learning models to identify queen bees in uploaded frame photos. Currently uses Clarifai API for detection with plans to develop internal model for improved accuracy. Part of automated frame analysis pipeline triggered on photo upload.

🏗️ Architecture

Components

DetectionOverlay: Canvas component rendering detected queens as bounding boxes
QueenIndicator: UI component showing queen presence status
FrameAnalysisPanel: Side panel displaying detection counts and confidence

Services

image-splitter: Orchestrates detection pipeline, stores results
Clarifai API: External ML service for queen detection (current implementation)
models-bee-detector: Internal model service (supports queens but low accuracy ~60%)
swarm-api: Stores queen presence metadata linked to frames

📋 Technical Specifications

Detection Pipeline

Frame photo uploaded → image-splitter
Create detection job (type: detect_queens)
Job processor invokes detection:
   a. Primary: Clarifai API call
   b. Fallback: models-bee-detector (if enabled)
Parse detection response (boxes + confidence)
Store detections in MySQL
Publish Redis event
Update queen presence in swarm-api

GraphQL API

type QueenDetection {
  id: ID!
  frameSideId: ID!
  bbox: BoundingBox!
  confidence: Float!
  verifiedByUser: Boolean
  createdAt: DateTime!
}

type BoundingBox {
  x: Int!
  y: Int!
  width: Int!
  height: Int!
}

type FrameSide {
  id: ID!
  url: String!
  queenDetections: [QueenDetection!]
  hasQueen: Boolean
}

query frameSide($id: ID!) {
  frameSide(id: $id) {
    id
    queenDetections {
      id
      bbox {
        x
        y
        width
        height
      }
      confidence
    }
  }
}

mutation verifyQueenDetection($detectionId: ID!, $verified: Boolean!) {
  verifyQueenDetection(detectionId: $detectionId, verified: $verified) {
    id
    verifiedByUser
  }
}

🔧 Implementation Details

Frontend

Framework: React with TypeScript
Rendering: HTML5 Canvas for detection overlay
Visualization: Yellow bounding box with confidence score label
Interaction: Click to verify/reject detection
State: Apollo cache for detection data

Backend (image-splitter)

Language: TypeScript/Node.js
Detection Service: Clarifai gRPC client
Fallback: Internal models-bee-detector REST API
Job Processing: Async queue with retry logic
Storage: Detection results stored as JSON bbox data

Clarifai Integration

async function detectQueens(imageUrl: string): Promise<Detection[]> {
  const stub = ClarifaiStub.grpc();
  
  const metadata = new grpc.Metadata();
  metadata.set("authorization", `Key ${CLARIFAI_API_KEY}`);

  const request = {
    userAppId: {
      userId: CLARIFAI_USER_ID,
      appId: CLARIFAI_APP_ID
    },
    inputs: [{
      data: {
        image: { url: imageUrl }
      }
    }]
  };

  const response = await stub.PostModelOutputs(request, metadata);
  
  return response.outputs[0].data.regions.map(region => ({
    bbox: {
      x: region.regionInfo.boundingBox.leftCol * imageWidth,
      y: region.regionInfo.boundingBox.topRow * imageHeight,
      width: region.regionInfo.boundingBox.width * imageWidth,
      height: region.regionInfo.boundingBox.height * imageHeight
    },
    confidence: region.value,
    type: 'queen'
  }));
}

Internal Model (models-bee-detector)

Framework: YOLOv5 (Ultralytics)
Weights: Matt Nudi's honey bee detection model
Dataset: Roboflow universe dataset
Accuracy: ~60% for queens (low precision)
Classes: worker, drone, queen, varroa
Issues: High false positive rate for queens

⚙️ Configuration

Environment Variables

CLARIFAI_API_KEY=xxx
CLARIFAI_USER_ID=clarifai
CLARIFAI_APP_ID=main

MODELS_BEE_DETECTOR_URL=http://models-bee-detector:8700
MODELS_BEE_DETECTOR_ENABLED=false

QUEEN_DETECTION_CONFIDENCE_THRESHOLD=0.6

🧪 Testing

Unit Tests

Location: /test/detection/queens.test.ts
Coverage: Detection parsing, bbox calculation, confidence filtering
Mocks: Clarifai API responses

Integration Tests

Location: /test/integration/queen-detection-flow.test.ts
Tests:
- Full detection pipeline (upload → detect → store → query)
- Clarifai API integration
- Fallback to internal model
- User verification workflow

Manual Testing

Test images available in:

/test/fixtures/queens/ - Known queen images
Sample images from dataset with verified queens

📊 Performance Considerations

Metrics

Clarifai API latency: 2-5 seconds per image
Detection confidence threshold: 0.6 (configurable)
False positive rate: ~30% (Clarifai), ~50% (internal model)
True positive rate: ~70% (Clarifai), ~60% (internal model)

Bottlenecks

Clarifai API rate limit: 10 requests/second
External API dependency (network latency)
Large images require longer processing
Cost: $0.0012 per Clarifai prediction

Optimization Plans

Train internal model with higher accuracy
Implement model ensemble (multiple models voting)
Cache results for identical images
Batch processing for multiple frames

🚫 Technical Limitations

Current Constraints

Accuracy: 60-70% detection rate (not production-ready for critical decisions)
False Positives: High rate requires manual verification
Marked Queens: Cannot detect marked queens differently (paint spots confuse model)
Image Quality: Poor lighting or focus significantly reduces accuracy
Clarifai Dependency: External API dependency for best results
Cost: Per-detection cost for Clarifai API
No Tracking: Cannot track same queen across multiple photos

Known Issues

Model confuses large worker bees with queens
Marked queens (with paint dots) often missed
Multiple false positives per image common
No confidence calibration (scores not well-calibrated)
Dataset bias: Trained mostly on light-colored queens

📚 Development Resources

GitHub Repositories

image-splitter - Detection orchestration
models-bee-detector - Internal model
web-app - Detection visualization

Key Files

Detection job: /src/jobs/detect-queens.ts
Clarifai client: /src/integrations/clarifai.ts
Frontend overlay: /src/page/hive/frameSide/DetectionOverlay.tsx

Model Training Resources

Dataset: Roboflow Honey Bee Detection
YOLOv5 weights: Matt Nudi's model
Future: Custom dataset with more queen examples needed

💬 Technical Notes

Current accuracy (~60-70%) not reliable for automated alerts without user verification
Clarifai used because internal model training incomplete
Future plan: Train internal model with larger queen-specific dataset
Consider ensemble approach: multiple models voting for higher confidence
User verification feedback could be used to retrain model
Marked queens (painted) need separate detection class
Queen detection more valuable when combined with queen management timeline
False positives less critical than false negatives (missing queen is worse)

Improvement Roadmap

Collect larger queen-specific dataset (1000+ labeled queens)
Train internal YOLOv8 model with transfer learning
Implement model ensemble (Clarifai + internal + voting)
Add user feedback loop for model improvement
Detect marked vs unmarked queens separately
Implement confidence calibration
Add queen tracking across multiple photos

Last Updated: December 5, 2025

🎯 Overview​

🏗️ Architecture​

Components​

Services​

📋 Technical Specifications​

Detection Pipeline​

GraphQL API​

🔧 Implementation Details​

Frontend​

Backend (image-splitter)​

Clarifai Integration​

Internal Model (models-bee-detector)​

⚙️ Configuration​

Environment Variables​

🧪 Testing​

Unit Tests​

Integration Tests​

Manual Testing​

📊 Performance Considerations​

Metrics​

Bottlenecks​

Optimization Plans​

🚫 Technical Limitations​

Current Constraints​

Known Issues​

🔗 Related Documentation​

📚 Development Resources​

GitHub Repositories​

Key Files​

Model Training Resources​

💬 Technical Notes​

Improvement Roadmap​