Web Search Plugin

The Web Search plugin is a standalone, lightweight Go binary designed for the BinaryCMS ecosystem. It acts as a universal web knowledge retrieval engine, providing other plugins (like the Local AI Engine) with the ability to perform web searches and read full webpage contents without relying on bulky external dependencies like Python, Docker, or Redis.

1. Multi-Provider Search Engine The plugin features a robust, interchangeable provider system allowing you to use free or premium search APIs.

DuckDuckGo HTML (Default): A native HTML scraper using goquery. It performs searches directly against DuckDuckGo’s lite endpoint. No API key required, no rate limits, completely free. Premium API Integrations: Drop-in support for high-quality, structured JSON APIs. Brave Search API: High-quality index with a generous free tier (2,000 queries/month). Google Custom Search: Access to Google’s massive index (requires API Key CSE ID). Bing Web Search: Microsoft’s search index (requires Azure API Key). Self-Hosted Options: SearXNG: Connect to a self-hosted SearXNG meta-search instance by providing its local URL. Auto-Fallback Resilience: If a premium API provider fails (e.g., rate limit reached or network error), the engine automatically and silently falls back to the free DuckDuckGo scraper, ensuring your application never loses web access.

1. Website Content Extractor (“Deep Reading”) Search snippets are often insufficient for AI comprehension. The plugin includes a powerful URL reading module:

Intelligent HTML Parsing: Uses goquery to download a webpage and algorithmically identify the main article/content area.

Context Optimization: Cleans up messy whitespace and truncates massive pages to ~10,000 characters to prevent overloading LLM context windows.

1. Local AI Engine Smart Proxy Integration The Web Search plugin seamlessly integrates with the local_ai_engine via a transparent reverse proxy (running on port 8087), requiring absolutely zero modifications to your frontend AI Chat interface.

Smart Query Detection (Auto Mode): The proxy intercepts chat prompts and analyzes them. If it detects a factual question (e.g., “what is the latest news”, “price of bitcoin”, or ends in “?”), it automatically performs a web search in the background. Top-Result Deep Reading: When a search is triggered, the proxy doesn’t just pass 5 short snippets to the LLM. It automatically fetches and reads the full text of the #1 search result, providing deep, factual context. Direct URL Support: If a user pastes a URL directly into the chat (e.g., “Summarize https://…”), the proxy detects the URL, triggers the Website Content Extractor, and injects the article’s text into the LLM’s system prompt. Citation Formatting: The injected prompt is specifically formatted to instruct the LLM to use the provided real-time data and cite its sources.

1. Admin Dashboard & Metrics A dedicated user interface within BinaryCMS to manage your search infrastructure.

Live Testing Console: Test search queries or read URLs directly from the dashboard to verify results without needing to use the AI Chat. Search History Log: A database-backed log showing recent queries, which provider served them, how many results were found, and the network latency. Global Metrics: Tracks total queries processed over time. Provider Configuration: A UI to seamlessly switch active providers and manage API keys securely.

1. Developer JSON API Because it is a standalone BinaryCMS plugin, its capabilities are exposed internally via a clean JSON API, meaning any future plugins you build (like a Content Generator, SEO Analyzer, or Market Research tool) can use it.

GET /api/plugin/web_search/search?q=your query GET /api/plugin/web_search/read?url=https://example.com