Search Engines play a critical role in information retrieval, processing millions of queries daily.
They not only provide search results but also log various analytics to monitor performance and user behavior
Understanding how search engines record, store analytics in an efficient way.
Focus on search engines like Apache Solr, MediaWiki, and insights from Google, Bing, and Wikipedia
Search Engine Analytics and Storage Techniques
Search engines track key analytics such as:
Query Execution Time: How long a query takes to complete.
Hits and Click-through Rates (CTR): Tracks the number of results clicked.
Search Trends: Monitoring popular search terms.
Storage Techniques:
Logs: Analytics are logged instead of stored in a database to save space.
Aggregated Data: Summarizing analytics like total clicks to minimize storage load.
Apache Solr – Efficient Analytics Storage
Apache Solr stores query metrics like QTime, numFound, and status in logs instead of a database.
Advantages:
Logs save storage overhead by not storing every search query result in a database.
Example: A typical Solr instance serving 10,000 queries/day generates logs 10-50 MB/day, which is more efficient compared to database storage that would consume ~150 MB/day.
Indexed data:
Logs save storage overhead by not storing every search query result in a database.
Solr indexes documents with fields (e.g., title, release_year) to allow faster querying, which indirectly improves performance by reducing response times.
This log shows a successful search query for Sci-Fi movies released from 2000 onwards.
The size of the response (486 bytes) indicates how much data was returned.
MediaWiki – Handling Large Analytical Workloads
MediaWiki uses MySQL to store various analytics such as:
Page Views: Recorded in the hit_counter table.
User Contributions: Stored in the revision table.
Efficient Storage:
Incremental Counters: Rather than logging each page view, MediaWiki increments counters in the database.
Indexing: Tables like page and revision are indexed for faster retrieval.
Real-world Example: MediaWiki handles 5,000 daily edits and efficiently stores page hits without overwhelming the database.
MediaWiki – Handling Large Analytical Workloads
This is how the values are stored in the hit_counter table by incrementing the counter values.
Performance Benefits of Logging vs. Database Storage
Logging analytics in search engines like Solr prevents database bloat
For 10,000 queries/day, logs take 50% less storage compared to relational databases.
Improvement: Log-based systems reduce query times by 20-30% compared to systems that query analytics from a database directly.
Data Aggregation: In MediaWiki, counters are used to aggregate data, improving performance when dealing with large traffic spikes.
For 100,000 page views/day, aggregation reduces database storage usage by 60% compared to row-based logging.
Insights from Google, Bing, and Wikipedia
Google and Bing are closed systems, but insights from data breaches show:
They track query logs, Click-through Rates (CTR), dwell time, and session data.
Performance Improvement: By caching frequently used queries, they reduce search times by 40%.
Wikipedia (powered by MediaWiki) stores page view analytics in incremental counters rather than logging each event. This allows them to handle ~200 million daily hits without overwhelming their database.
Applying These Techniques to Yioop
Key Takeaways for Yioop’s Analytics Improvement:
Logging Instead of Database: For most analytics, use logs instead of databases to reduce storage load.
Data Aggregation: Implement incremental counters like MediaWiki to aggregate hits, reducing query load.
Indexing: Use Lucene-like indexing for user-centric data, improving performance by 25-30% under high analytical workloads.
Conclusion
Efficient Storage of Analytics: Both MediaWiki and Solr demonstrate how log-based systems and aggregated data improve performance.
By adopting these techniques, Yioop can handle large-scale analytics data without performance degradation.
Next steps: Implement log-based storage, data aggregation, and indexing in Yioop to improve performance in handling large analytical workloads.
