Deliverable #1 : Experimentation

The purpose of this deliverable is to experiment with the current news feed features in the search engine.

Experimentation :.

I added few news sources(rss and html type) to the media sources and ran the news_updater by reducing its uopdate time to 60 seconds. If after running the news_updater, the index file was being created under feeds then the test was pass otherwise fail.

No of news feed sources Time(in mins) Created(Y/N)
10 1 Y
25 1 Y
50 1 Y
70 1 Y
80 1 Y
1001 Y
1251 Y
1501 Y
1601 Y
1701 Y
2001 Y
2501 Y
2801 Y

>

Conclusion :.

After adding few news sources, the news_updater was increasingly taking more time to update. After almost 275 news sources, it took more than 5-6 minutes to build the index file and after 350 news sources it took almost 10 minutes to generate the index file. As the no of news sources goes on increasing, there has been increase in the time it takes to build the index shard. In order to completely break the working, I will work on building the automated test framework instead of doing the manually testing.

Following chart plots the no of news sources and time taken to generate the index file for the same.

Referring to this chart, we can roughly make an estimate that in an hour single machine news updater can handle around 3-4 k news sources.