Search @ Fisheye : 2.5 million queries per day and rocketing!

A little more than two years ago we created our first search engine for News called Flamingo. We later created search engines for social media too. Inorder to provide our clients with fresh results as they appear on the web, we index the data our web crawlers fetch into two types of indexes. One of them, the "real time index" always contains data for the past 24 hours to power up realtime search and the other "monthly index" contains data for the past month. The latter can grow into hundreds of gigabytes.
Our web crawlers create these search indexes distributed across a cluster of machines. I will save writing about how our web crawlers work to another post.

Sharing some of our Numbers

We are growing at Fisheye Analytics. We are seeing more data and providing more intelligence to our clients. Some of our numbers are shared below.

1 million News + 1.5 million Blog posts + 21 million Tweets + 2 million Facebook public posts and more created are extracted and analyzed every day

25 TB of data extracted every month from the web (news & social media). For comparison, this could be roughly equal to the size of all 4 billion tweets Twitter receives over a month currently! (*)

We are HIRING

2010 was a year of big achievements for the engineering team at Fisheye. Last year, we scaled our technology that could serve 10 times more clients than we could in 2009. We also developed a new tool, Media Lens, which revolutionized the way clients monitored the media that was totally built on top of an open-source search server instead of the traditional way of reading from database. This year, we are expanding our tech team to build exciting tools that make a difference in the media industry.

How a day looks like in our server farms

Transcript of the above presentation :

How Stuff Works - Technology @ Fisheye analytics : How a day looks like in our server farm
                                   by Ashwin Reddy Gayam

An experiment that worked

When a traditional way of doing analytics with a DBMS didn’t offer good performance for our large loads, we looked to Sphinx, a full text search engine to do analytics, something which Sphinx is not intended to be used for. We experimented. We married MySQL with Sphinx to offer great performance in analytics crunching hundreds of millions of client data records. We did something unusual for serious production use but succeeded. Best part is that all the software used is free and open source. Keep reading for a detailed story.

Technologies used at Fisheye Analytics

What could be a better way to start a "Hello World" post in this new engineering blog than writing about the various technologies we use at Fisheye Analytics? Most of the technologies and tools we use are free and open source. I will give a short listing of some of the important ones here.