We have been having fun with our new crawler. 0pn's working name was the Skeleton Crawler and we have decided to stick with that name. Crawling has been commencing for about 1 week now and we will do a lump Upload to the servers, if we don't run into any pesky Bugs and gremlins, in about two weeks time.
You ask why it was called the Skeleton Crawler? We had been getting bogged down in getting as many results under as many search words and phrases as possible. The Skeleton Crawler is much different it actually looks for the best results under a certain word/phrase if the SCORE is greater than those already in the database then chances are it would be overwritten. This allows for our crawlers to work much faster and create BETTER results for MORE of the world wide web instead of getting bogged down on the POPULAR words.
We will be releasing more information on this crawler as we come out of the beta testing period. While we love to protect our trade secrets we also believe it is very important to be open on how our technologies work so that everyone can be on an even playing field.
