The new spider is working great, almost to the point of TOO great. The databases that have been created are so large that we now have to decide how to deliver them. Our current host is very limited on the space front but has been by far the best quality, speed and uptime host we have found. So the idea of changing to another location is just out of the question. We looked into upgrading but the expense is currently too great on that host.
So we are in limbo tryhing to figure out what to do with these datbases--stay tuned for the answers.
USER AGENT: Some of you may have come here because you have seen in your host logs 0pn Skeleton Crawler. This was just a recent update that wer should have done many moons ago on our other various crawlers.
The crawler is currently taking note of links on the page it scans it is not following these links our intial pass is all about front page of the domain names.
