|
Essential
Tools
Link
Popularity Checking Tool
Link
Building Software
Axandra
Search Ranking Study
Free
Trial
WordTracker
AdWords
Select Report
Web
Position Gold
Search
Engine School
|
|
SEO (SEARCH ENGINE OPTIMIZATION)ARTICLES
Crawl
the Web With Grubby!
You Can Crawl the Web from Your Own
PC!
by Mike Banks Valentine
Oh, for the simplicity of the good old days of 1995!
Where
all a web site owner had to do was put important
keywords
about her web site in the metatags of the HTML code
of the
front page of her site with a clear description
and title
to rank well in the search engines and make it easy
for web
searchers to find her.
What made it become so difficult to gain and maintain
a bit
of internet visibility? It got crowded on the web.
Crowds
made it more difficult to get noticed - so instead
of simply
including proper metatags, web site owners had to
begin to
make themselves stand out to get noticed. Many did
that by
resorting to trickery, like keyword stuffing and
link farms.
Since then, it has become increasingly difficult
to get a
web site indexed and ranked well at the major search
engines.
Strategies have evolved that seem to work, then
mergers and
partnerships change and so do all of the rules.
It never ends and the confusion grows with each
passing day.
It seemed that paid inclusion and pay-per-click
were the
only ways to get your site listed anywhere besides
Google
lately, let alone ranked well.
LookSmart has come up with a brilliant idea to grow
their
own index that, if it catches on, will offer huge
benefits
to both LookSmart and to the participants in the
program.
That seems to be the elusive win-win situation that
all but
Google have been looking for. It's called distributed
web
crawling and gives web site owners a hand in growing
the
Looksmart index as well as the ability to include
their
own site in the results!
This win-win situation requires participants to
download the
"Grub" distributed crawler client to their
PC and allow it
to run when the computer is otherwise idle. The
software to
run a LookSmart Grub web crawler is available at:
http://www.grub.org
It is based on open source software and is made
available
to all to distribute and even modify! Since installing
"Grubby" on my own PC three days ago,
I've personally
contributed over 100,000 URL's to the LookSmart
database!
It's fascinating to watch and the GUI is both entertaining
and attractive. It can run invisibly when minimized
in the
system tray or you can sit and watch as the URL's
fly by
during a crawl from within two available views on
your PC
desktop.
I love the concept and plan to continue helping
LookSmart
grow their database. I've also taken advantage of
the option
to have "Grubby" crawl both my own sites
and those of clients
by including a small text file with a digital ID
in the root
directory of each server I have access to.
There are no drawbacks here, just benefits. I help
"Grubby"
to crawl the web for LookSmart and Grubby helps
me to get my
own sites listed there by agreeing to crawl them
regularly.
No payment involved other than the donated bandwidth
and the
PC computing power that would be running a silly
screen saver
otherwise. They even provide a cool screensaver
of their own
that simulates the crawling process with colorful
orbs that
represent the sites you are crawling to help weave
the web!
When you visit the front page at the URL above,
you see the
latest statistics on your contribution to crawling
the web.
They've even introduced a measure of competition
into the
program by showing top contributors and ranking
them. Sunday
the stats at the top of the page read, "323
clients running
- crawling 24,355,058 URLs in the last 24 hours."
It will
be interesting and very telling to see if that number
grows
significantly or very soon because that will be
some measure
of the success of the program over time.
This process and the software all seem to harken
back to
friendlier times on the web where community mattered
more
than profit. This could even threaten the mighty
Google if
it spreads, since we all love to participate in
things that
interest us and LookSmart has a really interesting
little
Grub helping to grow the web from my home office
and yours.
More on Grub
Distributed Web Crawling
-----------------------------------------------------------
Mike Banks Valentine is a Search Engine Optimization
specialist practicing ethical small business SEO
http://searchengineoptimism.com/SEO_Tutorial/
Search Engine Placement, Optimization, Marketing
http://website101.com/Search_Engine_Positioning/
Return
to FREE articles index |
Search
Engine Marketing: Special
Reports from Page Zero Unleash Amazing Profits
with Google AdWords Select! You advertise your
product, service, or cause online. You've decided
to pay for targeted traffic on a "pay per click"
basis. And now you're considering Google
AdWords Select. Great decision. But if you
don't use the techniques taught in this
special report, you could cost yourself a
fortune. Use it right, and you'll clean up.
Limited
time Special
Get a FREE $10 credit toward your PPC campaign
@ Overture!
|
|
Visit Our Sponsor! WebPosition Gold