SEO Spider Review – Xenu on SEO steroids
One of the first tasks when getting to a brand new SEO project is to assess the state of on-site optimization. That is the place where we usually do the first measurement of ”how bad the things really are” or “how badly the previous SEO company screwed up” . One of the most basic tools that I always used for this task is Xenu Link Sleuth. It was great for looking at possible crawling problems, getting server headers, checking alt tags on images etc.
However, the problem with Xenu was that it is not an SEO tool. As the on-site SEO changed, so did the requirements of the indexing tool. All of a sudden, I wanted to see what pages have a canonical tag, what pages have duplicate metas issues, whether any pages have robots metatag issues and Xenu just doesn’t do this.
Enter Screaming Frog SEO Spider. If Xenu had a brother that did SEO, that’s what it would look like. It is a great tool and one of the landmarks of a great tool is constant discovery of new ways it can be useful. It has saved me numerous hours, mostly at the initial site audit stage.
I will not go into the nuts and bolts of how the tool works. The interface is very intuitive and if you have used Xenu in the past, you will feel at home with SEO Spider. Furthermore, there is a helpful video that explains how it works (British accent alert!). Instead of doing a comprehensive review of how the tool works, here are a few ways SEO Spider helped me in my SEO tasks recently:
(DISCLAIMER: SEO Spider has a free version that will index up to 500 pages. Full version – individual licence – costs £99 per year. I am paying for the full version and am not being reimbursed in any way, shape or form for writing this review):
- Investigating distribution of onsite links – When large websites want to promote a certain section of their site, one of the ways to do this is to increase the number of links to that page/section from other pages on the site. If the promoted page is found deep within the navigational structure, this will bring it closer to the root and increase the linkjuice flow to it. An overview of internal category/product pages that have a larger than usual number of on-site links pointing to them can help us identify competitor’s SEO campaign targets. Furthermore, if we cross examine these promoted pages with the number of offsite links to internal pages, we can gain some nice insights about promoted products, targeted keywords, linking strategies and other aspects of SEO campaigns.
- Identification of images without alts – this is especially true about images that serve as links. In addition to the capability of filtering images without ALTs, there are other options, as in filtering images that are larger than 100kbs or filtering images that have a long ALT tag.
- Searching link prospects for broken links you can replace – as a part of link building process, you can go to a linking prospect site, run the spider on it and get a list of all outgoing links that result in 404. Go to the linking pages, find out what was the linking purpose and the context of the broken link. Then you can either create a similar resource on one of your sites or tweak your existing content to fit the linking intention and contact the linking prospect, alerting them about the broken link and pointing out similar helpful info on your site.
- Identifying “bait and switch” links on your site – there is a possibility of webmasters creating useful content that you want to link out to. After a while, some of those websites may be taken over by other companies that want to leverage the existing links and authority and redirect them to their own resources. Thus, unwillingly, you may find your site linking to content that you did not intend to link to. In the mild case, your visitors will click through and will not find the useful content you intended them to. In worse cases your site may be flagged as linking to adult and maybe even illegal content. Hence you can get a list of all the external links that result in a redirect and check them out. A lot of the cases those will be simple canonicalization issues that webmaster took care of (adding the trailing slash, pointing to the www version of the site, reflecting a change in directory structure, etc.)
- Discover noindex/nofollow metas – A great way to get an overview and identify screw-ups left behind by your developers. Leaving noindex metas sounds like a beginner mistake, but you would be surprised how many experienced SEOs stumble here, either due to their own oversight or due to forgetful developers
- List all the canonicals – canonical tags are a great way to get rid of duplicate content issues on site (and across sites) but it can be used for other, more sinister purposes too (that’s for another post). Conveniently, SEO Spider will spit out all the canonicals across the site
- Search inside the HTML source of pages – this is one of the recent additions to SEO Spider. It can be very useful if you want to find all the pages linking to a target page with a specific anchor text or look for tracking code within pages. Up to 5 custom searches can be defined in the Configuration section.
These are just a few things I have found in the last few weeks. I am sure this list will just grow, so check it out at later stage, I may be adding more great ways to use SEO Spider
Posted in General, SEO
24 Responses to “SEO Spider Review – Xenu on SEO steroids”
-
I have been playing with the Screaming Frog tool for a couple of weeks myself and I have to say it does offer some excellent insights. I do also use the IIS SEO Toolkit as well and it does no harm at all to use both, the run time on the SEO toolkit is a little quicker (dependent on site size) but as the Screaming Frog tool develops I am sure that may already be on their hitlist.
On the positive side this tool is a hell of a lot easier to install than the IIS SEO toolkit.
Excellent article Branko.
By Dean Cruddace on Feb 24, 2011
-
The Screaming Frog SEO Spider is an excellent tool for analysis, especially when reviewing large ecommerce sites.
Great time saver, the paid version is worth every penny.
By Lee on Feb 25, 2011
-
wow, great tool! I use Xenu several times a week for many years, but this one blows it away! I’m testing it right now and I’m very impressed, and considering buying the unlimited version. Thank you for this great tip & article !
By Rene on Feb 25, 2011
-
Unfortunately the tool doesnt support special characters like german umlauts ä, ö, ü
The pages then get a 404 in the tool.By fiacyberz on Feb 25, 2011
-
Neyne, thanks for the great review we’re really pleased you like the tool!
@fiacyberz – This was something we tried to resolve with the last couple of releases, are you running the newest version (v1.30)? If you’re not, update and hopefully that should fix the problems. If that still doesn’t sort it let us know and we can look into it
By Matt Hopson on Feb 25, 2011
-
Xenu is my fav tool ever. Not elegant my any means but very functional.
By Lisa on Feb 25, 2011
-
Thanks for reminding me about this great tool.
It provides hours of valuable entertainment, even sniffing around my own site. So, many things to fix someday by just reviewing the external links or response codes tabs. Need more 200s and less 404s.
I will look forward to your updates. Not sure how I will know about them. Maybe add a subscribe to comments and drop a message when you update the post. I don’t really use RSS any more.
By Jim Spencer on Feb 25, 2011
-
The larger the site, the higher the demand for tools such as this one… Thanks for sharing your examples! The developers told me that they’re hoping to release a Mac version as early as next week. Woot!
By Jonathan on Feb 25, 2011
-
I’m going to confirm what you already know, just because. There are a lot of articles about SEO software, and the only reason I clicked on the link for yours is that you mentioned XENU. Very clever. And I’m certainly glad you are, because this description makes Screaming Frog sound awesome. I’m gonna download it right now.
By Michael LaRocca on Mar 1, 2011
-
I hadn’t heard of either of these before, but the Screaming Frog SEO Spider is definitely interesting.
Cheers for the tip. Gonna have a play around and see what it can do!
By Finch on Mar 1, 2011
-
WOW this is a fantastic alternative to Xenu, I have just installed after reading this post and tried it on one website, I think this is going to be well worth the small yearly charge
By Stuart Chester on Mar 1, 2011
-
I have problem with Xenu and other software to crawl really big sites, like with a million pages, when you can’t split it in parts. They all crash with an out of memory error..
Do you know if SEO Spider can manage it?
Thanks.
By JosebaU on Mar 22, 2011
-
This software looks amazing. Im going to have to test it out and review it on our site.
By Crenk on Apr 6, 2011
-
Really better than Xenu. Nice tools.
By Aashish on Apr 7, 2011
-
I’ve been using Screaming Frog for a few months now and just love the functionality and ease of use.
Well done on this write up. A very useful review and tutorial.
By Adrian (Bold Internet) on Apr 7, 2011
-
Wow. Ive been using Xenu for awhile now and have been annoyed at it’s limitations for large ecommerce sites.
This tool grabs the Anchor texts and H tags!
Thats amazing!
By Chris on Apr 7, 2011
-
First time I’ve been here, and obviously I’ve been missing out. I was searching for checks on bait & switch and I’m happy to find a clear answer and even better that I’m already using ScreamingFrog. Nice one.
By Dave Sottimano on May 5, 2011
-
firs off all’ thx for agreat article.
the Xenu is amust tool, to all peple thet working on abig sites.
i working with Xenu evry allmost evry day and it giving me new look on the insite.Thanks for sharing your Your experience with this wonderful tool.
By seori on May 22, 2011
-
I use both for case something will go wrong and for double checking. Must have tool for every seo – specialist.
By Dneprolab on May 31, 2011
-
I like the added advantage of usability as well as easy parsing of important SEO data. BUT… The tool is simply not as industrial strength as I’d like. I’ve communicated to the Screaming Frog team numerous times but have only been able to achieve small incremental results. I need a tool that can quickly crawl an eCommerce or large content site with over 100K pages. It take SF nearly an hour to get to a few thousand pages and then crashes. The most I’ve been able to achieve so far has been about 10,000 pages and took more than an hour.
Do I have unrealistic expectations here? Please advise. Thank you!
By Anthony Young on Jun 29, 2011
-
Really better than Xenu. Nice tools.
By seo arizona on Jun 30, 2011
-
Hey Anthony,
I feel you with the crawling limits. I think your expectations are somewhat unrealistic. It is a desktop tool and there are limits that are imposed by your system requirements as well. Also I don’t know of any other similar tool that can crawl that many pages in such short time.
If I was you, I would install the Screaming Frog on some VPS and let it run for as long as it takes.
By neyne on Jul 1, 2011
-
Like you we use the paid/pro version – worth every penny!
By Doublespark SEO on Jul 29, 2011
1 Trackback(s)
- Feb 24, 2011: Tweets that mention SEO Spider Review – Xenu on SEO steroids | SEO Scientist - Applying the scientific method to SEO -- Topsy.com
Sorry, comments for this entry are closed at this time.