Aviva Directories Blog

Background

On the evening of September 2, 2007 my wife Aviva ran to my office in a panic. She exclaimed that Aviva Directory no longer seemed to be ranking for its own name in Google. I tried searching this, and sure enough, she was right. Even searching for “Aviva Directory” with quotes did not bring up this website. She then said, try Alive Directory – and we found the same result. We then tried Directory Dump and eWebPages and found the same thing.

My wife then tested the ranking of all the directories listed in our Strongest Directories List and found that over 60 of the directories listed were penalized. This formed the basis of many lists of penalized directories that were posted around the net, including SEOMoz. Brent went through his list of directories and found many more directories that had been penalized – bringing the total to over 100.

Over subsequent months, the Google attack on web directories continued. The penalized directories got almost completely de-indexed in Google. There were at least two more occasions on which many additional web directories were penalized. After this, a wave of other websites got penalized, including those who sold text link ads, and those who were involved in paid posting. The Google smackdown even hit many of the most prominent bloggers (who all recovered quickly).

Since the penalization, we have worked hard to get Aviva Directory unpenalized, and to a large extent succeeded. Without even filing a reconsideration request, we now rank for our official name, and have many of our old rankings back. Our directory is well indexed and crawled regularly. As far as I know, Aviva Directory is the only directory that has recovered like this.

Here are the steps I have taken to try to get my Google rankings back. Hopefully, you will learn from my mistakes.

Steps Taken

1. No More Paid Links – Yes, I had some. I know it seems stupid now, but at the time the prevailing wisdom was that the worst that could happen is that Google would simply discount paid links. Otherwise, you could get your competitors in trouble simply by buying links for them.

I have now gotten rid of all paid links (there are still some sites that have not removed them even though I no longer pay for them). In retrospect, I am not even sure why I bought them, as I already had a very good backlink profile without them. A large part of it was that I just got caught up with things – everyone else was successfully using paid links, why not me? It was kind of like driving on a highway on which everyone is speeding – it is dangerous to be the slowest car on the road. And many cars were going much faster than I was – surely, if someone were to be pulled over for speeding, it wouldn’t be me.

2. No Sponsoring Templates – One technique I used to build links for Aviva Directory was sponsoring free templates for others to use – both templates for various directory scripts as well as templates for WordPress. This was an idea I originally got from a fairly reputable source. I was proud of the templates that I had available – I thought I was contributing something valuable to webmasters. I made sure that the templates I sponsored were of high quality, and in fact, many of them were very popular and included in “best of” lists.

Matt Cutts did warn about the dangers of sponsored themes in WordPress along the way. However, I took comfort in the fact that many prominent sites continued this practice without any apparent penalty. As well, other reputable sources continued to recommend sponsoring free themes as a method of building backlinks. Even though I specifically warned that this may be dangerous (comment #2), the author of that site specifically disagrees with me (comment #4).

That being said, as Matt Cutts has publicly gone on record as stating that he considers this type of link building to be similar to paid links, I have removed all of the free templates that I offer from my site.

3. No Linking To Bad Neighborhoods – One of the easiest ways for Google to penalize a low quality directory is its outbound link profile. A common trait of a low quality directory is many links to bad neighborhoods. This is one reason why for a directory, high editorial standards are important (another reason is obviously, that’s what makes a directory useful).

Aviva Directory has always had high editorial standards. However, I discovered that there were a few problems with our editing policies. The first is that we did not re-review websites often enough. It is actually amazing how quickly the web changes and what was once a quality website quickly becomes a spammy one. We run a link checking program on a regular basis that flags websites that have died. We did annual re-reviews of all websites. And we did spot checks on websites randomly throughout the year. However, this wasn’t enough. In the circumstances, we are conducting reviews of websites much more frequently than once per year – we are aiming to do it 3 or 4 times per year.

Another area of weakness was in our review of other web directories. As a professional courtesy, we accepted pretty much any web directory, even if it was just days old and had hardly any listings. This led to a case of Directory Spongiform Encephalopathy. We have deleted and refunded the listings for many directories and we now hold web directories to the same review standards as any other website. Unfortunately, this caused a lot of friction with colleagues in the industry. We also had “web directories” as a top level category, and have moved it to an appropriate subcategory. As well, we deleted the second page of our strongest directories list which ironically, contained all the directories that weren’t very strong.

Another area of weakness in our editorial policies was payday loan and similar types of websites. Frankly, the people designing these sites are getting good at hiding their spaminess! We now have very stringent guidelines for these types of websites and frankly, generally no longer accept them.

The same goes for “myspace resources” type websites. While there are some good ones out there, we now have very stringent guidelines for these types of websites as well, and generally no longer accept them.

Finally, we no longer accept gambling sites. I have nothing personally against gambling, and in fact have held company social events at the local casino and the local racetrack. However, I decided that it is almost impossible to determine what a “good quality” gambling website is, and as a result don’t want to take my chances with them.

In the end, we deleted and refunded several hundred websites.

4. More Quality Content – I think one of the unfair things about general web directories is that they are held to standards that other websites are not held to. For instance, if you have a category without many listings, people will point that out and say your directory is lacking. Yet if a website only contains a short article about a subject, people will not say the website is lacking and should include a more in-depth article about the topic.

Even niche directories get away with this. I reviewed many of the niche directories recommended by Rand Fishkin on his list of recommended directories (you need to be a paid member to see his list). Aviva Directory has more listings than most of them. Yet, we were constantly criticized for having insufficient listings.

I don’t think that any directories were penalized for having insufficient listings. That being said, Google hand reviews websites before deciding whether to remove a penalty. It would seem to me that Google reviewers would share the same biases that other webmasters have and a web directory, no matter how “white hat” it may be, might remain penalized simply because of a paucity of listings.

In the circumstances, we beefed up our editorial work substantially. Aviva Directory now contains about 50% more listings than we did at the time of the penalty. We continue to add more listings at an increasing speed.  

Matt Cutts Speaks

I am very thankful to Matt Cutts for taking the time out of his busy schedule to address some of the issues. He mentioned a number of additional issues with Aviva Directory, which I have since remedied.

5. Reciprocal Linking? - Matt Cutts pointed out that from Google’s point of view, Aviva Directory required a reciprocal link with submissions. This most definitely was not the case. Aviva Directory is based upon Php Link Directory Software. The standard installation of this software includes field for inputting a reciprocal link. A simple search in Google shows this. However, nowhere in Aviva Directory’s submission guidelines did it state that a reciprocal link was required. As well an asterisk (*) is marked next to all mandatory fields, and this did not include the reciprocal link field.

At one time, Aviva Directory did have an option for submitting with a reciprocal link, simply because I didn’t know better and that is the way the software came. Even when this was an option, it was only very rarely that submitters did use this option. This option was removed long ago, based on the advice of Greg Hartnett.

Reciprocal linking is something that Aviva Directory has never engaged in and I would have hoped that Google’s algorithm would have been sophisticated enough to detect that. In any event, I have had the field for reciprocal links removed from the submission form.

6. No Cross Linking – I had five directories that all had sitewide links to each other. At the time, I did not view this as manipulative. The sites are all obviously related to each other. As well, the links were there more for traffic and branding than any linking benefits – I was hoping submitters to one directory would submit to my other directories.

Matt Cutts never explicitly stated that cross linking of the directories was a problem. However, reading between the lines of what he said, it was my impression this was certainly a factor. I think part of the problem was also that Google found it to be deceptive that it was not revealed that these other directories were owned by the same company as Aviva Directory. I guess that really never occurred to me; I have never hidden the fact (in fact quite the opposite – I have used them to cross-promote) and most people in the industry know this. That being said, there would have been no way for the casual visitor to have known this. Finally, as is clear from Matt Cutts’ post, one of the other directories had issues, and a sitewide link to it was probably harming Aviva Directory.

In the circumstances, I removed all the cross linking, not just on Aviva Directory, but between all our directories.

7. Contact Details – Aviva Directory had private registration and the contact form did not provide any way to reach us other than via email. This was because I work at home, and don’t particularly care to have people know where I live, particularly as I have a young son. Google seemed to interpret that as a sign that I had something to hide, and I can see how people browsing the website might believe the same thing.

In the circumstances, I have removed the private registration from the domain. Our contact form now has our address and phone number. We have also spruced up our About Us page.

8. Too Many Directories – Our company has six main directories it focuses on – Aviva Directory, Sevenseek, KwikGoblin, Apahcinc, Web10, and Umdum. We also owned several other directories that were basically neglected. Most were ones we started, but never really launched. One we picked up in a package deal. We never promoted these sites. Much to my surprise, it appears that Google looks at all the websites you own in evaluating a particular website. So, from Google’s point of view, it looked like we had a lot of spammy directories. I’m not sure I agree with Google on this one – it seems unfair to me to punish a site for other sites owned by the same person.

In any event, we have divested ourselves of the neglected directories and now only own the six web directories that we look after with care. To be on the safe side, we have also gotten rid of several spammy non-directory websites (that also were abandoned long ago, but just left to languish).  We will divest ourselves of additional directories.

9. No Redirection of Expired Domains – One thing I do when I’m not working on Aviva Directory is domaining. Many domains have lots of traffic – either from type ins due to the keyword gravity of the domain or from the fact that there used to be a website there. Normally, what domainers do in such a situation is “park” the page – place one of those pages filled with nothing but ads (and an enticing photograph) that you have probably seen so often that you’re sick of them. (Tip: the click through rate on those ads can be phenomenal).

Instead of parking these domains, I 301 redirected them to the relevant category in Aviva Directory. So, if a domain was about widgets, it was redirected to the widgets category. I thought that this actually provided people with a better surfing experience – instead of landing on a page filled with ads, they landed on a page of editorially selected relevant websites.

301 redirects are neither bad nor good per se. However, excessive use of 301 redirects can certainly get you in trouble. Matt Cutts since then has expanded on the proper use of 301 redirects.

In any event, I have now parked all my traffic domains and none of them redirects to Aviva Directory any longer.

One thing that I think is important to point out is that Matt Cutts made it clear that no one of the issues discussed above is decisive; Google is looking at all factors in combination – the big picture. This seems like a good idea to me – it means that a lot of the “what if your competitor does X to your website” is misplaced. Even if your competitor does, say, buy lots of links to your website, the consideration of a wide variety of factors ensures that you are less likely to be penalized for the malicious act.

10. Nothing To Do With Top Hits and Latest Links – Several people noticed that we removed the “top hits” and “latest links” pages, and asked whether this had to do with the penalty. Removing these pages had nothing to do with this. The “top hits” page was being gamed by people so they could get an additional link. As for the “latest links” page, other directory owners were going through these listings and emailing the webmasters asking for a submission (while badmouthing Aviva Directory in some cases) so I felt it prudent to remove this page.

Conclusion

Even though Aviva Directory now ranks for its own name, and is well indexed and crawled by Google, I’m not sure that the site is completely unpenalized. In particular, its pagerank has not returned (just a pagerank 4), and the site doesn’t rank as well as it “should” for several terms. So, I have gone ahead today and submitted a reconsideration request to Google. Hopefully, there will be positive results from this.

A big thanks goes to Aaron Wall who wrote several very good blog posts on the issue. On the day of the penalization, he wrote Should Google Penalize Companies for Their Official Brand Names?. He followed that up a couple of days later with the article Publicity and Penalties, pointing out how Rand Fishkin’s public scrutiny of directories and the ensuing controversy, may have been what precipitated this. The following day he wrote How to Know the Difference Between an Automated Penalty and a Hand Edit which helped directory owners, including me, to realize that what happened was a hand penalty, rather than something algorithmic. Finally, a couple of weeks later, after directory owners were struggling in vain to find patterns as to why they were penalized, he wrote Why Google Hand Editing Seems Random.

I’d also like to thank several other people who helped me. Despite our many public differences, Rand Fishkin and I have remained friends and had many discussions about Google’s treatment of directories. He has also offered me many helpful tips and insights along the way. Michael Gray has also gone out of his way to help. Thanks to Debra Mastaler, Michael VanDeMar and John Scott for their support. And believe it or not, Andy Hagans is a nice guy (just don’t tell anyone that). I’m even thankful to Matt Cutts for taking the time out of his busy schedule (it was during a conference) to discuss with me about the issues. And I’m thankful to my wife, Aviva, for suffering through all of this with me, even though none of it was her fault, without ever complaining.

Ironically, I always considered myself fairly “white hat” in my approach to SEO. Re-reading my blog post I’m struck somewhat how far from that I strayed. I’ve certainly learned my lesson, and will never do this sort of thing again, either with Aviva Directory, or with any other website I own. Regardless of what happens with the Google reconsideration request, I am happy. The “clean up” of Aviva Directory has already significantly reduced any penalty suffered. Ironically, the penalty has made the directory much better than it was before, as a lot of the effort that was previously put into marketing has instead been put into the editorial side of things. I plan to continue this editorial growth and ensure that Aviva remains one of the top web directories.