Well that was a crazy ride!
On the 23rd of July 2012 the company I work for received an Unnatural inbound links Penalty.
It took a few days to find out any information about this, because as usual Google does not announce this stuff and surprise, you have a and no guidelines or information to go on. (This is one of the main causes of spam)
At almost the very same time I read this!
Hmm... Does this apply to my site? NO IDEA! This happens so often that it is a wonder anyone recovers from any penalties.
The interesting thing was that in the previous few years we had hired a marketing company who we met at a well known business show. This company is very well known, they said to us they would do PR for us. That is "Public Relations" NOT Press Release NOT PageRank but Public Relations, which was understood by my boss as what he knows it to be and as stated in Wikipedia "the practice of managing the spread of information between an individual or an organization and the public". They said that we needed to do that to increase our ranking position in Google.
This would lead us down the dark path we eventually got too. Around 2007 They started to market us in magazines, online and offline. They interviewed the boss for a press release and got him to create articles based on his knowledge of the industry. He is well known and respected in the field and they were informative. Looking back to those days Matt Cutts never said it was not OK to do this, however he said it was not ideal. To punish anyone for that is wrong. It is either against the guidelines or it is not. Now all the the Google lovers can go on a mission to show me that there are other articles to say that paid links were always a no go zone, nofollow has been around since 2005
. Very true, however this was not knowledge the average or even experienced web site owner was aware of. So by watching a video that suggest its was not ideal but was not wrong if done correctly recorded by Matt Cutts
himself you would not go looking for more answers. That being said in that video he does say its not ideal. However Going back before that he is even more vague and in some cases says that if done properly are not a bad thing.
At the same time they also created directory links for us, explaining that they are the same as Yahoo or Dmoz but carry much less authority.
As time went on the rules changed for Google, but the links were already in existence, why should we be penalized for something that was not spoken about in a load and clear voice that everyone at the time was also doing. A simple clear and very public warning would suffice? To be expected today to clean up links from as far back as 2005 that we found is simply unfair.
All the time that these rules were changing we were non the wiser as we were concentrating on our business and left the marketing up to the hired company. They had produced great success for us, they pushed out press releases that got media attention of which got news papers to call us interview us and write great articles about our services. that is what press releases are all about. All of which had a naked link at the bottom, no anchor text. (almost nobody was using nofollow back then as they did not know about it). People on Google forums always ask you questions based on why did you create this spam rather than realizing that many of the services out there are very old and are existence because they had a good service before it was abused, like Marketwire. (do i dofollow nofollow a links to them on this post? who knows any more. If I don't link at all maybe that is safer? What a joke.) Their service is to provide the media and others with information you want them to know. Posting on your website will not do that. The same as finding a site that has lots of traffic in your niche and getting an article on there, that's whole idea of marketing and advertising. Putting it on my site does not draw that crowd and if it did it took a hell of a long time to do that. So those are the reason for using services like that. Well it was back in 2005 and a few years onbefore it was abused for the links they could provide.
As time went on we were unaware of how strict Google was getting when it came to rules, our rankings were good and we had no reason to look any further into it. If we are ranking well then our Marketing company must be doing a good job, why would we think any different? If something was wrong Google would let us know? LOL how naive we were.
Then we get hit by Google Caffeine with no idea from Google about what was going on.
What we did not know was that the marketing company like many others were struggling to find white hat methods preached by Matt Cutts and others that ACTUALLY worked and could beat a black hat link building strategy. This forced them to adopt similar habits without letting us know in an attempt to keep us on as clients.
Google is so bad at dealing with spam results in a timely manner that it is very difficult for anyone to act responsibly as the message to the public is, site A has been at #1 for 2 months now taking in 70% of the traffic in the niche and is a one page spam site created as an affiliate site and we have not removed it despite all the webspam requests so you should do the same thing. Because as soon as it is removed another one pops up for just as long. for Google to expect webmasters to not adopt the same tactics is the height of stupidity. this has been going on since the start of Google.
Next we get hit with Panda, it has now become clear to us that there is a big problem and we lose all our main keyword rankings. The information was very weak from Google on what the problems were as we would later find out the main issue we think was our property listings that were various data feeds and the content was replicated across multiple sites. Does Google expect them to be different on each site? Surely not? However those listings mostly did not get affected by the update, so we had no reason to suspect that was the issue. So we focused on putting content on our other pages and services we offered. The problem we see today is that the content was created by many different people in our many offices across the world. they each created content describing the services, area etc.. but English was not their native language. So that content linking at it now is not of high quality, but was NOT the cause of panda as it was created AFTER panda to reduce what we thought was thin content pages. By Google not giving us ANY help in understanding the problem we were just making wild guesses as to what was wrong and making the situation worse. So many people did the same thing.
The marketing company was in full panic mode creating links and doing things that we had no idea about. We were watching our rankings drop each month asking what was going on and getting replies such as "this is happening to everyone, don't worry".
In desperation we decided to try Google Adwords, the Ireland office had called us up and gave us the sales pitch and we fell for it. (Interesting timing Google). They took £3,000 and it was gone in about 3 days. We did not see much traffic and no sales, which was strange because when we ranked top 3 we always made sales. They told us there were many reasons and click fraud in our industry was a big one. We never heard from them again.
Also at one point in time we started to notice traffic coming from forums more and more and got one of our internal sales team to go to good forums in our niche and give general advice. We did not see the harm in signature links as people were always asking what site we were from. The marketing company also noticed this and said they would create some accounts in our name and would ask us questions every now and then and they would post them for us. later on we would find out that they took this much further and were using the accounts all the time to ask all sorts of questions in SEO forums in an attempt to get us links. We recently discovered they had used those accounts also to ask questions and purchase services like link building packages. This actually helped us track down a few people to get links removed funnily enough. But we ere fuming when we found out they were not even doing work themselves but out sourcing it to low quality service providers in places like India, China and the Philippines.
This was the point where we had discovered what kind of quality work the marketing company were really doing and fired them immediately.
At this time we started to remove some links as there was rumor going around that the rankings were bad because of anchor text in back links. Just a few months later we got the unnatural links penalty notice and were left helpless to understand what to do.
THE RECOVERY PROCESS
As usual for Google no guidelines! We were left with no clue what we were doing so we contacted every site that we had contact details for and got those links removed. MAYBE A HUGE MISTAKE. We removed all the natural links from big newspapers that we got through publicity from being at events, hosting them, talks etc..
We had no way of understanding how to get a list of links then how to find out what were bad or what to do.
We filed multiple reconsideration request and read as much as we could in Google help forums and started our own thread.
Then came the release of the disavow tool on October 16th 2012 with very little guidelines and no barely an instruction manual.
Then came one of the most confusing messages from Matt Cutts
as it was released the same day. Matt Cutts says:-
So at the point where you have written to as many people as you can multiple times, you've really tried hard to get in touch, and you've only been able to get some certain fraction of those links down, and there's still a small fraction of links left, that’s where you can use our new tool, called the disavow links tool.
This is what we decided to do. We did not want to get into a penalty for misusing the tool so we only put about 5 links in there, because Matt said do all you can first and use this as a last resort. We wasted a lot of time because of this. We paid for links to be removed because people demanded it, we spent weeks trying to find the owners of sites that had not contact details etc.
Eventually somehow I found John Mueller
from Google on the Google forum and asked him a question. He actually took the time to look at my site and commented on the fact that I had only put 5 links in my disavow file. I told him why. His response was very different from Matt Cutts
it's really something where you want to go through all of your links and really make an assessment which ones are good and which ones are kind of iffy and first try to contact these websites to have those links removed as much as possible but you can also at the same time add them to your disavow file so at least in the meantime they will be processed the next time they are crawled as having a nofollow attached.
Matt Cutts made it clear to use it only for a small fraction of links and only once you have tried everything. John Mueller says, put them in the disavow file right away, then start to contact them. Had we done that we would have recovered much faster. Matt Cutts should edit that video or come out with an update.
We went through lots more miscommunication along the way, this included the webmaster tools list of links being vastly different from just one month previous and then reverting to and old one and also a point where webmaster tools had a glitch
that showed very few links to our site. Google was not admitting to a problem and we were unable to continue our work properly as we were unsure what was going on.
Eventually we stumbled across Google Hangouts
as the information I was getting in the Google forum was conflicting. This is what proved to be one of two lifesavers.
I meticulously went through the videos and made notes of every question that was related to our situation that I was unable to get answered by top contributors in Google forums. I started to understand what the problem were and replayed those same questions back to the forum, only to get different answers... How strange? What was going on? Why Did John Mueller, Matt Cutts and Top Contributors all have different answers to the same questions? the answer was because the whole process is so badly communicated internally that nobody knows what is correct. this is why there needs to be some sort of guidelines.
The other lifesaver was a fantastic person who goes by the name of StevieD
on Google forums. i stand by my words on this site that very few top contributors have the answers because it is true. StevieD was the only one to recognize a serious problem that would have gone unnoticed for a very very long time.
This is what he noticed.
There were many sites in our webmaster tools list that were not in our disavow, they were links that had been checked up to 6 months or more back as missing, 404's, pages not loading. Well it turns out that because so many links were on very poor sites/server that many times they load up infrequently and can give the appearance they are no longer there. WHY would anyone go back and check them again especially when you have 16,000 links to go through one by one?
So it turns out that many of those links still existed and were the worst of them all. No wonder we were failing the reconsideration requests! So include all the missing, 404 etc.. domains in the disavow file.
He also pointed out the dofollows can be changed to nofollows, so include all those links as well that are iffy.
Then there was the big debate on what Google considered to be a bad link! I don't like to be told to do something without a good explanation. So i asked the hard questions. there were links that were no way ever created by anyone associated with our company. But I was told to include them in the disavow. I think the whole process is bonkers personally, but in the end the answer was why would you want low quality links pointing to your site even if you never created them? I argued that it was not the proper use of the disavow file. However John Mueller makes a good point
here regarding the whole matter.
Also one thing that became clear after a while was that we had links in directory sites that moved from page 5 to 6 to 7 etc... as more links were added to that directory. this mean that the page we disavowed just a few weeks back was no longer valid and the link was again dofollow in the eyes of Google. This is why it is vital that you use the "domain:example.com" format for almost every domain NOT individual links. Matt Cutts mentions this here
. Also John Muller Talks about it here
So, after a long long process, I finally got the answers I needed that could have resolved the whole Unnatural Links penalty issue in just a matter of weeks/months.
- Download the list of links that point to your site from Webmaster Tools.
- Go through them and determine the dofollow and nofollow links, this will help you also find the missing ones. Check out the blackhatlinks.com free tool here (I spent hours doing it by hand before I found this tool. It does have a few small problems)
- Look at all the domains and work out what ones are not really really good sites with genuine links, not ones that juts have good PR and you want to keep, those are the ones Google is probably most aware of as being spam.
- Create and submit a disavow file with all those Dofollow, Nofollow and missing link/404 domains.
- Next create a spread sheet and start contacting all the sites to get your links removed. Article sites are the easiest, they react quickly and are a major source of what Google hates. If you are unable to find a contact page or email. Leave it and move on. Try to focus on quantity. Trying to find contact details for a site that does not have some or hides them is unlikely to remove a link or even read your removal request.
- if you are having serious problems get onto the Google webmaster central forums and ask for some help. But be prepared for some nasty people as you are branded a guilty spammer right from the off by some before they ever understand the full history of how it came about. But you stand a good chance if you run into someone like StevieD that will help you if you are not just a blatant spammer and have a genuine business. Even then he will kindly tell you briefly that you are such a site :) Also make sure you upload your webmaster tools links file to Google docs for people to see as you are the only person with that file. Also include four other files if you can. Dofollow links, nofollow links, missing/404 links and then a list of links you want to keep. You will quickly find out he issue when you do this.
- Submit your disavow file, Share your disavow file with the forums if need be for a second opinion before you submit it this can help you spot any mistakes.
- Once you have done lots of work, upload the spreadsheet to Google docs, to show that you have tried to contact and remove as many links as possible and submit a reconsideration request. Details and examples of a reconsideration request and what to include are here.
- Wait up to a few weeks and fingers crossed you get the wonderful message saying "Manual Spam Action Revoked"
The whole process took 10 months, I will not find out for a while (3 days - 3 months according to various people) if my site will recover any places in the search results. But either way I am happy that the penalty has been lifted and we can start to look forward.
The question of our Panda penalty is now the main focal point and we are working hard on resolving any issues there as well.
I will keep you updated on our progress as things happen. Good or Bad :)
Please feel free to contact me and ask me any questions, I will do my best to help you.