Reconsideration Requests
Show Video

Google+ Hangouts - Office Hours - 30 January 2015

Direct link to this YouTube Video »

Key Questions Below

All questions have Show Video links that will fast forward to the appropriate place in the video.
Transcript Of The Office Hours Hangout
Click on any line of text to go to that point in the video

JOHN MUELLER: --new people. Great. All right. Welcome everyone to today's Google Webmaster Central Office Hours Hangout. My name is John Mueller. I'm a Webmaster Trends Analyst at Google in Switzerland, and part of what I do is talk with webmasters like you all and help answer your questions, any things that might be open, and bring your feedback back to the teams here as well. As always, if one of you wants to get started with a question, feel free to jump on in.

GARY LEE: Hi, John. I just wanted to ask you-- I've seen some interesting situation with the servers at the moment, and our site has dropped for a major keyword to page two. I mean, we were at about fourth or fifth place. We're now at about 15th place. So something's kind of changed. But at the same time, we haven't done any sort of marketing, and advertising, and that kind of stuff to really boost the site. And the thing is that we're scared about what to do, number one. And second of all, we don't know who or how to approach it. And I know you've talked about this before. But even Google's preferred advertisers that you've got with AdWords, agencies and things like that are doing underhanded tactics. Not underhanded. They're doing bad link-building practices. So I want somebody to do marketing, to help us do what we need to do for our business. But the same time, everybody's still stuck in this old school kind of pattern of creating crap to try and please us. And I just don't know where to start.

JOHN MUELLER: Well, I guess on the one hand, at least you have an understanding, and you kind of see what these people are doing. So you can kind of evaluate what people are offering you. So I think that's a good first step. It's hard to say what exactly a site should be doing in general with regards to marketing. I mean, there are lots of possibilities there. With regards to the search changes recently, I think these are essentially just normal changes that we have, as they always happen. When algorithms get updated, things change. When the general structure of the web kind of changes, those are the things that we kind of pick up on. So I don't really have any concrete feedback where I can say, hey, you should be doing exactly this, or going to exactly this agency. I know there are lots of really good agencies out there. There are definitely also some out there that are doing crazy stuff. So if you kind of have an eye for the things that you know you shouldn't be doing, then I think you're on the right track there.

GARY LEE: Yeah. I mean, the problem is we're still seeing all of our competitors doing kind of the wrong things, and still getting away with it, temporarily possibly, until another update. But by us not doing anything, we really are losing out. So I just don't know what to look for, to even ask for. And I wonder if there is something that you guys at Google can do to help people kind of recommend what they should be asking from a marketing company on a very, very high level, rather than you should ask for this, and that, and the other. But just like, this is what you should be looking for in a company. Because this is a major part. If companies like us can find the right people and the right processes, then so everybody else for follow suit, understanding what it is that they actually need. And I think this is the biggest hole in the industry right now. Nobody knows where to go or what to ask for. We're sort of guessing and doing the wrong things.

JOHN MUELLER: Yeah, I mean, we have a page on Help Center about how to pick an SEO. That covers part of that. But it essentially focuses on the things that would be like red flags. That if an SEO company offers you guaranteed number one ranking, then probably, they're doing something crazy. So those are the kind of things we generally point at. But I'd definitely take your feedback on board that maybe having something more in the sense of, this is what you should avoid, but also, this is what you should be looking for to help people find the right partners for this. I think that might be an idea.

GARY LEE: Because you got things like market outreach, and you want to do some news, and you finish it out. But then the problem is that these companies distribute your news to these dodgy, you know, all kinds of different news places. And then you happen to have a link, just a URL link, at the bottom of that news thing, and that's gone out, and now you've built loads of links. Or somebody else takes on an article, and you've got a link in that, and now that's been put on their blog, and somebody can say that you've built that. There's just too many questions. Now I feel like I'm questioning ourselves far too much and too rigorously, that we don't give ourselves a chance to actually do anything. It's just, it seems, an impossible situation we're in right now. But if we don't do something, we're not going to move, and we're going to struggle as a company.

JOHN MUELLER: Yeah. Finding the right balance is always tricky.

GARY LEE: Yeah. OK. Yeah. Well, I'll let you go. But yeah, a weird thing in the search to see the stats kind of drop from a few others. And the results don't look too great, to be honest, John. And if you look at the top 10, it's not something that Google should be proud of.

JOHN MUELLER: So you're specifically talking about the ones you're looking at?

GARY LEE: That's in virtual office.

JOHN MUELLER: Virtual offices?

GARY LEE: Yeah, virtual office, or virtual office, London.


GARY LEE: Yeah. So especially the top 20. If you really add in what's on page two, it needs to be run through another analyzer, or a bunch of people to look through that and say-- there's things that where-- I can't remember one of the names. It's a really, really poor result. And it's basically, they've called their back-end system a virtual office. And there's nothing to do with it whatsoever. So, yeah.

JOHN MUELLER: OK. That's good feedback. Yeah. I'll pass that on to the team.

NEMANJA DIVJAK: OK, John. Can I ask a question about video site maps?


NEMANJA DIVJAK: Do we have to have a link to the original file mp4? They say that you have to have title, description, thumbnail, and that's it. But they require a link to do a Flash file, or original mp4. So do we have to have it inside it, linked inside?

JOHN MUELLER: I'd look at the specification. I'm not exactly sure how that's defined there. But as it's defined in the Help Center, it should be the way that we process it.


JOHN MUELLER: I'm not sure 100% which of those fields are required for video site maps.

NEMANJA DIVJAK: I'm going to share it with you in one second. Sorry if it's--

MALE SPEAKER: OK. I know I did something like that some time ago. A video segment for some site. I know that you have to have the Flash file, or the file that plays on the default. You have to have a few alternatives, but they're not mandatory, I guess.

JOHN MUELLER: Yeah. There are definitely some optional fields there. But if it's specified as required in the Help Center, then you should really define it. And if you don't have it in your site map, then I think that should also be flagged as an error when you submit it to Webmaster Tools.

NEMANJA DIVJAK: Yeah, but there are no errors.

JOHN MUELLER: OK. Then that sounds like you're doing everything right.


JOHN MUELLER: All right. Quick question about the Panda update. There hasn't been one since end of October. We're waiting for our site to recover. Google shouldn't really delay or pause algorithms once it starts them. We do run these things regularly. Sometimes, it's more regular. Sometimes it's less regular. But we tend not to preannounce these kind of changes. So this is something where I imagine it'll just be run again at some point again. So these things, we try to keep them running as regular as possible. Sometimes, that's not always possible. But it's not the case that we artificially stopped them just to bug webmasters about that. Currently, we have a site that operates international websites. However, do you know any issues in georedirecting traffic based on channel? In other words, if I'm only georedirecting organic traffic, and the rest of the traffic not. I guess from a search point of view, there are a few things you need to watch out for. On the one hand, if you're georedirecting, you need to also redirect Googlebot like that. So if you're redirecting all users from the US to a specific part of your site, then Googlebot, when it's crawling from the US, would need to be redirected there as well. On the other hand, if you use something like hreflang, you can use the x-default for the georedirecting page, and for the individual landing pages for the individual languages and countries, you can specify that in the hreflang, as well. The important part there is that the individual country and language landing pages don't redirect. So you have your redirecting homepage, for example, and then you have the individual language versions that, essentially, anyone can access directly. So if someone goes to your home page, they get redirected to the right version. But they can also go to the right version directly if they want to do that. And if you set it up like that, then essentially, that doesn't really matter, which channels you handle like that. So if you redirect traffic from other sources in different ways, that's not something we would really worry about from a search point of view, but which might confuse users. If they go to your website through search and they land on this specific page, but if they go to your website in a different way, maybe with a different referrer, and they land on a different page, then that could be kind of confusing to them. But essentially, there's a blog post we did, I think like November, around then, about handling international home pages, which tells you a bit more about how to set up to this redirector. Webmaster Tools now sends out mass notifications on mobile usability alerts. Our clients are receiving the mobile sites as Google's new online mobile friendly test. What's up? Is it because our sites use JavaScript-based redirects? I think first of all, if you have some kind of redirect between your site, make sure you're using the markup and the techniques that we recommend in our Help Center, in the developer site for mobile-friendly sites. That includes things like the rel=canonical back to the desktop page. If you have a mobile-friendly page that we see the rel=alternate for the mobile-friendly page, that if you're redirecting by JavaScript, that we can at least crawl the JavaScript and see that redirect as well. Ideally, you'd be redirecting on the server side, not on JavaScript. Because everything you do in JavaScript adds additional latency. So especially if you're on a mobile device, and you're redirecting users by JavaScript, then at first, they have to load your HTML. Then they have to load the JavaScript file. Then they have to process the JavaScript file. Then they see the redirect. And all of this adds up. So you're easily adding a couple seconds extra latency for anyone who's using a mobile phone to access your site. So that's something where I'd recommend using server-based redirect if possible. But essentially, if you set this up properly for your website, then we should be able to recognize that your site has the mobile site attached to it, and that if the mobile-friendly site that you have there passes the mobile-friendly test, then that should be OK. What might also be happening there is that you have some pages that pass this test with the redirect, and other pages that don't. So we've seen that, for example, I think with our help form, that was a case, that we had a bunch of the template pages actually redirect you to the mobile-friendly version, but there was one set of template pages that didn't have a mobile-friendly page that we somehow forgot. So like 70% of the site had proper mobile-friendly pages, but there was a large chunk of the site that still didn't have mobile-friendly alternatives. So what might be happening in this case is you have most of it set up properly, but some of it still isn't doing it right. And you can see that in Webmaster Tools with the Mobile Usability Report, where you'll see a graph of the issues that we found across your site. And you can drill down to look at the individual pages there. So that's another thing I'd recommend doing. So first, make sure that you have it set up technically correct with the redirect, so that we can pick that up properly. And then make sure that you're not missing a subset of your pages that are still stuck in this unmobile-friendly state. I see a lot of the same faces in these Hangouts. Can you limit them and throw them all out? It looks like we have some different faces today, so I will refrain from throwing everyone out. But as always, I try to give people a chance to sign up and say they'd like to join these Hangouts as well. And if people let me know about that ahead of time, I tend to add them a few minutes before we start. So if you've really been wanting to get in here, and you've never had a chance to get through, then feel free to sign up there.

ROBB YOUNG: John, have you considered changing the times of them? Have you found a difference in the people who turn up when you change the time? Or are you limited on your schedule because of that?

JOHN MUELLER: I try to take something that's like, early morning European time that works for Asia as well. And something a bit later European time. But I know some people here join in even at crazy hours during their time. Right, Gary?

GARY LEE: Twenty minutes past 2:00 here in the morning.

JOHN MUELLER: Yeah, so I think if you really want to get in--

JOSHUA BERG: I'll be here any time.

JOHN MUELLER: --there is a chance. But I totally understand the point that sometimes, we talk about the same topics over and over again. And it would be nice to get some new faces here. But again, if you really want to join these, just let me know ahead of time, and I'll add you a bit earlier. And everyone is refreshing and jumping in too.

MALE SPEAKER: Can I ask a mobile-related question since we were on the mobile topic?

JOHN MUELLER: All right. Go for it.

MALE SPEAKER: So I have a few clients that I've been noticing a few trends on the-- they have a separated sub-domain for mobile sites, that classic m dot. And unfortunately, I've been trying to move them to either dynamic serving or responsive. But in some cases, there is just a bit of, I don't know, confusion how they look at the performance of these domains in Webmaster Tools, like the desktop version and the mobile version. Because if you add both of them to Webmaster Tools for both of the versions, mobile and desktop, you have the option in search queries to filter mobile and desktop. So once you add the canonicals to the mobile version and everything is set up properly with alternates pointing to the desktop version, and the canonicals to the-- I mean, the other way around. The canonicals put into the desktop version, and the alternates pointing to the mobile version-- what's the expected behavior there? Because it's a bit confusing. I've seen some sites that start dropping in search queries for the mobile version for the web, but then they show an increase for the mobile on the desktop. And for other sites, the behavior is not that, and although everything is set up correctly. What's the expected behavior there if everything is correct?

JOHN MUELLER: I don't know for sure. I haven't actually looked into that in detail. But I can double check. I know to some extent, it's a bit confusing, because some of the impressions are clicks we track on the desktop page because we think it's the canonical. But at the same time, if the mobile page gets the traffic, then when you track the clicks, maybe there. I would have to take a look with the team to see what should be happening there.

MALE SPEAKER: Yeah. OK, because I looked--

JOHN MUELLER: Because it should be consistent.

MALE SPEAKER: I looked in the Web Help Center trying to look for some clarification, and I didn't find any. So that's why I came and bugged you here. Because I mean, when I'm on the mobile version, and I filter on mobile, what am I looking at? If I'm on the m dot of my site, and I go to filter on mobile traffic, what does that represent? And what does web represent then?

JOHN MUELLER: Yeah. I don't know. I'd have to take a look with the team at the details there. I know we looked into that a bit for the redesign that we're working on. So I hope that's a bit clearer there. But I'd have to take a look exactly what the current status.


JOHN MUELLER: And I guess if you guys haven't seen it, I put out a call for a sign up for the alpha versions of an update we're doing for the search query feature. If you haven't signed up and want to take a look, the old feature will still be there if you sign up, by all means, go ahead and do that. I think we'll probably leave it open for another week or so, and maybe get in touch with you guys next week or the week after that to preview the new version.

MALE SPEAKER: I signed up for all the tests, so--

JOHN MUELLER: Fantastic. It's still a very early version, so I wouldn't assume that it's going to look exactly like this when it actually gets live. But we'd really love to see your feedback on things like the usability. Are you finding the right information? Are you able to figure stuff out? Does it encourage you to click around and try things out to find new ways of looking at your data? I wouldn't look at it in the sense of, I want to have exactly the same view as I had in my old setup. So kind of take a step back and see what you can see there, and we'll be asking for feedback as well. So that'll be interesting.

MALE SPEAKER: OK. I might have to leave in a few minutes. So anyway, thanks for your time, and I'll be watching a few more minutes, and then I'll be going.

JOHN MUELLER: Great. Thanks.

MALE SPEAKER: Thanks, John.


ROBB YOUNG: And where are those beta versions again, where there's--

JOHN MUELLER: I put the sign up on my Google+ account somewhere in the--

ROBB YOUNG: In the last-- yeah, I remember you mentioning it on the last call.

JOHN MUELLER: With the big robot. It's from the Google Webmaster's post that we shared.

ROBB YOUNG: All right. Thanks.

JOHN MUELLER: OK. What markup are exactly needed to have a description and link shared on Google+? Some pages shared on Google+ have description below the image and title, and some don't. And how do we reload the cache for this when we do changes in markup? We have, I think, on the Google+ Developer side, there's markup for a Google+ snippet that you can put on your pages that we pick up on. So we pick up some of the things like the Open Graph markup as well. But there's also a specific structured data markup you can do for the snippet, where you can also specify the image that you want to use, if you have a bigger image on the page. So I'd take a look at that. I can pull out the URL of that after the Hangout if you want. But it's with the Google+ Developers side. From January, some site owners are faced with weird problems, including me. When an article is published, the original site doesn't come up when I search for a title. But after seven to eight days, everything becomes normal. What and why is this a problem? Is there any solution? This is something I see from time to time. And usually, what's happening there is that our algorithms are picking up on lower quality signals of the site that's actually publishing this article, where we kind of assume that the site that's publishing the article is iffy, and we're more cautious around that. So we don't pick up everything as quickly as possible there. So that's something where you might want to take a look at and see if you can really improve the overall quality of your website, so that you can be sure that Google's algorithms really are able to trust your site completely, and able to take everything that you publish as quickly as possible. So for example, if you have it in RSS feed and we can pick that up, then chances are we could pick that up within a couple of minutes of your publishing it. But if we don't know that we can really trust your website, then that's something where maybe we'll be a little bit more cautious and kind of hold back a bit. So if you're seeing these kind of delays, then that's something where I'd really recommend taking a step back and thinking what you can do about your website overall to really make sure that it's sending all of the right signals, and not sending lower quality signals, looking kind of sneaky in some sense. Sometimes we see this, for example, with sites that rewrite or scrape articles from other sites. I don't know if that's the case with your website. I don't want to blame you for that specifically. But it's something definitely worth looking at. It's usually less of a technical issue when we see that happening. Why does Webmaster Tools think that the robots.txt file is a web page? We have html errors for the robots.txt file, and now it's our only mobile usability error. I thought this was kind of funny. I had definitely passed it on to the team to look at. But essentially, things like the robots.txt files, site maps files, those kind of things, are also normal files on your server that we can show in search sometimes. So depending on how we ran across the robots.txt file, it might be that we're treating this as a normal web page, and you can maybe find it in search if you specifically search for it. It's not going to be the case that it'll rank for any of your normal keywords, because we have a lot of more relevant information for the other pages on your website. But since we have it in our search index, we can show it in some of these tools as well. I think specifically for the mobile-friendly test, that's something we should probably exclude, maybe together with other text files on a site. Because obviously, if it's a text file, you can't add mobile-friendly markup to it. But this isn't something I'd worry about. In the worst case, what will happen is if someone's searching for your robots.txt file in Search and sees it in Search, then it won't have the mobile-friendly label on it. But that's just your robots.txt file. That's not the rest of your website. Another thing you can do if you want to have it dropped from search is to use the X-Robots-Tag no index header for these kind of files. So that might be something. If you really want to clean things up completely, you can do that. In most cases, it's not that you need to do anything about that. But if you want to take the extra step, go ahead. There are first reports saying that mobile-friendly is a ranking factor now. Can you confirm this? And if yes, in which countries? And what's the status here? So we've had various mobile issues already be ranking factors. So we did a blog post, I think, 2013 even, about various types of mobile issues that we've run across, which we use for ranking as well. Back then, those were mostly technical issues, like wrong redirects. Or if it's a Flash-based site, for example, that's something we could have used there. And with the new mobile-friendly tag, we've said that we're definitely experimenting with ranking changes there as well. So I would certainly expect that things move more in this direction. And the thing to keep in mind in this specific case is that since it's about mobile-friendly sites, it'll be something that smartphone users will see. People who are using mobile phones. So it's not the case that if your site isn't mobile-friendly, that we'll remove it from Search, or that we'll remove it from Desktop Search, or demote it in Desktop Search. It'll be just specifically for people searching on a smartphone, that if we have other results that are similarly relevant, but they are mobile-friendly, then we'll probably show those first in the search results. And I think that makes a lot of sense for the users. And to some extent, this email that we sent out about these mobile-friendly issues that touches upon this as well is meant to let the webmaster know that, hey, we're going to be taking this into account. If this is something you want to work on, and mobile is definitely a giant traffic source, then that's something you should focus on early, so that you don't have to worry about this ranking change when it comes later on down the line. And these kind of changes would be relevant for all locations. Essentially, anyone who's using a mobile phone sees the same issues. If the site doesn't work well on a mobile phone, then it doesn't matter which location they're in. It's essentially a global kind of thing. Let's see. There's one about a blog here. Probably, you already know there's too much competition between blogs. So I'm going to give you a fresh example to understand my question. Let's say a blog has 30 articles about Nexus 6. Another blog has five articles. And somehow, this question got cut off. Maybe it's somewhere down the line. OK. Let's see if I find more of that later on. Why mobile site maps? I find mobile site URLs as web pages, but not as mobile pages index. The important part with mobile site maps is to keep in mind that they're specifically for feature phones. So for feature phone pages, not for mobile smartphone pages. So if you're using mobile site maps for smartphone pages, then you're essentially using them wrong. I only use this for feature phone pages, if you have specifically like WAP/WML, that kind of content on your pages. Our site struggles with Penguin. After a cleanup, we earn many new natural links, but our site is still suppressed. Is there a time frame after a site is considered normal and links an account like a site that was never affected by Penguin? Essentially, we have to reprocess the links to your site, which can take a while. And then the Penguin algorithm has to run again, which is also something that can take a while to be updated again. So this is something where you're not likely to see a big jump just after cleaning these things up. It's more gradual of a thing, and this kind of ramps up again as you clean up your site as you show us that things are really well. And it'll probably take a bigger jump when the next Penguin update happens, which at the moment, we aren't preannouncing yet. Is that an SEO factor, to rank higher the specific keyword Nexus 6? I didn't quite understand this question. I think this is a part of the other one from before. If you're watching along, maybe if you can phrase the question in a brief way so that I can see everything in one slot, that would be kind of helpful. When queries are not revealed through analytics, are the unrevealed queries the same keywords as the 10% approximately revealed queries? Or are they unrelated queries, sets of completely different keywords? They don't necessarily have to be related. It's not the case that we take a 10% sample and just show that in analytics. For some types of setups when people search, I believe we still kind of pass the referrer on. And that's probably what you're seeing in analytics. To get a bigger picture view of what actually people are searching for, I'd really recommend using Webmaster Tools with a search query feature. Or if you want to try out the new feature, make sure that you're signed up there, so that you can look at that as well. But it's not the case that this 10% is like a representative sample. It's essentially just those queries where we happen to still pass a referrer on. And that might be a specific subset of your users, that maybe you're using older browsers, or special setups, or within specific experiments in search. It's not the case that this is a representative sample of all of your queries. We're an e-commerce site from Nepal and intend to sell homemade products in countries like the US and Canada. Do we need a special sub-domain or URL, like /CA, /AU, to rank in a specific country, or is having the same website for each English country essentially also fine? Essentially, you can do this either way. So if you want to use geotargeting, if you have something specific for individual countries, if you have specific information that's kind of unique for those specific countries, then by all means, you can use geotargeting, which you can use if you have a generic top level domain, like a dotcom website you can use on a subdomain. Like, or a subdirectory,, for example. You can set the geotargeting up in Webmaster Tools for that. If you don't have unique content for each of these countries that you're targeting, you can also just use one global website and say, this is our global website. These are the products that we're selling. And we deliver these to these countries, for example. And that's essentially up to you. One advantage of using a global website is that you're kind of concentrating your efforts on that global website. You have less content to maintain on your website. It's a little bit easier to keep up with everything. And it helps us to kind of concentrate all of the signals for your website for your business on that main global website. Whereas if you really have unique content for individual countries, then splitting it up might make sense. But it's something where you have to kind of keep in mind that having individual countries on your website also means that you have more content to maintain, more things that could break at some point, that could go wrong on your website for technical reasons. So it's kind of a balance there. If you really have something great that you want to offer individually, then by all means, do that. If you prefer to get started with one global website, then that's also fine. And moving between these variations is certainly possible. So if you start off with a specific site for the US and Canada, and later decide, I'll just fold it into one global website because that's easier for me, then that's something you can also do. How does better testing a new website on the same domain not optimized for SEO, and not meant to be visible by Google on Safari only can affect SEO? Does having Google Analytics tag on a new website give access to Googlebot user agents by default? No. Google Analytics tags don't by default send that off to Googlebot. I think there are some things that we might pick up on. For example, if you have an RSS feed, and that happens to be published, then that's something we might pick up on and use for Google search. But in general, if you're blocking access to those kind of preview developer site, if you're blocking it based on the user agent, or based on the IP address, or with a login, then that's something where Googlebot wouldn't be able to kind of get around that. And essentially, this wouldn't affect us at all. If we can't crawl those pages, then we can't really pick those up and use them for search. The only effect that I think might be possible is if people are linking to those pages already, if they're recommending them to their friends, then we kind of get stuck, because we see these links pointing to these pages. We can't crawl them. So we don't really know what to do with those links. This is especially more problematic if you use something like the robots.txt to block access. Because in that case, we won't even try to crawl those pages, because a robots.txt file would say, don't crawl any of these pages on my developer site. But at the same time, if we find links to those pages, then we'll say, well, we could index these URLs individually based on the links that we have there. So if you have a developer site, and you're blocking it with a robots.txt file, then it's possible that we'll index those URLs without having to crawl them. On the other hand, if you use other types of blocking Googlebot, like based on the IP address, based on the login, those kind of things, then we would try to crawl those pages. We'd see that we can't crawl them, and we kind drop them from search immediately. But in that case, if you're blocking Googlebot with an IP address or with an authentication, then make sure you're not also serving a robots.txt file, because then we're stuck with the robots.txt file again. So the best practice is therefore kind of like a developer's site, or a testing site that you're running is really blocking by IP address, or with the server side authentication, so that we can try to crawl. We'll see that we can't get to the content, and we won't index it at all. A question about forums. Is it OK to display posts that have only 500 plus words in Google? Sure. You could do that if you want to. Will that help eliminate low quality topics? Can you give some general tips for forums and SEO besides keeping the forums spam free. So in general, I think you have to keep in mind that when we look at your website, we look at it overall. And if the overall quality of the content that you publish there is lower quality, then that's something we might take into account in our algorithms. And it doesn't really matter where that lower quality content is coming from, if that's something that you're publishing as the owner of the website, or if that's something that you're publishing based on things that people have submitted to your site, like on a forum, or blog comments, like those kind of things. So essentially, this is something where you're responsible for your website, even if its content primarily comes from other people. So if you want to make sure that your website overall is a really high quality, then I'd definitely try to take a look at what you can do to make sure that the forum posts, for example, if you have a forum, are generally of high quality, or that at least those that are indexable are of high quality. So if you know that there may be new forum users, and they're not complying with your rules, or they're kind of dropping spammy stuff on your site, and you can recognize this somehow, be it by the number of words that they submit, or be it because they're just new users and have been in your forum for, I don't know, less than a week, or less than a month, or whatever frame you choose, then doing something like allowing those pages to be crawled, but having a no index on them really helps us recognize that this is something you don't want to have indexed. And we'll therefore not take it into account when we look at the overall quality of your site. So if you can find some factors like that, it might be the length of the topic. It might be the number of replies. It might be something like the age of the poster. It might be a combination of all of these things. And using that as a proxy to automatically determine if this is probably a high quality post or not, that's something you could use to trigger a no index or not. But this is something where you know your website best. You know your forum best. You know your users best. And what works for some forums might not work for other forums.

JOSHUA BERG: Hi, John. I have a site map question I'd like to throw in regarding an anomaly in Webmaster Tools.


JOSHUA BERG: So I was reviewing a site for a client, and they have a site map, and we're getting the warnings this is blocked by your robots. Some of the URLs are blocked by robots. So they have a main site map. That's the main XML site map. And then the way they had it set up is there's three site maps under that site map, or in that. Like the first one has 50,000 URLs. So it says 50,000 warnings. 50,000 submitted, and then 49,500 indexed. The funny thing is none of the URL-- and we've checked-- were blocked by robots. So something else is going on there. And the second one also has 47,000 warnings. So that's odd. And then it also says processed, originally gave an earlier process time. So it wasn't clear that that was the most recent process time out of 2014. But anyway, now it has a more recent process time, and it's still showing those errors.

JOHN MUELLER: So what we sometimes see is that a site has some kind of fluctuating robots.txt file in the sense that either there is something on the server that sometimes sends us the wrong one, or that maybe there's a content delivery network involved, where some of the nodes have the right robots.txt file, and other nodes have the wrong one. Another one we've sometimes seen is that may be like, a developer site robots.txt file is accidentally uploaded. These kind of things. You should be able to see a bit of that in the robots.txt testing tool in Webmaster Tools, where you can also see the status of the previous robots.txt files. So I'd kind of double check there, that there's nothing kind of fluctuating happening there. If this is just like a one-time thing where accidentally, the wrong robots.txt file got uploaded, then obviously, that's fine. If this is something that regularly flip-flops back and forth, that's something where you probably want to figure out what exactly is happening there, so you can fix that problem.

JOSHUA BERG: And all URLs being indexed fine.

JOHN MUELLER: Yeah. I mean, that's theoretically possible. So if the robots.txt is temporarily blocking everything, then it won't remove the existing URLs right away. It's something that will take effect the next time we crawl. And if this, for example, comes up one day out of the whole year, then we'll still show that as a warning, as a problem, but it's not that it would negatively affect the indexing of your website, because the next day and all the days before, we'll be able to crawl normally. So it's essentially something that usually isn't a critical issue. But if it's like active or not active from time to time, that's something you probably want to chase down and make sure that whatever is causing that to happen is kind of resolved. And sometimes, we've seen that this same issue can affect web pages as well, where like sometimes, you'll see the normal homepage. And then, every 1,000th time or whatever, the user will see a 404 page, or a 403 page that says, you have no access to this page. And these are the type of things where it's really hard to recognize manually, because you reload the page a few times, and it all works. You don't realize that once a day, it's a specific time. Maybe when you're doing a backup, the server sends back a code that says, hey, nobody's allowed to access this server, or sends back the wrong files. Those kind of things. But with the--

JOSHUA BERG: I found something that when I click on it and go to it, go to the errors, that it still has detected June 2013. And when we do the test site map, it shows that there aren't any errors. So will the errors just always show there like that? Or does that mean there are still the same errors from more than a year?

JOHN MUELLER: That should go away, then. So I mean on the one hand, that means that your robots.txt file is probably OK now. So if that was in June 2013, then that's quite a while back. But that's something where I think we should be able to handle that better. So if you can send me the sample URL, then I can definitely pass it on to the site maps team, so that we can see why we're still showing these errors. Because I think, like you mentioned, that wouldn't make much sense, to show the errors from 2013.

JOSHUA BERG: And so by detected, that doesn't necessarily just mean the first time it was detected. Because I don't have another date there but June 4, 2013. So I mean, that may be long gone. But it's still just showing the errors.

JOHN MUELLER: I don't know. That seems confusing. That seems like something we could be doing better in Webmaster Tools.

JOSHUA BERG: I'll send you the link.

JOHN MUELLER: Yeah. That would be great. What would your advice be as far as procedure and what to look for on the server log file analysis, mainly regarded to Googlebot activities, Googlebot always downloading the entire file? Sometimes the size is smaller. Googlebot doesn't always download the entire file. So sometimes, it'll send a conditional header with the request, saying if this file was modified after this date when we last crawled it, then send me the file. Otherwise, it'll just get a response code saying, OK, this file hasn't been modified. So for those kind of requests, it would be completely normal to see the file size fluctuate between the full size and a very small size, where basically we're just getting the header back that's saying, nothing has changed here. You don't have to fetch the whole file again. With regards to the log file, what you should be tracking, I think it's kind of up to you. It depends on how far you want to dive into this. Obviously, depending on the website, analyzing a log file of what Googlebot is crawling is easier, or sometimes really tricky. If you have a website with millions of pages, and we're crawling millions of pages every day, then that's going to be really tricky to figure out, or to kind of analyze without using some kind of specialized tools specifically for that. So I'd say, for the most part, the average webmaster doesn't need to analyze their log files. But if you want to dig into the technical way that Googlebot is crawling your site, and understand which parts of your site we're crawling when, then by all means go ahead and do that. It's not always easy. But it's always an interesting exercise, I think.

MALE SPEAKER: Hi, John. I have a question.


MALE SPEAKER: Yeah, my question is about in the News segment of web search. Basically, what happens when Google's picks are used? And at times, what happens [INAUDIBLE] somewhere from the body, and just picks something shown there. So the headline is not there, but something else is there. So why is it happening?

JOHN MUELLER: Is that in Google News, or in the search block that's kind of like in the news?

MALE SPEAKER: Yes. The web search in the news section.

JOHN MUELLER: OK. I believe we essentially just use the normal title changes that we usually use for web search there. So we have a Help Center article on titles. And essentially, what we're trying to do is understand very briefly what this page is about. And that means that if we recognize that the title that's specified on the page is extremely long, then we have to kind of rewrite that anyway. If the title specified on the page is kind of keyword stuff, where it's repeating the same kinds of keywords over and over again, then that's something that we'd say we'd like to rewrite in a more kind of readable way. There are some other cases also mentioned in the Help Center there. But essentially, what I'd recommend doing is really making sure that the title you specify on your site, on your pages, is as short and as direct as possible, so that it's really clear that this is something we could easily show in Search. It's not too long. It matches the content on the page. And it's something that tells the user very briefly what this page is about.

MALE SPEAKER: Do you have any specific to-- yeah. I'm just giving an example. So this particular one, let's say, a movie review. So I mean, that's title says this specific, let's say, [INAUDIBLE] movie review. That's my headline of the story. So when did Google start picking up the headline as their star cast, so [INAUDIBLE]. So if my news is about the review, but the news section, particularly that news answer box showing that star cast there. So somewhere, I just feel like this is not relevant for my audience.

JOHN MUELLER: So it's picking up other parts of your pages? Or--

MALE SPEAKER: Yes. It is actually picking up my star cast. Basically, the actor and actress names in place of the headline.

JOHN MUELLER: I probably need some examples. So if you can post in the chat some and URLs that you're seeing there, then I can definitely pass that on to the team. But I need to take a look to see what exactly you're seeing there, so I understand what's happening.

MALE SPEAKER: Sure. I'll do that.

JOHN MUELLER: OK. I'll pass that onto the team to take a look at to see what we can do there to recognize the content a bit better. We're using two websites about laser engraving machines-- one for small businesses, one for corporations. The context differ. Text and images. But the keywords and company data are similar. Would a single website perform better than two separate ones? I think when you're looking at two websites and they're really targeting different audiences like this, then that's essentially a marketing question. It's not so much of an SEO question of whether you should have one website or two websites. For some kind of audiences, it makes sense to split these. Sometimes it's just easier from a technical point of view to maintain one website, which makes it easier for you, and makes it easier for our systems to kind of focus everything on just one website. So in general, if you can reduce the number of websites, that's a great thing. With two websites, I don't think it's the case that it's completely unmaintainable to have two websites. So that's something where two websites are fine. It's not something where I'd say from Google's point of view, you have to do it like this, or have to do it like that.

MALE SPEAKER: Hi, John. Actually, this question was from me.

JOHN MUELLER: Oh, great.

MALE SPEAKER: The question was because all the car manufacturers tend to have everything on one website. So I won't talk about brands here, but just imagine you're talking about personal cars and trucks. They put everything on one single website. And because everybody knows, OK, this brand-- I'm not going to name anything-- is manufacturing a lot of different sets. So same will be true for us. I mean, maintaining one website is definitely easier. Totally go along with your view. But the thing is, we are one company, and the information, the source, is just one single company. So would that be a problem from your point of view?

JOHN MUELLER: No. I think that would be fine. I mean, if you really want to separate these two sites, I don't see a problem with that. I would see a problem if you're going past five or 10 websites, and saying, oh, for each product color, I have a separate website. Then that starts looking like setting up doorway sites. But if you have two sites, and you're saying these are really different audiences that you're talking to, these are two websites with different audiences, that's perfectly fine.

MALE SPEAKER: Because a lot of external agencies, they propose that we merge them into a single one, and--

JOHN MUELLER: Sure. I mean you can do that too.

MALE SPEAKER: --there's just one more specific thing about this. We would then only have one single home page. So the home page would be about getting the audiences into the right areas. So for example, you're a small business, this is your area. You're a big corporation, we recommend this area. And now, we have home pages with very specific information already. And this helps us to rank better. Because homepages are a very influential part of the whole website, I'd say.

JOHN MUELLER: Yeah. I mean, essentially, it's up to you. You can do it either way. I don't want to say you need to do it this way or that way. With two websites, I see that as completely up to you. That's not something where from search, you have to do it like this or like that.

MALE SPEAKER: OK, got it. Thanks for your help. That's a relief for us.


MALE SPEAKER: Thank you.

GARY LEE: Hey, John. I just had a quick question. I posted a link in the chat. I'm not sure if you can see it there. It was about 10 minutes ago. It was about virtual office in Hampstead.



JOHN MUELLER: Oh, OK. As an example where--


JOHN MUELLER: OK. I copy these chats out. So that'll be-- let me just grab that. And yeah. OK. Good. Great. Thanks. That's always useful. I'll take a look at that with the team. Since we're running towards the end of time, do any of you guys still have any questions that we need to get answered?

ROBB YOUNG: Can I ask our usual question, John?


ROBB YOUNG: In regards to the conversation we had on Monday and the content, I have just a few questions, but I'll stick to one for now. The content on the old domain, because the new domain is going to be an exact copy, as you know, should the content on the old site just stay sitting there, or is that going to cause a problem, that it's basically the identical to the new one? You said, remove link connection, so there's no real connection. But would there be a content connection as such, that it sees, actually, this is basically the same one, whatever issue you're having? Are they going to see that that looks the same?

JOHN MUELLER: I wouldn't see so much of a problem there. I think you mentioned that you need to keep it up for legal reasons anyway.

ROBB YOUNG: Well, you know--

JOHN MUELLER: So I think that's generally fine.

ROBB YOUNG: I mean, we can deindex it. We don't need it spidered. Because as long as customers have existing links and existing emails and everything, and visit can read it, that's fine. We don't need Google to see it. So is that the safest? Just essentially, over time, deindex the whole thing?

JOHN MUELLER: You could do that. Sure. Yeah.

ROBB YOUNG: Could do that, or?

JOHN MUELLER: I don't see--

ROBB YOUNG: I just don't want to--

JOHN MUELLER: Yeah. I think what might be confusing for us, if it's really one to one copy, if it's like, exactly the same HTML files on both sides, then that's something where we'd say, well, this is the same thing. We have to pick one of these as the canonical. But if the old one like has, I don't know, the old headings, and the primary content is essentially the same, but not like a one to one copy of the exact same HTML file, then that's something that we would treat separately. If it's exactly the same HTML file, then that's something where we'd say, well, these are essentially the same. We'll fold them together.

ROBB YOUNG: Well, given how poorly you view the original site, wouldn't you by default, then, choose to show the new one, even though we've been struggling for two years to get you to show the old one anyway? It would be bizarre for that one to start.

JOHN MUELLER: Yeah, yeah. I mean, we would just treat it as a duplicate. So if you really want to have that connection broken, then just making sure that the old site has the old HTML and the new site has the new HTML, so that we see that they're separate pages.

ROBB YOUNG: Right. OK. I mean, we can't use canonicals, or hrefs or anything, because I don't want to have an actual connection between them. So, OK. I mean, we could probably change the HTML on the old one, actually, and let it re-index for a couple of weeks.

JOHN MUELLER: Sure. Yeah. Exactly.

ROBB YOUNG: Because it's easier to do that, because we don't care about that one. And the new one, we don't want to mess with the user.




ROBB YOUNG: Thanks, John. I'll think of another one next time.

JOHN MUELLER: Sure. All right.

GARY LEE: Hey, John. I'm back again.


GARY LEE: Hello? Sorry. Are you still there, John?



GARY LEE: OK. Yeah, so the link that I posted earlier, I'm not sure if you got a chance. I just got cut off there for some reason. But as you can see in the results, there's a very big difference between the two services, one of them being a virtual office, and the other one being a service office. They're two very, very different products, yet you're seeing as a perfect example in this that over half of the results that are coming up are related to serviced office products, rather than virtual office products. And um--

JOHN MUELLER: I don't really know that market that well, so--

GARY LEE: Yeah, well, I think that's an issue that specifically there is going on. There's also on page two, there's websites outranking us that have, clearly even in the metatags and the index data you can see, it says zero results. So it's simply just like that search results page with no results. And even that's outranking us. So I kind of feel like something there is saying that our site itself just-- even though our page is good quality and a great result, the site itself is still under some sort of penalty. Or something's going on.

JOHN MUELLER: Let me just jot down

GARY LEE: To be outranked by a page that has no results, and says in the actual meta description sort of being pulled into Google results. It's pretty poor for us to not outrank that site.

JOHN MUELLER: Yeah. I'd have to take a look at this with the team.

GARY LEE: Yeah. If you could pass that on, that'd be much appreciated, and see if they can get back to me or do something. Thanks, John.

MALE SPEAKER: Hi John. If you have some-- So I just posted this thing my search query, and under the search [INAUDIBLE] that particular result. So if you went searching through this thing. So I have URL in this chat, itself, so if you can just go to this one.

JOHN MUELLER: OK. Oh, and you mean the Michael--


JOHN MUELLER: --Edward Norton, that kind of stuff.

MALE SPEAKER: Yeah. Exactly. Yeah.

JOHN MUELLER: All right. Let me just take a quick screenshot so that we have it. All right. Yeah. That looks like a weird title. That's probably something we should be able to do better for.

MALE SPEAKER: Yeah. This is basically something we are seeing of this thing. I mean, like earlier it was picking the right headline, but now it is picking up the star casting, so there is something [INAUDIBLE] us.

JOHN MUELLER: Yeah. And that's like an element on your pages? Is that right?


JOHN MUELLER: Yeah. I can take a look at that with the team here to see where we're picking that up. If you have a thread in the Help forum, for example, or on Google+, then I can perhaps give you some feedback there. I don't know if there will be something that we can tell you, or that you could change there. But it might be useful. So if you have a thread somewhere, maybe just copy that into the event listing, and I can take a look, and see if I can reply there as well.

MALE SPEAKER: Sure. Sure. I'll do that.

JOHN MUELLER: All right. Then let's take a break here. Lots of questions. Good conversations. Not so focused on Penguin and links, which is great. Something nice to have, and some new faces as well. So maybe we can keep that up. But thank you, regulars, for joining in as well. All right. Then I wish you guys a great weekend. Thanks again for joining. And hopefully, we'll see you guys again in one of the future Hangouts.

MALE SPEAKER: Thanks, John.

MALE SPEAKER: Thank you very much.

MALE SPEAKER: Thanks, John. Thank you.

JOHN MUELLER: Bye, everyone. | Copyright 2019