SEO: Technical Guidelines From Google

Listen to podcast:


Good morning internet fans, Ryan Perry here Simple Biz Support. Today is Thursday, therefore, it is Internet Marketing Thursday and I have Virginie Dorn with Business Website Center. I apologize for starting this episode a little bit late. We were dealing with some technical issues where Virginie’s screen or her video wasn’t popping up. We could hear her, but the screen was all black so we were trying to fight through that and… But we’ve succeeded. We’re here, Virginie is here and good morning Virginie. Now, I can’t just hear you ’cause you are muted.

Virginie Dorn: Oh, gosh. Okay, I’m back, my voice, my visual is there and all I did is close the browser and restart it a second time and there it stopped working. Go figure.

RP: Very, very weird. So it is… Technology is wonderful, but it is what it is. I was actually… I found a link this morning about a computer show back in the ’90s and they were talking about the internet. And one of their presenters, was a gal from CompuServe who was one of the leading… I don’t know, not users, but however you access the internet. And it was funny because it wasn’t… They didn’t talk about it as WWW, it was always the World Wide Web and just FTP, File Transfer Protocol. They had to explain everything. So it was very interesting.

VD: It was not that long ago, we were all on a dialup. I don’t know if you remember.

RP: Oh, I remember. That’s how I started out in the AOL days. So technology doesn’t always play nice even though it’s so much better than it was back in the ’90s. We are going to continue our SEO conversation today. We are gonna get a little geeky, we are gonna get a little technical, this is gonna be more on the programming side. And I think this episode is really important for people who feel they have a very good looking website, they have been creating content that is keyword loaded. Maybe they are still not getting visibility and their maybe a technical issue why there website is not being indexed or ranked by Google. So we are going to get kind of into some of the nuts and bolts behind the scene stuff that you may not see, but it’s still very, very important.

VD: Yes. So once again Google is giving us a number of 10 guidelines on a purely technical, just as you mentioned. We won’t be able to explain in great detail what each of them are because of the limited amount of time we have, but at least if you could understand in general what they are, you can go back to your webmaster maybe have him watch that webinar and make him informed and browse those 10 recommendations from Google on the technical side of your site. And if you don’t do those, you would be penalized by them for your ranking.

VD: So if you want, we can just get started on the first one, then Google mentioned, is to… Unless really necessary, to not put any code into place to prevent their crawler, which is the Googlebot, to crawl every single file of the website. And that’s an old technique when perhaps people would just want to protect their images. They don’t want Google to crawl certain files for some odd reason and Googlebot doesn’t like it. So in the best case scenario, you want to allow Google to crawl every single file and not just pages of content, but also the files that render that content. So Google can take a look at it, so you can get a very clean understanding of what the website structure is.

RP: Right. And the… If I’m not mistaken, is the only way you can do that is through your text file correct? Your…

VD: Robots?

RP: Yeah. The ‘robots.txt‘ file?

VD: No, no. Actually, there are scripts you can put in the head of your code to prevent Google from crawling. So there has been techniques to fool codes preventing to view, I don’t know, maybe the image for odd reason. Again, Google doesn’t like it. And not putting this kind of code in the robots text file, is the fourth recommendation. So they talk about not preventing them from crawling twice in their recommendations. That must be quite important to them because they repeat themselves and that’s…

RP: Right. And it’s interesting because the ‘robots.txt’ has been out for probably I don’t know, the beginning of time and I always found it interesting because it’s really, it’s kind of like the pirate code. It’s more of a guideline because you put something in the ‘robots.txt’ file, doesn’t mean that certain browsers are going to necessarily adhere to those instructions. They may go through and crawl everything if they want to anyhow. At least that’s my understanding.

VD: I can’t say because I don’t know behind the scene of Google and other search engines, if they can’t overpass or override those restrictions. But the thing is clear, is Google doesn’t like to be prevented from crawling any files on your website. They want full access so let’s give it to them unless there is a really specific reason why you would not want them to crawl those particular file.

RP: Right. And the reality is, if you want to be found on the internet, then you really shouldn’t be hiding anything anyhow. Now if the opposite is true where you are, “Hey, I’m not worried about Google search. I don’t necessarily want to be found.” then go ahead and do what you got to do in order to protect your information. So what’s number… So we covered number one, number four, simply because they are both relevant to the same issue. And then number two…

VD: Has to do with tracking movements on your website. Often times that technique is used to track the movement of your regular visitors, the humans, where do they go, how long do they stay? Think of a kind of analytics type of programming you might have, Google doesn’t like to be tracked, eventually telling us on that second recommendation, “We don’t like to be tracked, so put something in place that doesn’t track the movement of our bots. We don’t want you to know where we go, how we go to places.” And it’s unclear as to why that’s their second recommendation, but nonetheless it is. If you want tracking of the visitors movements on your website, why don’t you use Google Analytics? After all, it’s one of their products, so it should not be penalized by Google, since they’re the one offering it. So, yes, this has had us wonder here in the office as to why that was such a big issue of wanting to be tracked.

RP: Maybe it’s just a way of keeping other analytics off people’s websites.

VD: Yeah, possible, you know what? I wouldn’t be surprised.

RP: We’ll start a conspiracy.

VD: There’s plenty out there about Google.

RP: Yeah.

VD: The third recommendation they say, “If you have concerns about the bandwidth of your website and Googlebot taking too much of that bandwidth when it’s crawling to look for good indexation, then you can put something into your code that tells Google when the last time it was updated.” So if it says it was not updated since June 1st and they already crawled you since June 1st, they’re not going to crawl your website one more time. So that’s their recommendation, “If you don’t want us to crawl it as we please, just tell us when was the last time it was changed, and based on that information, we’ll decide if we should crawl it one more time.”

RP: Right, because my understanding is they have two settings, one is just automatic, “We’ll figure it out depending on how popular your website is,” essentially how important it is to them. And then number two, is you can kind if dictate when you would like them to crawl. But what I always found interesting about that is that I’m sure if you said, “Hey, crawl me everyday,” because you’re adding new content, if they don’t want to, I doubt they’re going to.

VD: That is correct, you can always request it from them, that doesn’t mean it’s going to be granted. But yeah, I always wonder as well. Though, what we do like to say is that if you do create a new page, you should submit that unique page that is brand new, directly to Google, and have it crawled, so you can do that for your webmaster account. And that’s true for other search engines like bings and so forth.

RP: Right, and I think even better than that, all you need to do is put that page out on Twitter or Facebook because Google lives in both of those areas, simply because there’s so much content being created all the time, that they will pick up that URL really quickly on those platforms, and if they stick it in their database and they go, “Oh, this page hasn’t been indexed,” then it’s a real easy way to kind of let them know, “I would like this page to be added to the index.”

VD: Very true, I recommend that as well. Now, the fifth recommendation, it’s something funny again, they say, “Sometimes your online advertising can hurt your organic optimization,” and they mention AdSense. Then sometimes, the script you put into your site for your, again, online advertising, can again prevent their crawler from crawling your webpages. So, they’re giving us the heads up, like, “Make sure that is not happening.” So of course, for that kind of technical guideline, you have to refer back to your webmaster and ask him or her to go into the script and see if there’s anything in place that is telling Google not to crawl that ad, because again, it confuses their bots and Google doesn’t like it.

RP: Right, that’s interesting, I hadn’t heard about that. I know that earlier this year or last year, I’m trying to remember which roll out of the Google Analytics… Or… I got Analytics stuck in my head. You know, like Panda, one of the algorithm updates where if you had too many ads above the fold, they would actually penalize you as far as your ranking, it could adversely affect your ranking, but I wasn’t aware that those ad platforms could possibly block the robots, which is pretty interesting.

VD: Yes, and I think it goes back to the analytic portion of that code, and perhaps the AdSense script, some of them might tell, “Well, we want to have a true number, so please do something about the bots, don’t clip part of the numbers.”

RP: Right, I was gonna say, because that can be frustrating when you’re looking at analytics and when the bot numbers get crammed in there, it’s a little frustrating, especially now, I cannot think… There are a couple of websites out there that generate quite a bit of traffic, at least I’ve seen it on a number of my clients, a number of traffic, and it’s all junk. And then if you go into the referral base, you’ll see that it’s website, and it’s some marketing website of some sort that’s just… I don’t know what they’re doing, but they’re really screwing with the numbers. And then you have to go through and really go through your numbers to find out what the real true numbers are without the bots in it.

VD: So it’s a give and take, either you get a better number and you prevent the crawler from being counted, or you please Google and let them do what they please and don’t disregard them.

RP: Yeah, and the key thing is, they’re all workaround on these type of issues, but again they’re more technical. This is somebody, if you don’t have a lot of technical experience, if you don’t understand coding, if you’ve never heard of a ‘robots txt’ file, then these are things that you don’t wanna go and just do a Google search and try and fix yourself because you could end up creating more problems. However if one of these problems exist, it really could be an indication of why you’re not being found on Google search.

RP: Is essentially,

[A], Google’s not able to index you, therefore they don’t know that your content exist, or, you’ve got a couple of things in there that’s just frustrating them, and they’re going, “Look, your competitors’ website doesn’t have all these roadblocks and they’re making it easier for us to rank them, so that’s what we’re going to do.” The live version you said you’d definitely wanna get somebody who has technical experience, who’s been dealing with this for awhile because this is geeky stuff that you don’t wanna try and figure out and fix, especially on your very own website.

VD: Yes, and to wrap it up, there’s still some guideline, but the truth it’s a bit of a repeat again because six and seven has to do with stopping them from crawling every page. The six has to do with if she had a content management system, make sure then the pages generated by your CMS don’t have that code, then again prevents the Googlebot from doing its job. Number seven has to do with the robots.txt file, in the best case scenario, do not want you to put anything in place to prevent them from crawling anything they want. So again, I think there are four or five guidelines out of ten that has to do about being preventing them from crawling every time they please. So it must be very close to their heart if they keep repeating themself.

VD: Now to quickly wrap it up ’cause I know we’re running out of time is number eight: Is about browser compatibility. They mention there are many big browsers out there, Firefox and Chrome, and now we have Microsoft Edge. They want you to make sure it renders properly on all browsers, and then they would know if it doesn’t look good on Edge or Firefox. I don’t know how they would know ’cause that’s to do with the visual aspect of it, but that’s nonetheless, their eighth recommendation.

VD: Nine is about page speed, they want you to have pages and download as quickly as possible. It’s not just important to them, but it’s important to your visitors, especially those using mobile devices. And to wrap it up at number 10, is that it gives you just free links to page speed test you can do online to test your own website. And don’t just put your homepage, maybe put your photo galleries that are tend to be a little slower and see if it rates properly. So all those 10 guidelines are available if you Google, ‘Technical Guidelines From Google.’ You’ll be able to go directly to them or again watch that video or watch Ryan’s post.

RP: Perfect. And then the other thing I wanna go back to real quickly… Yeah, our time is up, is number six with the CMS. So if you’re working with WordPress, there is actually a button in there and it’s a radio button, “Do you want to block Google?” Yes or no type of deal. And I’ve actually seen that turned on in a couple of different websites. So I forget exactly where it’s located in the settings, but there is a button there that basically will turn you off, and if that button is on, you’re never going to be found regardless of what you do.

RP: So perfectly, or perfectly Virginie… We’re gong to work on my English here. That’s it for today’s show. We’re gonna continue our SEO conversation next week, we’re going to keep kind of in the technical realm. And all the information that we’ve been pulling from the last couple of weeks and the next couple of weeks going forward, it’s all straight from Googles’ guidelines. So this is some of the boring encyclopedia stuff that a lot of people don’t read, but it’s really important because this is coming from the horse’s mouth going, “Look, these are the things that you need to do if you want to rank well.” Virginie, as always I appreciate your time. We will see you next week.

VD: That sounds wonderful. Take care.

RP: All right everybody, that’s it for today’s show, and we look forward to talking with you next week.


About the Author:

Ryan Perry is the founder and CEO of Simple Biz Support, Inc. Ryan started video blogging in 2009 as an alternative to written blogs to create visibility and credibility online. During the workweek, he enjoys helping small business owners harness the power of video to grow their companies. On the weekends, he enjoys hiking and searching out waterfalls throughout the state of California.

Leave A Comment