Home – New Forums Tech talk A question for you SEO guys

  • This topic is empty.
Viewing 9 posts - 1 through 9 (of 9 total)
  • Author
    Posts
  • #977963
    BrettM33
    Participant
    • Total posts: 1,372
    Up
    0
    ::

    Just wondering if you block access to your whole site to Google/Bots with robots.txt such as..

    User-agent: *
    Disallow: /

    But then want to allow them back, such as:

    User-agent: *
    Allow: /

    Does anyone know how often they check back after they have seen that they have been disallowed?

    Thanks!

    #1103754
    createdevelop
    Member
    • Total posts: 171
    Up
    0
    ::

    The crawl rates can be set in your webmaster tools, or through your telling search engines how often you update through your sitemap.xml.

    On most sites, search engines will check every day. You can make sure they check by sending them a “ping” either through their own method or through some sort of bulk ping service.

    If you have disallowed access to all, it might take a few days for a search engine to do a complete crawl, and index all of you pages. If you are a brand new site this might take a week or so.

    #1103804
    createdevelop
    Member
    • Total posts: 171
    Up
    0
    ::

    The crawl rates can be set in your webmaster tools, or through your telling search engines how often you update through your sitemap.xml.

    On most sites, search engines will check every day. You can make sure they check by sending them a “ping” either through their own method or through some sort of bulk ping service.

    If you have disallowed access to all, it might take a few days for a search engine to do a complete crawl, and index all of you pages. If you are a brand new site this might take a week or so.

    #1103757
    RAds
    Member
    • Total posts: 126
    Up
    0
    ::

    They’re very invasive little bots – they’ll try anything to index your site, and they’ll even keep probing at 404s in the hope they come back for YEARS.

    Its actually much harder to get them to drop content. I had a penalty on one of my sites for a very long time after the bad content was gone, only went away when google itself actually did a cache reset on one of its updates.

    #1103805
    RAds
    Member
    • Total posts: 126
    Up
    0
    ::

    They’re very invasive little bots – they’ll try anything to index your site, and they’ll even keep probing at 404s in the hope they come back for YEARS.

    Its actually much harder to get them to drop content. I had a penalty on one of my sites for a very long time after the bad content was gone, only went away when google itself actually did a cache reset on one of its updates.

    #1103759
    BrettM33
    Participant
    • Total posts: 1,372
    Up
    0
    ::

    Haha thanks guys…… was aware you could set the crawl rate, just wasn’t sure if it differed once you told them to “go away”.

    #1103806
    BrettM33
    Participant
    • Total posts: 1,372
    Up
    0
    ::

    Haha thanks guys…… was aware you could set the crawl rate, just wasn’t sure if it differed once you told them to “go away”.

    #1103761
    Vilson
    Participant
    • Total posts: 184
    Up
    0
    ::

    Hi Guys,

    User-agent: *
    Disallow: /

    if you set above then also Google crawl your site and but it is not doing indexing and not showing in search results.

    But then want to allow them back, such as:

    User-agent: *
    Allow: /

    Now Google will do crawling, indexing and show on search results.

    Create Online Test or Quiz on ScoreBell
    #1103807
    Vilson
    Participant
    • Total posts: 184
    Up
    0
    ::

    Hi Guys,

    User-agent: *
    Disallow: /

    if you set above then also Google crawl your site and but it is not doing indexing and not showing in search results.

    But then want to allow them back, such as:

    User-agent: *
    Allow: /

    Now Google will do crawling, indexing and show on search results.

    Create Online Test or Quiz on ScoreBell
Viewing 9 posts - 1 through 9 (of 9 total)
  • You must be logged in to reply to this topic.