• BrikoX@lemmy.zip
    link
    fedilink
    English
    arrow-up
    4
    ·
    1 year ago

    They don’t pretent to be googlebot, they use their own crawler they just don’t share the name they use for it, so sites can’t exclude it with robots.txt. They just scrape the same sites that googlebot does, so if the site is excluded by googlebot they also skip it.