I was wondering if the search engines are indexing the fediverse. Because recently Google search became so bad so I was always putting Reddit at the end to get some not so generic results and I haven’t even noticed to get any replies pointing to the fediverse ever. So I was wondering if search engines were actually indexing the content here?
Some platforms like Mastodon allow individual users to discourage search engines from indexing their profile. But, by default, as far as I know all platforms allow indexing. Lemmy seems to not provide any option to control this kind of thing so everything should get indexed.
The thing is, using “thingtosearch reddit” you’re not using any search engine properly, that’s kind of a hack. What you would do is actually “thingtosearch site:reddit.com” to limit searches to a specific site. This works with any site, of course, so you could for example do “thingtosearch site:feddit.it” (that’s my instance), and you will get specific results (which actually might include results from other instances, due to how this indexing works, even though they will be displayed from the site of the instance you specified). (I just noticed btw that DuckDuckGo doesn’t list anything for site:myinstance… well, that’s strange, Google has no problem.)
DuckDuckGo uses their own web crawlers along with a few others (but not google’s). Probably just haven’t indexed it yet.