this post was submitted on 20 Jan 2024
168 points (88.2% liked)

Futurology

1750 readers
49 users here now

founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[–] Endorkend@kbin.social 4 points 9 months ago

No need to invent.

That's how originally search engines, including Google, Yahoo and all the other big ones worked.

You didn't get indexed by default.

You either got indexed by being submitted or by being referenced often by one or more well represented sites.

It's only later in the game they started crawling everything.