So I’m a Knucklehead eh?
This small post is mostly in response to an article I saw this morning in my feeds: Building a search engine to rival Google could cost billions — and that’s not the only problem by James Purtill.
Challenge accepted!
I challenge anyone with a few spare tens of millions of $AUD to come talk to me about this. Put your money where your mouth is (so to speak). Does it really cost 100’s of Billions of dollars ro build, crawl and index the Web? I think not!
What, wait? Who are you?!
Who the hell am I you ask? Really?! You’re crazy!
Maybe 😉
I don’t expect most of you to know me too well (except those that do!), I’m just a software engineer that has worked in quite a few places over a period of more than 15 years, including several years at Facebook Inc. I’m very familiar with the kinds of Infrastructure and Engineering it takes to build “Web Scale” products like Search or a large-centralised Social Media!
It is quite sad to read other failed attempts at building a search engine, mostly Government backed attempts. The reality is a lot of government backed projects tend to be inefficient and the funds wasted before the project even has a chance to get off the ground or suffers from political issues (example: NBN Co in Australia where many of us still have Copper Last Mile over *DSL).
What would it take?
What would it cost to build an Aussie search engine? Well I put together one over a weekend called Spyda because my wife asked me to! I said I’d have it ready by Monday. It was a day late and launched on Tuesday.
Spyda is available at spyda.dev where you can currently search a measly 1,000+ pages. We’ve only been able to crawl and inex a thousand pages so far. This is becuase it runs on my home infrastructure which has an ordinary 100/40 Mbps VDSL NBN Co connection backing it (behind Cloudflare).
In other words it cost a day of my engineering time to put together, plus the cost of my home infrastructure (with initial outlay of $10k and $5k maintenance over the years).
So basically nothing so far to prove the concept is feasible. 😝
What now?
So let’s scale this up!
If someone were to approach me tomorrow offering to invest in my expertise through a new company founded to compete with the likes of Google, Facebook, Twitter and all these other tech-giants that have successfully built up a multi-Trillion dollar empire by profiting off the data collected from their “users”, come talk to me!
Let’s build an Aussie search engine, that’s Aussie built and owned, but does not sell its users data!
You can reach me at James dot Mills at shortcircuit dot net dot au – Or just Google™ me 🤣
