Hi all, it's been a while since i last blogged about stuff and the main reason was that i was busy with putting a team together, getting that team to build a big data platform system. I believe that work is iconic in the respect that no company i'm aware of, in Singapore or the Southeast Asia region worked on anything remotely like it and now the product's closest competitor is IBM's "Big Insight" with Pentaho's & Oracle's product suite tailing us closely. But enough of patting the team and myself on the back. This post is about a braindump of the lessons i've learnt - needless to say, the views expressed on this post is solely my own & represents how events are interpreted from my point of view.
You've read a lot of stories about how Mr/Ms So-so put together a great team of people, blazed through the "stars" and out came a software product. Well, in my case it didn't happen quite like that ...
Putting a team together is hard work; anyone who has ran a startup in Asia (in particular Hong Kong & Singapore) knows it can be a uphill task when it comes to hiring experienced software engineers. In our case, it's a little more challenging that the usual since what i'm looking for are software engineers who are experienced in building distributed systems (in addition to using distributed systems). I'll discuss more about this in the next post ...
A big data system needs to be able to scale.period. I should also mention it needs to be highly available - and it took us a fair amount of work to get it to a state where the entire can be deployed, automatically, orchestrated from anywhere on Earth (with an Internet connection, of course). The architecture team consulted countless papers from the usual sources e.g. usenix, acm, google research, facebook research etc. to get not only an understanding of the landscape but more importantly the tradeoffs. Distributed system aficionados know that building a 100% fault-tolerant is close to impossible - my/our goal was for a different purpose.
I'll discuss this in a future post; in case you like to know what we did a high-level fashion i have provided the brochure of our product here.
Bootstrapping a team
You've read a lot of stories about how Mr/Ms So-so put together a great team of people, blazed through the "stars" and out came a software product. Well, in my case it didn't happen quite like that ...
Putting a team together is hard work; anyone who has ran a startup in Asia (in particular Hong Kong & Singapore) knows it can be a uphill task when it comes to hiring experienced software engineers. In our case, it's a little more challenging that the usual since what i'm looking for are software engineers who are experienced in building distributed systems (in addition to using distributed systems). I'll discuss more about this in the next post ...
A Big Data System Architecture
A big data system needs to be able to scale.period. I should also mention it needs to be highly available - and it took us a fair amount of work to get it to a state where the entire can be deployed, automatically, orchestrated from anywhere on Earth (with an Internet connection, of course). The architecture team consulted countless papers from the usual sources e.g. usenix, acm, google research, facebook research etc. to get not only an understanding of the landscape but more importantly the tradeoffs. Distributed system aficionados know that building a 100% fault-tolerant is close to impossible - my/our goal was for a different purpose.
I'll discuss this in a future post; in case you like to know what we did a high-level fashion i have provided the brochure of our product here.
