Posts tagged: megabus

The Elite Squad

The tagline Fearless, the Elite Squad Team and the phrase ‘impossible is nothing’ was all put together one evening at Kraya…

It was an interesting evening and it harked back to a time a few years earlier.

The first project where we had to pull out stops was the pilot project for megabus.com.

The original version of the site was based on an idea and needed to be built from scratch and taken live within a very short timescale. Working on this was exciting, not just because of the adrenalin rush to be working on a project that could change the world (which it did promptly), but also because of the enormous amount of pressure to succeed.

We have since been involved in several such high pressure projects where failure is simply not an option. To date, we have delivered.

My favourite example of this is the work we did last year for the Edinburgh Fringe Festival.

The fringe had a serious issue on their hands about a week into their programme launch in that they were unable to make sales due to problems with their new ticketing system.

On the Thursday (12th June) evening the Fringe called asking if we would be able to build a booking system for them over the weekend. John who answered the call (and initially thought they were kidding) roused the troops.

This started with a meeting on Thursday late evening with the Fringe to identify exactly what needed to be done and the best way to move forward.

We had already been involved on a limited basis in trying to resolve the issues with the booking system. We had also integrated the booking system into the website. This meant that we had a good idea of what was involved.

We had also built the website in such a way that it was fairly straightforward to plug another ticketing system in.

With these things working for us, all we had to do was put together a booking system over the weekend.

In reality, we started work in earnest on Friday morning and the site went live on Tuesday morning. It took four days and about five or six of us. We ran into numerous problems and hurdles, a large number of which were outside of our control including problems with the data.

The edFringe website went live on Tuesday, facilitated online bookings, and we also set up a section within the website for agents to sell tickets over the phone or in person.

All over four days. The last part was the hardest, working about 36 hours straight apart from a wee nap…

Sleeping Shri

Sleeping Shri

There were a few issues that cropped up and had to be resolved over the next few days which was expected considering the timescales involved.

Over the first week or two, the sales had almost caught up to the similar numbers as the previous year. The site was making all the bookings, not just a percentage (as it was the previous year) and it held up with minor issues. Load on the systems did not cause it to fail at any point.

We had completely revamped the infrastructure on which the site was hosted to allow for the kind of traffic that was expected. It was unfortunate that we didn’t really have a chance to take that live properly but it did help to reduce the number of issues the site faced.

The interim system we put into place was since replaced with the original booking system around mid july.

We are responsible for keeping megabus.com running day in and day out and it sees over 100,000 visitors every single day. With this kind of pressure being the basic level of service delivery that we do, there is a certain level of excitement and adrenaline rush that comes from doing something that seems so far out of reach and impossible.

We are currently involved in another project of a much higher caliber that I’ll write about once its completed. Projects like these remind us that we are really are fearless and impossible really is nothing…

Database Systems Compared

My first experiences of a computer started with DBase III+ which is now dBASE, then went on to Foxpro, now Microsoft Visual Foxpro. I have since used Filemaker Pro, Microsoft Access, Microsoft SQL Server, MySQL, PostgreSQL, SQLite and HSQLDB. I have not yet used IBM DB2, Oracle. Wikipedia has a list of database systems.

Having worked with this range of database systems and having done copious amounts of research into DB2, Oracle and other DB systems I have not mentioned, I like answering the age old questions. Which is the best database system?

Ah! if only it was that simple. There is no database system that is appropriate for any given requirement. But then, if you have been in the technology sector long enough, you would already know that. It’s all about using the right tool for the job.

I separate these systems into two broad categories and Oracle. There are the Desktop based database systems:

  • DBase
  • Foxpro
  • SQLite
  • HSQLDB
  • Filemaker Pro
  • Microsoft Access
  • MySQL

DBase, FoxPro, Filemaker Pro and Microsoft Access are essentially a GUI frontend that has a database backing.

Access is the best choice for this purpose under the majority of circumstances. Filemaker Pro is relevant in some. The usual reason to use DBase or FoxPro is simply that the developer is used to it. This is not a good enough reason.

I have used DBase III+ for developing an office management suite back in 1994. I have since used Filemaker Pro to develop a simple contact management database in 1998, Microsoft Access to develop a patient management system for a clinic.

SQLite, HSQLDB and MySQL are database engines that are to be utilised by popping a frontend on top; sometimes the frontend is Microsoft Access. Microsoft Access can also be used for its database engine.

Access is usually the worst choice for this except as a stopgap. There are exceptions to this. One is for a web frontend if the site is not too busy and its running on a microsoft platform. You don’t have to go to the hassle of installing anything on the server. The drivers will take care of it all.

HSQLDB becomes an obvious choice for a light java based application and SQLite for any other lightweight applications.

MySQL is substantially more powerful and scales a lot better. I include it in this section because it is a server grade database system that can also work well in a desktop environment.

I have used Access for several web based systems and I have used HSQLDB for unit testing hibernate and for a quick and dirty MP3 library that linked into musicBrainz. I have used SQLite in passing to be utilised by open source products.

I have used MySQL with an Access frontend as a management suite for a website as well.

And we have the server based database systems:

  • MySQL
  • Microsoft SQL Server
  • IBM DB2
  • PostgreSQL

MySQL was used as the backed database system for the edFringe.com website. This was the perfect choice since the most important requirement was speed. Particuarly with the Query Cache and Master Slave replication, MySQL was the best choice.

SQL Server was used as the backend system for an online course for the Scottish Enterprise around 1999/2000. While MySQL would have been a good choice this, it was not of production quality at the time.

We have also used Ms SQL Server for an insurance company since all the infrastructure was based on Windows and PostgreSQL did not have a viable Windows version at the time.

We use PostgreSQL for megabus. While speed is absolutely critical, it is a ticketing system which means that transactionality is absolutely critical.

While MySQL now has transactionality with innodb, it is still nowhere near as good as the transactionality provided by PostgreSQL through MVCC (Multi-version Concurrency Control). We could have used Ms SQL Server but the cost savings are dramatic.

To summarise, each system has a specific use, specific strengths and weaknesses and which should be used is highly dependent on what it is to be used for. I am hopeful that the summary of what we have used each of these systems for us useful in determining which one is best placed to solve any specific problem 😀

We have not yet used Oracle and it was a strong contender for megabus but the serious heavyweight functionality provided by Oracle comes at a price and it is not yet a cost effective option.

Making Twitter Faster

From my perspective, Twitter has a really really interesting technical problem to solve. How to store and retrieve a large amount of data really really quickly.

I am making some assumptions based on how I see twitter working. I have little information about how it is architected apart from some posts that suggests that it is running ruby on rails with MySQL?

Twitter is in the rare category where there is a very large number of data being added. There should be no updates (except to user information but there should be relatively very small amount of that). There is no need for transactionality. If I guess right, it should be a large amount of inserts and selects.

While a relational database is probably the only viable choice for the time being, I think that twitter can scale and perform better if all the extra bits of a relational database system was removed.

I love challenges like this. Technical ones are easier 😉

If I didn’t have a lifetime job, I would prototype this in a bit more depth. Garry pointed me in the direction of Hadoop. Having had a quick look at it, it can take care of the infrastructure, clustering and massive horizontal scaling requirements.

Now for the data layer on top. How to store and retrieve the data. HBase is probably a good option but doing it manually should be fairly straightforward too.

From my limited understanding of twitter, there are two key pieces of functionality, the timelines and search.

The timelines can be solved by storing each tweet as a file within a directory structure. My tweets would go into

/w/o/r/d/s/o/n/s/a/n/d/

The filename would be -

For the public timeline, you just have a similar folder structure, but with the timestamp, for example, the timestamp 1236158897 would go into the following structure as a symlink

/1/2/3/6/1/5/8/8/9/7/

For search, pick up each word in the tweet and pop the tweet as a symlink into that folder. You could have a folder per word or follow the structure above.

/t/w/i/t/t/e/r/- OR

twitter/-

You would then have an application running on top with a distributed cache with an API to ease access into the data easier than direct file access. Running on Linux, the kernel will take care of the large part of the automatic caching and buffering as long as there is enough RAM on the box.

This can in theory be done without Hadoop in between and separating the directory structures across multiple servers but that can have complications of its own, especially with adding and removing boxes for scalability.

You are also likely to run into issues with the number of files / sub-directories limits but they can be solved by ‘archiving’ – multiple options for that too…

Thinking about this problem brought me back to the good old days of working on the search mechanism within megabus.com. We needed the site to deal with a large number of searches on limited hardware when the project was still classified as a pilot.

With some hard work and experimentation, we were able to reduce the search time to a tenth of the original time.

I’ll admit that I don’t know the details or the intricacies of the requirements that twitter has. I have probably over-simplified the problem but it was still fun to think about. If you can think of problems with this – let me know; I wanna turn them into opportunities 😉

99 Problems

I thoroughly enjoyed reading Nathans post about Sunday Softness. There are so many interesting points that I would like to make about this – even though it’s 3 in the morning… 😉

Shri – Soft? from an article titled Destroy Something? Its no surprise that both my post and Chris’s seem similar since they are just two approaches to problem solving… I’ll clarify that in a minute… 😀

Trying something new is something that I thrive on. In fact, I believe that is one of the reasons for me doing what I do today… and every day… I like to think that Kraya is a team of problem solvers – that is the biggest part of our job. We have been involved in solving some pretty heavy and serious problems. How do you design and develop a ordering system in 5 days was one of the interesting ones… Not to mention the megabus.com pricing algorithm.

To be able to solve problems – you need to have critical skills. One is to analyse and understand a problem and the second one is to have tools that can be used to solve that problem efficiently and effectively.

The reason I personally, and Kraya have such an exposure to such a vast array of technologies is that we need the tools to solve a problem. We might not need it today or tomorrow but we might need it one day. On that day, we might not have the time to learn it all…. I know that the majority, if not all of the people at Kraya have already got their hands dirty with a large number of technologies. I would also like to think that they have been exposed to some new technologies since they joined Kraya. In fact, if this has not happened, my job has not been done…

Now, to separate the problem solving technique I covered from Chris’s. We are both covering ways to understand a problem space. Chris suggests understanding your problem space by knowing it, understanding it and loving it. I agree with this. Understand the problem and you can solve it.

When I suggest you try to:

break out of your habits, out of your box, destroy your preconceived ideas and look at it differently, try something new…

What I am suggesting is that you consider the problem as a black box and throw something unexpected at it and see how it reacts. This will not solve the problem. In fact, it might break it further but you can learn something new about the issues at play and therefore give you a clue as to how to solve it.

I’ll demonstrate with an example.

I wanted to set up IDENT authentication for postgresql so that we didn’t need to store password on text files to log into the database server but it was not working… I had the ident server installed but it just gave authentication failures. I tried replacing the ident server, tested the ident server and it still failed. Tried restarting the db server and still no joy.

The log files showed that it was requesting auth and it came back with the correct response. The db server was simply ignoring it.

I was stumped. What else could I try? Everything was set up and it should have worked. There was no way to understand the problem space better. I had two choices… Set up an ident client and test the ident server thoroughly, then setup a custom ident server and test the client.

Instead, I simply switched off the ident server. Something that definitely broke the auth on the db server.  I broke it in a controlled fashion to try to identify where the problem lie. Sure enough the db server logs told me that it could not find the ident server. This helped me to understand the problem was at the db server end.

In fact it pinpointed that the problem probably lay in the one line in the configuration file that specified the ident authentication configuration and sure enough it was. I solved it.

This is obviously a trivial example. However, the point I am trying to make is that while it makes sense to understand exactly what is going on – sometimes, this can be a little too time consuming and laborious.

In these circumstances (and it might be true that I take this approach more often than is “safe”), a good approach is to see if you can break it further or how you can break it further as a quick way to identify where the problem may llie.

I believe that this also applies to people and have to disagree with the following comment

to suddenly change the way one of you is and do something different can’t possibly work; it’s breaking one of the foundations on which the relationship was built.

This might be because, I, as a person tend to be more unpredictable and variable than most… at least until this becomes expected behaviour. (You might be surprised at the number of times I am asked – “what have you done now Shri?”)

When a situation (or relationship) has reached a state of staleness or stalemate, you have two options. Do what you can to understand the problem(s) and take steps to solve it. Alternatively, throw a snapper (or a spanner if you prefer) into the works and see what falls out. You can then proceed to fix it…

No prizes for guessing which approach I usually take. 😉 To create something beautiful, you often have to destroy something first… >:D

More often than not, when faced with a problem, I often look to see how it could be broken further before figuring out the best way to solve it. Understanding a problem is solving a problem. Use all the tools at your disposal (and then some) to understand a problem. The better you understand a problem – the better you can solve it… :-)

So, break something today (and then fix it)…

Censoring Law

Krish asks a good question about censorship & blogging.

Unsurprisingly, it was me who broke the rules and made an overtly political statement which Krish had to censor that raised this point.

Censoring is such a grey area. Less grey in a corporate environment but grey nonetheless. My last post about Tinkerbell is for the benefit of the Kraya crew and of course the Mascot himself. However, the question has to be asked as to how it will be perceived by our (potential) clients and our (potential) new staff…

Some of our clients (the ones for whom we do a lot of work) know that we are off the wall. In fact, the reason we are able to do some of the amazing things we have done is because we are a little crazy and off the wall… :-)

It has been made clear in Kraya that it is a lifestyle and not a job. The crew at Kraya are so fantastically good at what they do, it would be unfair (at best) to restrict them with rules, structure and guidelines. All geniuses were eccentric at best or outright crazy or insane. Take Van Gogh, Beethoven, Mozart, Einstein, Newton.

No matter what people say, technology is an under-appreciated art and some of the most creative people in the world today are in the technology sector. Simply because technology is so pervasive, it is simply taken for granted. The genius in the technology sector is easier to find because the desire to conform is in general far less. In fact, technology encouraged the people who were more traditionally in the sidelines and to let themselves loose and discover their potential in a field where they could still hide in the background but still make an impact.

Kraya is a good example of this.

In 2003, we built the first megabus site; in around 6 weeks. It is now a force to be reckoned with in the transport sector with over 100,000 visitors every day (across the uk & us).

We have our name on the site and we tell people that we are responsible for the site. However, it is impossible to convey the true genius behind the site. It handles around 5 or 6 searches every second (at peak), with each search being returned within a second or two. This does not seem that impressive but then that is because it is difficult to convey the complexity of the availability & pricing algorithm and the amount of data it has to deal with just to return one set of search results.

As another example, lets take the Edinburgh Festival Fringe website. Since launch, on the day of opening bookings, it has been unavailable for day(s) due to the sheer amount of traffic.

The first year that we took it on, without making any changes to hardware but by pretty much completely re-writing the internals of the site, we took the site live with disruption to service limited to a mere 2 hours.

This is difficult to notice from the public’s perspective since the site looked exactly the same. The fact that the site was now substantially faster, more responsive and reliable was just not registered!

For the Fringe website this year (2008), we organised a complete infrastructure revamp and carried out more work on the internals to ensure that the site would work and be completely available with NO disruption to service at all.

Unfortunately, due to issues with the ticketing system, this never went live as planned.

Kraya has largely stayed in the background through all of these. Like an artist, we signed our work (our name is there both on the megabus site and the Fringe site). However, like with all pieces of art, the beauty is in the eye of the beholder.

It is unfortunate, that organisations like Apache, GNU and even Microsoft, people like Linus Torvalds, Richard Stallman and even Steve Jobs or Bill Gates, and products like Linux that make a real difference in the world stay largely in the background while Tracey Emin’s Bed is so much better known.

It seems the best way to measure the success of a technology product is to see how much it is taken for granted. Google is the perfect example of this.

Are we now living in a world where “to use something” is the best way of saying “thank you.” In other words, is the best way to thank google to just use google?

To come back to my original point of censorship. I agree that as individuals in a community, and as an organisation in the marketplace, there are some things that would be inappropriate to say in a public blog. However, this is often hard to define. More importantly, I believe that each and every single one of us at Kraya are capable and sensible enough to decide what is (in)sensible to be in a blog. The person most likely to pop something into a blog that shouldn’t be there would be me.

I blame this on the fact that I have never differentiated my life in general from my life at Kraya!

WordPress Themes