Category: Innovation

Fluidic Navigation

I have a dream – to think as little as possible. I agree with Arthur C. Clarke when he suggests that “Any sufficiently advanced technology is indistinguishable from magic.”

Therefore, how can we get technology to “magically” make things easier for us to do. The invisible interface, which I have covered before, is one outcome of such a concept.

In this particular case, however, I am referring to web navigation. In some ways, arguably the biggest user-interaction challenge over the medium of the web.

Read more »

Making Twitter Faster

From my perspective, Twitter has a really really interesting technical problem to solve. How to store and retrieve a large amount of data really really quickly.

I am making some assumptions based on how I see twitter working. I have little information about how it is architected apart from some posts that suggests that it is running ruby on rails with MySQL?

Twitter is in the rare category where there is a very large number of data being added. There should be no updates (except to user information but there should be relatively very small amount of that). There is no need for transactionality. If I guess right, it should be a large amount of inserts and selects.

While a relational database is probably the only viable choice for the time being, I think that twitter can scale and perform better if all the extra bits of a relational database system was removed.

I love challenges like this. Technical ones are easier 😉

If I didn’t have a lifetime job, I would prototype this in a bit more depth. Garry pointed me in the direction of Hadoop. Having had a quick look at it, it can take care of the infrastructure, clustering and massive horizontal scaling requirements.

Now for the data layer on top. How to store and retrieve the data. HBase is probably a good option but doing it manually should be fairly straightforward too.

From my limited understanding of twitter, there are two key pieces of functionality, the timelines and search.

The timelines can be solved by storing each tweet as a file within a directory structure. My tweets would go into

/w/o/r/d/s/o/n/s/a/n/d/

The filename would be -

For the public timeline, you just have a similar folder structure, but with the timestamp, for example, the timestamp 1236158897 would go into the following structure as a symlink

/1/2/3/6/1/5/8/8/9/7/

For search, pick up each word in the tweet and pop the tweet as a symlink into that folder. You could have a folder per word or follow the structure above.

/t/w/i/t/t/e/r/- OR

twitter/-

You would then have an application running on top with a distributed cache with an API to ease access into the data easier than direct file access. Running on Linux, the kernel will take care of the large part of the automatic caching and buffering as long as there is enough RAM on the box.

This can in theory be done without Hadoop in between and separating the directory structures across multiple servers but that can have complications of its own, especially with adding and removing boxes for scalability.

You are also likely to run into issues with the number of files / sub-directories limits but they can be solved by ‘archiving’ – multiple options for that too…

Thinking about this problem brought me back to the good old days of working on the search mechanism within megabus.com. We needed the site to deal with a large number of searches on limited hardware when the project was still classified as a pilot.

With some hard work and experimentation, we were able to reduce the search time to a tenth of the original time.

I’ll admit that I don’t know the details or the intricacies of the requirements that twitter has. I have probably over-simplified the problem but it was still fun to think about. If you can think of problems with this – let me know; I wanna turn them into opportunities 😉

Breaking Software Down

Jeff Atwood likens software development to tending a garden. I can relate to this. In fact, I would like to ask, if you have a nice plant in one of your gardens, how complicated is it to “copy” that across to another one?

I realise that I am moving away from the analogy here but there is an important concept here. Libraries were born out of the desire to share and distribute code to be re-used.

The idea for Remote Procedure Calls dates as far back as 1976. Microsoft brought along OLE and then COM made this more generic and better.

RPC is widely in use these days and there are several other mechanisms for inter process communication including CORBA, REST & SOAP.

I don’t think software is broken down into small enough components. *nix is great in that you can tag a whole bunch of commands together on the command line to do some amazing things. I have personally piped data through a dozen or so commands and scripts to do some interesting things.

If we could break everything down into individual components that could be linked together, we would have a massive arsenal of interoporable tools that each user can pick and choose to put together very powerful solutions.

How many times have you found a piece of software that does one thing really well, but fails in something else. Then found another piece of software that does the other thing really well.

For example, the extensibility of Firefox is fantastic but I love the rendering of Safari. I love the Contact Management within Evolution and the Mail capabilities of Thunderbird.

Why don’t we break each software down into each of it’s individual components (and I am not talking about libraries here) and allow them to be deployed as services usable by other pieces of software.

In other words, release the contact management capabilities of Evolution as a product of it’s own right with a pre-defined API that any application can link into (including perhaps a web interface). Release the Mail management component of Thunderbird as a service, Release GUI’s as a component. Then we can pick any GUI we want, link into a specific mail component and another addressbook component.

Do one thing and do it well. In fact, let’s take it one step further and release a public API for each software component – an API for Mail, one for Contact Management and so on.

Each software component can then be a black box that delivers this API.

Choice can be a bad thing if it makes it difficult to choose – Subclipse vs Subversive is a good example of this. Let us however, not confuse choice with flexibility.

Let’s say that you want to find all the files within a folder modified within the last 3 days containing the text “abracadabra” and then replace all occurences in those files of the world “super”  with “hyper”.

To do this in linux, all you would do is chain find (to identify files modified in the last 3 days), grep (to identify only the files that contain “abracadabra”) and sed (to do the replacement).

If you know these commands well enough, you could chain something together in half a minute or so. You could probably figure out how to do this with the search tools in Windows within a minute or so but where this really shines is if there are thousands of files that needs to be processed. With other search tools, you would have to wait for the original search results to be returned before running to replace operation. This takes up the users time.

With the chaining of commands, I have run it and worked on something else while it completes.

Let me visualise a brave new world:

In this world, all software would be interoperable components. For example, there would be components for:

  • Mail account management (Perhaps genericised into configuration management)
  • Text composition (usage for mail, documents, plain text et al)
  • Text reading (again, usable for mail, documents, plain text et al)
  • Spam Filtering (already available to some extent)
  • Contact Management (optionally linked into organisation’s LDAP server)
  • Task Management (Standalone Mylyn if you know the product)
  • Scheduling (or calendering if you prefer that term)

If all of these components were interoperable, then there would a GUI that is generic and could bring all of these together. In this way, the people working on each of the components could concentrate on doing one thing and one thing well.

If we then start working on public API’s in a collaborative fashion, each of the component could be fleshed out to be as flexible and complete as necessary to gain maximum benefit.

If these components provided the services as a network based API, it would also allow for the components to be distributed across a network providing redundancy and efficiency. This makes it easier to turn each desktop into more of dump terminal concentrating purely on user interaction and getting closer to the invisible interface.

Software as a service has taken a step in the right direction. Can we take a leap and have software component as a service…

Your Time

I have always been curious about how I spend my time. It would be useful or at the very least cool, if I could figure out what I did with the 24 hours I get each day. Considering that this is overkill, it would be useful to find out how much time I spend on each task / client at work…

This is something that a lot of companies would find useful, especially in the professional sector. Lawyers, for example need to know this information for the purposes of billing. Time tracking has always been a personal bug bear of mine and is something that I often ramble on about.

In fact, in Chris’ Blog, the team management application he speaks about towards the end is something that I have been talking about and working on in the background for about a year now (actual effort is probably only around a week since I keep getting distracted with other things)

My aim has always (unconsciously) been to have an invisible interface, more because if there is effort involved, its going to take away from the task at hand and unless you are an obsessive compulsive (which I sometimes end up being), its going to be inaccurate. More importantly, its going to take up time to track and log time and ends up being a task of its own which requires management.

The most common way to log time is to just have log sheet which is filled in at the end of the day – just putting numbers against tasks and making sure it adds up to the number of hours you worked in the day.

Mylyn for Eclipse, which is a task/bug management plugin has a cool and useful concept of task contexts. This means that when you activate a tasks, it remembers the files you are working on, as well as which lines you were on and saves it as the task context. You can then de-activate the context or activate another task. When you go back to this task, it loads up the context. In other words, the files you had open, at the lines that you were at. This is a real boon and can save a lot of time once you get the hang of creating a task for everything you are working on and practice a little bit of self discipline in terms of activating and de-activating tasks.

Mylyn could use a little more intelligence in context management and creation on activating tasks with no context. But the concept of contexts is great and it should be possible to apply this at the operating system level.

For example, I am working on Task A. This involves a number of URL’s in firefox, a number of emails from Thunderbird, some files in Eclipse, some terminal windows, couple of Documents, and a spreadsheet.

Task B involves a url in firefox or two and a document.

and so on.

It should now be possible for me to simply tell my “Task Manager” that I am about to work on Task A and have all the applications load up in the locations that I last left them and hide away or close the other applications. When I want to switch to Task B, it should close everything else and open up the applications for Task B.

My desktop environment (GNOME), like most Linux desktop environments have the concept of workspaces. You can also get this as an addon for Windows. I have used these to emulate this kind of behaviour but this is far from perfect.

In theory however, this should make it possible for an application to track the time spent in each workspace. If the application is told what task is associated with each workspace, it could automatically log time against that.

The real power for task management would come from a powerful use of task hierarchy. Lets say we have the following task list

Client A => Project A => Subtask 1

Client A => Project A => Subtask 2

Client A => Project B => Subtask

Client B => Project A => Subtask

In this setup, the task management could be told (or even better figure out on its own) that there are application hierarchies… In other words, for all tasks related to Client A – the user will always need application X (with a particular configuration – say firefox on a particular url)

For Project A, there could be the requirement of another url in firefox.

This way, when a new Project or subtask is created, the system can automatically start a few steps ahead…

Client B could have completely different application setup / layout.

Automatic time tracking, task managment and planning then becomes so much easier.

For this to be viable, however, there are a lot of things that still needs to happen.

  • Context awareness for applications and ability to save a context
  • Ability in applications to merge contexts (for hierarchical contexts)
  • Efficient Application swapin/swapout when tasks switch for instant task switching
  • Applications needs to be more lightweight to make task switching quicker

Most, if not all modern applications provide extension points and it should be possible, at least in theory to add this functionality on to most applications. More importantly, it is theoretically possible to add this functionality on to all the open source software since the source is available. However, since it requires a fairly radical shift in thinking about application state, this is probably far more complex than it seems.

Once we have something like this working, it should be much more straightforward to add workflow using the task hierarchy to make it seamless including passing the contexts to the another person if necessary…

In any case, a few years into the future, who knows, these features might be built into all applications and perhaps we will be talking about task management, not application / process management.

WordPress Themes