The hack got hacked recently, the following text can be read on their frontpage:

Security breach on

Earlier this month, a number of servers in the infrastructure were compromised.  We discovered this August 28th. While we currently believe that the source code repositories were unaffected, we are in the process of verifying this and taking steps to enhance security across the infrastructure.

What happened?

  • Intruders gained root access on the server Hera. We believe they may have gained this access via a compromised user credential; how they managed to exploit that to root access is currently unknown and is being investigated.
  • Files belonging to ssh (openssh, openssh-server and openssh-clients) were modified and running live.
  • A trojan startup file was added to the system start up scripts
  • User interactions were logged, as well as some exploit code.  We have retained this for now.
  • Trojan initially discovered due to the Xnest /dev/mem error message w/o Xnest installed; have been seen on other systems.  It is unclear if systems that exhibit this message are susceptible, compromised or not. If developers see this, and you don’t have Xnest installed, please investigate.
  • It *appears* that 3.1-rc2 might have blocked the exploit injector, we don’t know if this is intentional or a side affect of another bugfix or change.

What Has Been Done so far:

  • We have currently taken boxes off line to do a backup and are in the process of doing  complete reinstalls.
  • We have notified authorities in the United States and in Europe to assist with the investigation
  • We will be doing a full reinstall on all boxes on
  • We are in the process of doing an analysis on the code within git, and the tarballs to confirm that nothing has been modified

The Linux community and take the security of the domain extremely seriously, and are pursuing all avenues to investigate this attack and prevent future ones.

However, it’s also useful to note that the potential damage of cracking is far less than typical software repositories. That’s because kernel development takes place using the git distributed revision control system, designed by Linus Torvalds.  For each of the nearly 40,000 files in the Linux kernel, a cryptographically secure SHA-1 hash is calculated to uniquely define the exact contents of that file.  Git is designed so that the name of each version of the kernel depends upon the complete development history leading up to that version. Once it is published, it is not possible to change the old versions without it being noticed.

Those files and the corresponding hashes exist not just on the machine and its mirrors, but on the hard drives of each several thousand kernel developers, distribution maintainers, and other users of  Any tampering with any file in the repository would immediately be noticed by each developer as they updated their personal repository, which most do daily.

We are currently working with the 448 users of to change their credentials and change their SSH keys.

We are also currently auditing all security policies to make more secure, but are confident that our systems, specifically git, have excellent design to prevent real damage from these types of attacks.

Now, you know how I think about linux security experts and all that, and this would be an excellent opportunity to point out that linux is not all that ‘secure by design’ as some people who like to point to statistics think it is.

But I’m not the type to dwell on that. I think the most important lesson that can be learnt here is that the biggest security problem is in how the tarballs are distributed. Yes, they point out that they use Git, which means that many developers will have uncompromised copies of the code, so it is hard to tamper with. However, this is not being exploited fully. By distributing pre-packaged tarballs on unchecked servers, there is still room to tamper with the code. Perhaps not the code that developers see in Git, but that is not the biggest problem. If most users (which probably includes most major distributions) depend on the tarballs rather than Git to get a kernel release, they are wide open to such attacks.

So the simple solution is: don’t do it! Instead, use a script that will generate the source tarball on-the-fly by fetching it from the Git repository directly. If you want to distribute pre-built binaries, perhaps it is even feasible to do this via a script as well. It may not be feasible to build the binaries on-the-fly (at least, not yet, but we may one day get to the point where the sources can be rebuilt almost instantly… perhaps we could also modify the build process to make it faster, such as caching the internal bytecode, more or less like how Java/.NET work), but they could be fetched and rebuilt automatically from Git periodically, and cached in-between. A few simple extra checks could be applied during each fetch, for extra insurance (Is the filesize and date exactly the same as when it was last built? Does it verify against a set of hash checks? If not, delete it and rebuild).

This entry was posted in Software development, Software news and tagged , , , . Bookmark the permalink.

2 Responses to The hack

  1. Pingback: First, now hacked | Scali's blog

  2. Pingback: The myth of linux/open source security | Scali's OpenBlog™

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s