Increased Write Performance With New File System

June 8th, 2009 9:36 am
Posted by Kurtis Holsapple

While I have been waiting for the up-coming Linux Kernel for some time (2.6.30), I thought that I would right a quick summary of all the excitement this new kernel holds.

With the new kernel right around the corner, most of the hype has been directed at the support for new filesystems. Changes to the EXT4 file system has gotten a lot of press, along with changes to EXT3 and BTRFS, but what I am most excited about is the inclusion of the relatively new file system NILFS (or NILFS2). NILFS stands for New Implementation of a Log-Structured File System (Version 2) and has quite the unique approach to file structure. Without getting into too many technical details, NILFS is a Log Structured file system (great article from wikipedia explaining this). Unlike tree style file systems, data is written to the head of a log, which is the file system. This makes seek times much shorter as everything is sequentially ordered.

Another great feature for this file system is it's check-point and snap shot features. Because all the data is written to a log, these logs save a check-point at regular intervals (every couple of minutes).  When you want to save one of these check-points, a simple command can turn a check-point into a snap shot. As these can take up some file space, check-points are regularly removed when they become obsolete. By converting a check-point into a snap shot, you are basically saving an exact image of your file system. Even better, each of these check-points and snap shots can be mounted for very easy file recovery, or system recovery.

The best part about this new file system can be seen directly in the HPC community. With shorter seek times, hardware performance is drastic. The whole idea behind this file system was to reduce the amount of write time needed (with the assumption that most read functions would be cached). This is most noticed on solid state drives.

As nothing is ever quite perfect in the technology world, there are some downsides to NILFS. If used as the root file system on a Linux machine, there is a lot of R/W activity. On a SSD, this would kill the drive faster because of the NAND chip top number of rewrites. For now, I would recommend setting up the root of your machine with a tried and true file system, and mount a NILFS drive inside your working folder. With the great amount of activity in the HPC community, I would love to hear some feedback on anyone using NILFS in a testing or production enviroment. Please comment on this story if you have any benchmarks or questions!

For more on this subject, check out the following links:
Linux-Mag: NILFS: A File System to Make SSDs Scream
Wikipedia: NILFS
Dongjun Shin Benchmarking results (Feb 2008) (PDF)


You must be a Registered Member in order to comment on Cluster Connection posts.

Members enjoy the ability to take an active role in the conversations that are shaping the HPC community. Members can participate in forum discussions and post comments to a wide range of HPC-related topics. Share your challenges, insights and ideas right now.

Login     Register Now

Author Info

Kurtis works in the tech industry as a programmer, mainly focused in upcoming web technologies. He has experience working with Linux on man levels, from desktop, to file/print server, to LAMP stack and custom device programming. His favorite languages are Python, PHP and LISP (it just works so weird!) using libraries to create many different types of applications including games (pygame/pyglet, actionscript), productivity software (pyQT4) and more.
Since 2004, Kurtis has worked in various fields in the tech industry. He worked in the real estate sector providing IT support/software development, traffic data analysis and collection. Currently, Kurtis works mainly with web based projects. You can read his personal blog at