Software

 

RapidDisk

RapidDisk is a collection of open source Linux kernel modules and administration utilities.

 
 
rapiddisk_diagram

 

Feature Benefit
Simplified Management Dynamically attach, detach and resize multiple RapidDisk RAM disk nodes. Create and manage volume sizes from as small as a few MBytes to as large as many TBytes. Unleash performance that is reach 20-30 faster (and even more) than your local magnetic SATA disk drive.
Simplified Archival Archive and compress or decompress and restore RapidDisk user data.
Thin Provisioning Consume memory only when needed. RapidDisk volumes allocate memory pages as they are requested during write operations. For instance, if you attach a 64 GByte RAM drive but only store 4 GBytes of data, only 4 GBytes of DRAM is consumed.
Advanced High Speed Caching Accelerate disk I/O with Write-Through or Write Around Caching to DRAM and limit access to persistent and sometimes slower storage media.
High Availability Enable Highly Available cache mappings to ensure that a node failure does not slow you down.
Improve Drive Longevity Increasing the life of the physical storage device by writing or caching data into memory. That is, you are either not moving mechanical components seeking from sector to sector or not reducing your total cell PE count on an SSD.
Energy Saving Reduce overall power consumption when handling user data, specifically by offloading most if not all requests to stateless DRAM.
Secure Drive Data Enable data encryption on RapidDisk and/or RapidDisk-Cache volumes and ensure that the drive’s contents are secure.

 



Highlights

HP presented RapidDisk / RapidDisk-Cache as part of a demo in the High Performance Computing and Open Source & Linux Technical Excellence Symposium 2014 with performance benchmarks as high as 40 Gigabytes per second! The presentation was titled: Addressing Database Performance problems using the HP Enigma RAM Storage Solution and was presented in Grenoble, France.


 
 

RapidDisk

RapidDisk was designed with simplicity in mind. Utilizing a management binary application, dynamically add new RAM based block devices of varying sizes, resize those same block volumes, remove existing ones, and more. The best part of the kernel module is that it has been designed to allocate storage space from the system’s memory pages as it becomes needed and is capable of scaling upward to address Random Access Memory sizes in the Gigabytes if not a Terabytes. To date, RapidDisk has been adopted by many companies and institutions and used either in their internal data centers, testing infrastructures, distributed software / hardware products, to even as a learning tool.

 
 

RapidDisk-Cache

RapidDisk-Cache was designed to leverage the high speed performing technologies of the RapidDisk RAM drive and utilizing the (Linux native) Device Mapper framework, map an RapidDisk volume to act as a [slower] block device’s Write/Read-through or Write Around Least Recently Used or LRU (FIFO) cache. This can significantly boost the performance of a local or remote disk device.

 

What is Write-Through and Read-Through Caching?

This is where an application treats cache as the main data store and reads data from it and writes data to it. The cache is responsible for reading and writing this data to the permanent storage volume, thereby relieving the application of this responsibility.

In RapidDisk-Cache, all writes are cached to a RapidDisk volume but also written to disk immediately. All disk reads are cached. Cache is not persistent over device removal, reboots, or after you remove the Device Mapper mapping. This module does not store any cache metadata on RapidDisk volumes but instead in memory outside of RapidDisk. The best part of RapidDisk-Cache is that you can map and unmap a cache drive to any volume at any appropriate time and it will not affect the integrity of the data on the persistent storage drive.

 

What is Write-Around Caching?

Write Around caching shares some similarities with the Write-Through implementation. However, in this method, only read operations are cached and not write operations. This way, all read data considered hot can remain in cache a bit longer before being evicted.

 

Performance

The performance numbers obtained here with RapidDisk will always be consistent as opposed to the Flash memory. It has even been brought to my attention that certain companies using RapidDisk have obtained far greater numbers than the ones provided below. Some of whom achieved performance that reaching as fast as 40 Gigabytes per second!

Performance Diagram

Performance Diagram

Real world RapidDisk-Cache numbers vary as a result of the data being in the cache or not. The advantage to using RapidDisk-Cache is that it will significantly decrease random access latencies for cached data. When the accessed data is cached, the performance numbers will always be consistent. It has even been brought to my attention that certain companies using RapidDisk-Cache have obtained far greater numbers than the ones provided below.

These numbers were obtained using an older 64-bit system running a 3.1 GHz Xeon Quad Core processor. I used fio 2.0.3 and ran the test on volumes formatted and mounted with an Ext4 file system. Each job file ran 4 processes (one at a time): (1) Sequential Reads, (2) Random Reads, (3) Sequential Writes, and (4) Random Writes.

 

Profile RapidDisk SATA (mechanical HDD)
1M xfer 1G file 32 depth Seq Reads: 5.3 GB/s Seq Reads: 154 MB/s
1M xfer 1G file 32 depth Seq Writes: 3.2 GB/s Seq Writes: 131 MB/s
1M xfer 1G file 32 depth Ran Reads: 7.3 GB/s Ran Reads: 93 MB/s
1M xfer 1G file 32 depth Ran Writes: 3.0 GB/s Ran Writes: 106 MB/s
 
4K xfer 1G file 32 depth Seq Reads: 1.7 GB/s Seq Reads: 116 MB/s
4K xfer 1G file 32 depth Seq Writes: 1.5 GB/s Seq Writes: 70 MB/s
4K xfer 1G file 32 depth Ran Writes: 1.6 GB/s Ran Reads: 2.2 MB/s
4K xfer 1G file 32 depth Ran Writes: 1.4 GB/s Ran Writes: 1.9 MB/s

 

Profile RapidDisk-Cache SATA (mechanical HDD)
4K xfer 32M file 32 depth Seq Reads: 802 MB/s Seq Reads: 104 MB/s
4K xfer 32M file 32 depth Seq Writes: 68 MB/s Seq Writes: 70 MB/s
4K xfer 32M file 32 depth Ran Reads: 907 MB/s Ran Reads: 7.9 MB/s
4K xfer 32M file 32 depth Ran Writes: 2.8 MB/s Ran Writes: 2.8 MB/s