The 5% rule of Ram Based Storage

By Randy James, President, ITPE

So your mouth is watering after you read about the jaw dropping performance of a ram based storage solution…but…then you looked at the cost of buying a terabyte or two of space…and now you have sticker shock!   Many ram based storage solutions have list prices in the $500 per GB range…what is a stressed out technology manager to do?

First, step away from the screen and take a few deep breaths!   It’s going to be ok!

Not every byte of your data needs to be hosted on such a screaming platform – in fact, the general rule of thumb is that about 5% of your data needs to be on ultra fast storage to maximize performance.

In response to this fact, storage manufacturers have created multiple “Tiers” to match the demand for I/O performance against different price points.  While the definition (or number) of storage tiers is not standardized, the concept is easy to understand:

• Tier 0 is the fastest, highest cost per GB, and usually refers to semiconductor based storage (Ram or Flash)

• Tier 1 refers to high performance, high reliability, high cost (“enterprise class”) hard disk storage (typically a Storage Area Network or SAN) using 15K Fibre Channel or SAS drives

• Tier 2 refers to lower cost, lower performance, lower reliability hard disk storage (typically SATA based using 7200 or 10K drives) – often used less frequently assessed files (last years financial reports, old order history, etc.), and for disk to disk backups and archiving of regulated data (emails, health information, etc.)

• Tier 3 is usually off-line storage – using high capacity magnetic tape, DVD, Internet cloud based storage, etc. where a “restore” operation to a higher Tier is needed before it can be utilized.

So how do you select the data that can benefit from Tier 0 storage?

Let’s take a look at a typical SQL database.   If it is small enough, or if you have enough money, you can place the entire database on a ram based platform – and stand back and watch it scream!   But in many cases, you can’t afford or need 100% Tier 0 storage, so the goal then becomes to buy a smaller amount of Tier 0 storage and use it for maximum benefit.

Let’s assume that we have a 500GB database for our following discussion.  When you take a close look, you will find that some tables (files) are accessed much more frequently than others, including:

• TempDB (scratch space) used for joins, sorts, etc.

• Indexes

• Transaction logs/journals used to roll back transactions and provide backup

• Frequently accessed/updated tables

TempDB can benefit from Tier 0 since it is essentially “virtual memory” for your database, and it is very write intensive.

Fast access to indexes is also another common use of Tier 0 storage – it provides great performance that can speed up the fetching of data from your SAN.  As most databases are also write intensive (adding new records also creates new indexes and causes current indexes to be modified), you’ll be able to update indexes hundreds of times faster than hard disk storage.

Transaction logs (sometimes called journals) also benefit because they are nearly 100% writes – the faster the log record is written, the sooner the application is notified of the successful I/O completion, and sooner that the next step of processing can continue.  You also gain the safety of having your logs on a physically separate and fast media - very useful when a database recovery is needed.   This is a key point – many IT specialists don’t consider the restore/recovery speeds of their backups.

You will likely identify some larger tables, such as a customer master table, that are accessed with every transaction – the order entry staff must search the table to check if this is a new or existing customer.  They may search by a variety of index fields – customer name, address, customer ID, SIC code, sales region, key contacts, etc.   Other tables, like the order history table, may benefit if the table has been partitioned, so that only the most recent orders are stored on Tier 0 – everything else is on Tier 1 or 2 media.

Balanced I/O is Important

To maximize the performance of your overall application, it is important to have the right “mix” of high speed and lower speed storage media, with the right amount of I/O capacity and bandwidth.  For example if your indexes are on Tier 0 media (capable of 100,000+ I/O’s a second), and your database is on Tier 1 storage spread across 4 disk drives (capable of a combined 1000 I/O’s a second) shared with multiple other applications, don’t call your ram based storage provider and complain about the poor performance!

Don’t forget the Infrastructure

It is critical that all elements of your infrastructure have enough capacity and are well tuned to maximize the performance of your applications.  I can’t stress this enough!  Nearly every client I visit, I find that the infrastructure (networks, servers, storage) has been given little attention, and is incorrectly and under configured.   IT administrators often assume that if a device is “on-line” that it is working well – when there are major issues that need to be addressed.

Infrastructure with a poor design, or not enough capacity, or improper configuration will cause performance and stability issues with the application that your DBA or application developers will NEVER fix.   If you don’t have the skills (or the time) to optimize your infrastructure (and few IT teams do), hire someone who specializes in this area.  It will save you a LOT of time and money and make you a hero!

As a final note, don’t overlook the design of your application.   I have stopped counting the number of poorly designed databases or applications that I have observed.  A poor design will consume ALL available I/O resources, no matter how much capacity you have.

Do your homework (and don’t be afraid to get some expert help) and you will see some amazing results when using Tier 0 storage!

Introduction to Solid State Disks

By Randy James, President, ITPE

Our site features education, news, forums, tips, case studies, white papers, and more to help you successfully apply this emerging technology to your organization!

Ram based storage is more than just “fast disk” – it represents a new era in data storage that is rapidly evolving.   New terminology is emerging – for example, the use of the term “microsecond”…here is the Wikipedia definition…

http://en.wikipedia.org/wiki/Microsecond

Storage specialists are familiar with hard disk I/O access speeds described in “milliseconds”, which equates to 1/1000 of a second – using the “ms” abbreviation.  A typical hard disk (146GB 15K drive for example), is able to perform a random I/O in about 3 milliseconds, or 333 I/O’s per second (1000ms/3ms = 333 I/O per second).  A 146GB 10K hard disk drive is able to perform a random I/O in about 4ms, or (1000ms/4ms = 250 I/O per second)

By contrast, a typical ram based storage device can complete a random I/O in about 15 “microseconds”, which equates to 1/1,000,000 of a second – using the “us” abbreviation.  This translates into 66,667 I/O’s per second (1,000,000us/15us = 66,667 I/O’s a second)!  

WOW!   So a single ram based device can be 266X faster than a single 10K (4ms access) hard drive!   State of the art ram based devices can support multiple links, and can offer a combined 600,000 I/O’s a second or more!   As my teenage daughter would say, “that’s crazy fast”!

Let’s use a “real world” example to compare a ram based storage device to hard disk arrays on a SAN.  

I have a customer that is evaluating ram based disks – looking to introduce a newly developed feature in their application that requires aggregating a large amount of data for each customer request.

They are a Windows shop, with SQL databases, and provide real-time stock pricing and trading services to thousands of on-line clients.   Like many large businesses, they depend on a SAN (they use an EMC Clariion Storage Area Network) that is equipped/configured to deliver 10,000 I/O’s a second to the server farm via switched fibre channel connections.   Note that the specific configuration of a SAN DIRECTLY effects the performance and throughput – we will discuss this topic in a future article.

The SAN has dozens of disk drives in RAID arrays.   They have several options available to them, including:

1. Do nothing, and use the existing I/O capacity of the SAN

1. Add more disks (spindles) to the SAN to increase I/O capacity

1. Use server based ram disk (uses server memory to emulate a disk)

1. Add ram based storage (i.e. solid state disk)

They estimate that they will need an additional 10,000 I/O’s per second to meet their performance needs for the new features – so they will need a total capacity of 20,000 I/O’s per second. We will assume they would utilize 15K hard disk drives for the best performance.

Using Option 1, adding additional load will clearly result in much slower response times for their existing users.

Option 2 will require the addition of at least 30 new drives (10,000 needed I/O’s / 333 I/O’s per disk per second = 30 drives) and associated hardware (controllers, shelves, power, rack space, cooling,  usage licenses and support costs.  This of course assumes no other use of the drives or overhead such as raid or spares, etc.  It also assumes that the distribution of I/O requests will be equal across all drives – ie. no hot spots.  It also assumes that the demand for I/O’s is balanced, and does not exceed the specified 10,000 I/O’s a second, or application response times will be negatively effected.

Option 3 will require a 64 bit server capable of having significant additional RAM added.  The use of a server based ram disk will add CPU load to the server, and represents volatile storage – if the server crashes, all data will be lost.  Servers capable of handling large quantities of RAM are relatively expensive.   To have high availability, it may require the purchase of 2 systems.

Option 4 is the use of a ram based storage solution (for example, a Texas memory Systems RamSan), connected into their Fibre Channel infrastructure.   A current RamSan can easily provide 150,000 I/O’s a second, and can be shared by multiple systems and applications, and is nearly bullet proof when protection of data is required.   The RamSan would easily be able to handle any spikes in demand or increases in use without any expansion.

The customer is considering options 2 and 4, and is conducting an on-site evaluation of the TMS RamSan solution at this time.  We will follow-up with the results of their evaluation in the next several weeks.

HBA Selection for Solid State Storage

By Randy James, President ITPE

Why is selecting the right Fiber Channel Host Bus Adapter (HBA) for your needs so critical when utilizing solid state storage?  Let’s count the ways:

1. Need for maximize performance – if you are purchasing solid state storage, you have applications that require the highest levels of performance that is just not available from a hard disk based storage area network (SAN).

1. Need for maximum ROI - the economics of a SSD purchase require that you obtain maximum value for your investment.

1. Compatibility with all components - the SSD, FC switches, operating and utility software, servers, cables, etc.

The PCI Bus – One size Does Not Fit All

One of the first challenges you will face is choosing the speed of the Fiber Channel “network”.  You may assume that if you have a 2Gb/sec FC SAN today that you would connect the SSD at 2Gb/s into that network.  While that will technically work, you would be VASTLY limiting the performance of the SSD.

On the other end of the curve, you want/need maximum performance, so choosing the highest speed FC technology (currently 8Gb/sec) would seem like a no brainer…but…you may need to make other major investments in your FC switch infrastructure to enable the use of 8Gb/s HBA’s. 

You may learn that many SSD providers don’t even support 8Gb/s, since TRUE 8Gb/s throughput (not data rate) is unattainable even with today’s PCI-Express 2.0 at x8 bus width.  However, since SSD’s are easily capable of saturating a single 4Gb/s FC link, the use of 2 or more “bonded” FC links via a technique called Multi-path I/O (MPIO) allows high performance servers to fully utilize the incredible bandwidth of solid state storage.

PCI technology was first introduced in 1992, but is constantly evolving.  I recommend reading the following 2 publications to become better acquainted with PCI technology:

An excellent (but old - 2003) PCI primer is available at http://www.digi.com/pdf/prd_msc_pcitech.pdf.

An overview of PCI-Express V3.0 capabilities, which is the next generation of PCI expected in late 2010 – see http://www.pcisig.com/news_room/faqs/PCI_Express_Gen3_faq_FINAL.pdf for details.

There are multiple “flavors” of PCI, with different capabilities and speeds.  For example:

** PCI-X Limitations: Highest speed grades are supported exclusively with point-to-point loads.  Use of 2 or more cards on a single bus significantly limits the throughput.

FC Ports versus PCI Slots

A hard disk based SAN that performs 10,000 I/O’s a second of 8K data blocks (typical page size of a SQL database) will deliver approximately (10,000 * 8192 bytes=) 81,920,000 bytes a second to the server.  In simple terms, 2 single port 4Gb FC HBAs, each in a dedicated PCI-E slot will easily handle the load presented by the SAN, and provide some load sharing and failover capabilities.

A Texas Memory Systems RamSan 440 SSD can deliver 4,500,000,000 bytes per second using 8 4Gb FC ports – essentially saturating 8 PCI-Express 2.0 slots…this is very different than the SAN load!

Many fibre channel suppliers offer multiple PORTS on a single HBA Card, so by definition, the ports (1 or 2 or sometimes 4 ports per card) share the bandwidth of a SINGLE PCI-Express slot.  So the use of 4 dual port cards, or 2 quad port cards (each solution provides 8 physical FC connections) will yield vastly less throughput rates than one port per card per slot, negatively effecting the ability to support more users, and reducing the value of the SSD asset.

HBA Selection Rules:

Rule 1: If you have to buy new FC switches (because you need more ports or because you are running 2Gb or less FC today), go ahead and buy the 8Gb switches, knowing that you will be limited to 4Gb for the next 12-18 months.  The 8Gb switches will easily allow the full use of each FC port at 4Gb/s, thus maximizing throughput in the short and long term.

When the PCI-E V3.0 products start to become available in 2010-11, note that that you will also need new servers with faster chip sets for PCI-Express V3, enabling the full use of 8Gb or higher data rates. 

Rule 2:  If you need to buy HBA’s for servers, it will be a benefit to buy the 8Gb products IF THEY ARE SUPPORTED BY THE SERVER PROVIDER, even though you will run them at 4Gb/sec.   Advances in the HBA chip sets will still benefit 4Gbs users through lower latency (internal to the HBA) per I/O.

Rule 3: Use a state of the art server that supports the highest PCI-E speeds and bus widths, and has a high number of dedicated PCI-E slots.

Rule 4: ALWAYS use a single port per PCI-E HBA card.  Do not attempt to use multiple ports on your HBA cards, as the SSD bandwidth will be limited by the PCI bus.  However, keeping a spare port available in case another port or card fails is a nice high availability strategy.

Rule 5: Do not use an old server with limited bus slots and/or bus bandwidth!  “Don’t even think about it!” - not even for development and testing.  When using SSD, it is CRITICAL that testing be performed in an environment where a full load test can occur.   Consider using SSD in the development and testing cycles to speed such testing.

Rule 6:  Check with your server and switch providers to learn what HBA cards are supported – and ONLY choose from that list!  Straying from supported configurations is just asking for trouble, given the huge load that an SSD will place on the devices in the infrastructure, and random, unsolvable issues will be the result.

Using a Dell 1950 Gen III server as an example, Dell does not currently offer 8Gb HBA cards for that server.  The 1950 is a 1U server with 2x PCI-Express 2.0 x8, and 1x PCI-X 266Mhz slots.   Dell’s web site states that it supports one and 2 port 4Gb/s HBA’s from Qlogic and Emulex in a Dell 1950 III, using a riser based PCI-E slot (small form factor).

HBA Selection Criteria:

• The HBA’s are certified and supported by your server vendor.

• Physical compatibility – there are full height and half height cards; servers vs blades.. Check on what your specific servers need.

• Supported methods of connection – Point to Point, Switched, or Arbitrated Loop

• Adapter data rates (8/4/2/1) in Gb/s.  4 and 8Gb are your choices; most modern cards are also backward compatible to 2Gb/s.

• Actual throughput (I/O’s a second, MB/sec) at the typical block sizes that you will use.  The high I/O rates that an SSD can sustain ware for the smallest block sizes (512 byte blocks), and I/O rates decline as a function of higher block sizes.  Oracle and SQL default to a 8K size pages, but check with your DBA to see what your environment is using.  Remember that 8Gb/sec throughput rates per port or card are not currently attainable.

• Only use one active port per FC card per PCI-Express slot.

• The media connecting the SSD with a switch or HBA will be fiber (older FC products supported copper for a short time).

• Ensure the SSD and HBA vendors supports Multi-path I/O (MPIO), to enable bonding of multiple FC ports together as a single, logical path to access the same LUN on the SSD.  MPIO can provide higher throughput, and also offers failover protection if properly configured.

• HBA software functionality – investigate the vendor provided tools to load and upgrade firmware, tools to configure and monitor HBA performance, etc.  Can the HBA firmware be loaded without downtime?  Does the vendor have a solid record of reliable firmware upgrades?

• HBA vendor stability – stay with the mainstream providers for best compatibility and support.

• HBA Support – what are the methods and hours of support?  Most will offer phone, email, chat, forums, user groups, etc.   Research available training or engineering support, warranties, spare policies, RMA policies, local service depots, etc.

• HBA Documentation – is it easy to understand, does it contain relevant examples, common questions, common issues and troubleshooting tips, etc.

HBA’s are a critical link in the performance of Solid State Storage – invest time to educate yourself on the topics and decisions needed to design and implement a high speed infrastructure for your applications.