Improving ESXi iSCSI/RAID/Disk Performance Through RAM Cache

Improving Through RAM Cache
Improving Through RAM Cache

DISCLOSURE: This post may contain affiliate links, meaning when you click the links and make a purchase, we receive a commission.

Disk performance is one of the most important aspects of any home lab. But, in most cases, it gets overlooked which is unfortunate. If you want to get an impressive performance from your system, it is undeniable that you would need help from the storage drives.

It does not matter if you have the fastest CPU or the best motherboard, it won’t help if your disk performance faces a bottleneck. There are two solutions to this problem. First, you can use SSD drives and SSD arrays. But the problem with this solution is that you have to spend a lot of money on SSD compared to HDD.

And, the second solution is to take advantage of RAM caching. In our opinion, we think this is the best option for most of us home lab owners. In this post, we will cover the methods of using RAM cache technology in a detailed manner. Moreover, we will run several tests regarding the disk performance with/out the RAM cache enabled and will share those test results with you.

So, without wasting more time here, let’s get to the business…

Improving ESXi iSCSI/RAID/Disk Performance Through RAM Cache

What to use for RAM cache?

There are a lot of softwares to use RAM cache on your system. So, you can use any one of your favorite ones. But, in our case, we will be using a software called PrimoCache (was known as FancyCache). We have used this software on a number of builds of ours successfully. It really helps to improve the performance to a greater extent. In this post, we will be showing how to use this software. But prior to that, let’s learn about the build on which we are going to use it.

The setup for the testing purpose:

For this test, we will be using one of our pre-built SAN. This build is a mid-range one. So we believe it will perform well enough for this test. We will be using PrimoCache/FancyCache software for the RAM cache. We are using a 15K SAS drive RAID card on the SAN build. So the test will be for multiple RAID levels. However, for the primary test, we will be using RAID0, which is the fastest RAID settings of all.

Here is the list of the hardware of this SAN build.

  • Mobo (Motherboard): From ASRock 970 Extreme 3
  • Processor (CPU): AMD FX-8350 @ 4.0GHz speed, Octa Core Processor (Vishera)
  • PSU (Power Supply Unit): Corsair HX 750 (Professional Series)
  • RAM: (4 Sticks x 8 GB) Total= 32GB DDR3-1600 bus speed
  • RAID Drives: Seagate Cheetah 15.7K 6Gb/s SAS Drives (146GB)
  • RAID Card: From LSI, Model-MegaRAID SAS 9260-8i, with 8-Port @ 6Gb/s RAID Card
  • NIC Card: Quad Gigabit NIC card from Intel, Model- Pro/1000 PCI-e
  • Operating system Drive: 128 GB SSD card from Crucial, Model-M4 @6Gb/s

At this point in the article, we will be sharing the process of installing and configuring PrimoCache. This is fairly easy to do. So, let’s learn it.

Installing and configuring PrimoCache:

You can install PrimoCache normally as you install any other software on your PC. So, just double click on the installer icon and follow the further instructions to install the software. After the installation is completed you need to restart the system to finalize the installation. That is it. Now you can start to configure the software to set up the RAM cache.

So, double-click on the PrimoCache shortcut icon to launch the software. The primary screen will most likely look like the screenshot below.

Check Image: User Interface of PrimoCache

Now, you can start to create the cache. You can select any one of the ‘Volume’/’Local Disk’ as the target of the cache. It is not compulsory to select any physical drive. It can be a dynamic disk, RAID volume, virtual drive or a typical physical drive.

Now, remember, you can set the RAM caching to any amount, but it will use the amount of RAM you have free on your system. So, we would suggest that you should not use more than 80% of the memory for RAM caching.

The more RAM you have, the better the performance would be. Now, select the preferred drive of yours from the software as the target Drive. In our case, the target drive we would be using is the Drive: D; as this is our RAID volume.

Check Image: Adding a Cache

After selecting the drive, click on the ‘Next’ button to proceed to the cache configuration window. Now, you can set the size of your Cache memory on this screen. You have to enter the amount in MB. Also, you will need to set the way you want to use the cache (Cache Strategy).

For instance, you can set it to increase the performance of the reading or writing or even both of these two performances of your disks. You can also use a Level 2 cache with an SSD drive using PrimoCache. You also have the option to change the block size to improve performance. For better performance, you should use a smaller block size. But, you will need a powerful CPU for that.

Check Image: Configure the Cache Settings

In our case, we will be using a ‘Custom’ configuration. We will leave 16 GB of RAM for the cache (from a total of 32 GB RAM). The block size will be at 4 KB. Next, you need to check-mark the ‘Enable Defer-Write’ option and set the latency to 10 sec. However, you definitely have to set the parameters according to the configuration of your system. You can take a look at the screenshot down below for a clear idea.

Check Image: Customizing the Cache Settings

When you have configured all the settings, you have to click on the ‘Start’ button. And, the caching will be activated instantly. You will see a pop-up notification on the PrimoCache software saying that ‘Succeed in starting cache’ like the image below. You need to click on the ‘OK’ button to close the ‘success’ notification.

Check Image: RAM Cache started successfully

In doing so, you will return to the main screen of the PrimoCache software. But at this moment, you will see all the detailed information about the current cache settings. You will also find stats about the write and read performance. You can easily monitor the cache performance from this screen of PrimoCache. You will find all the additional ‘settings’ options on the top bar of the software. From there, you can resume, pause, stop, flush etc. the RAM cache.

Now that we have set up the cache, we can run the performance test.

Performance test of PrimoCache:

To conduct a perfect performance test, we will be testing it both ways — by turning the caching on and off. Then we can compare the performance perfectly. Before we begin, we want to share the current settings of our RAID card.

  • Size: 875 GB
  • RAID level: 0
  • Stripe Size: 64 KB
  • Read Policy: Always Read Ahead
  • Disk Cache Policy: Enabled
  • IO Policy: Cached IO
  • Current Access Policy: Read-Write
  • Default Write Policy: Always Write Back
  • Current Write Policy: Write Back

Check Images: RAID Settings

Running the Benchmark Tests:

Here are some of the summary of the tests we have run on our system.

  • ATTO Disk Benchmark: 256 MB of total length, 4 queue Depth with overlapped I/O, 0.5 to 8192 KB Transfer Size
  • IOMeter: On 100% read and 100% write the sizes are 8K, 64K, and 256K; 50%/50% read and write; 75%/25% read and write; 25%/75% read and write; for 30 minutes with 10 workers per test (random, mixed, sequential)
  • Crystal DiskMark3: Using Sequential 1024 K blocks – 50 Megabyte, 100 Megabyte, 500 Megabyte, 1000 Megabyte, 2000 Megabyte, and 4000 Megabyte Test Sizes; Using Radom tests – 512K, 4K, and 4K with a Queue Depth of 32 (Read/Write)
  • HD Tune Pro: Random access tests, Read and Write tests
  • Server Exchange Jetstress Tool from Microsoft: Disk throughput tests for 2 hours at n00% drive capacity (with & without cache)

Some Additional notes about the tests:

During the tests, we have noticed that some issues have arisen because of the large size cache. Normally, the RAID cards come with adjustable algorithms which helps them to adapt to the workload over time. So, it is sure that you will get better performance later. But, it is not possible to show that performance right now.

And, as we have tested using 60%/40%, 50%/50%, 75%/25% drive capacity, we have to admit that in reality, the drive capacity wouldn’t be always like this. Rather it will vary from time to time. So it is better to get a closer look at the general performance.

Another issue is that most of the time while the benchmark test is running, the cache rate will hit 100%. That implies that we are testing the performance of pure RAM. This is OK for ‘write’ benchmarking though. However, for ‘read’ caching it gets difficult to guesstimate the performance like this. Because it takes a lot of time to hit a decent read cache performance as the caches are using most used items already.

On top of all these, if you use the Level 2 caching with PrimoCache, you can use SSD drives for the caching. By doing that, you can hugely improve the read cache hit rate. Added to that, you can use PrimoCache with Starwind’s iSCSI software to improve the responsiveness of your system further. We use both of these softwares together on a number of our builds to take advantage of RAM caching.

Now, we will discuss in detail about all the tests we have run so far down below.

About the Results of HDTune Pro Benchmark:

You might have already know that HDTune bypasses the RAM cache. So, in this section, you will find the performance result of the system without the RAM caching feature. On top of this, we will also share some data about a few latency tests, and IOPS tests too.

Here are the screenshots of the Benchmark Results:

About the Crystal DiskMark3 Benchmark Results:

These particular tests were performed to find out about the read and write speed of the storage drives. From this, we will find out about the performance of the RAID Array speed wise. Once again, we will be using the tests both ways — with/without the RAM caching. By default, the Crystal DiskMark3 conducts each and every test 5 times and then show the average result. This is very good if you are looking for an accurate result. The results will be in MB/sec.

Check Image: Crystal DiskMark3 Benchmark Results

From the image above, you can see that the results are astonishing. On some occasions, we have seen that the read and write performance increased up to 31,000%. Incredible, isn’t it? This type of result indicates that the RAM caching is actually worthy. This type of performance highly needed for high-write software. Also, we need to mention that, these results are from deferred writes. So it means over time, the performance would improve further.

About IOMeter RAID Benchmark Results:

For benchmarking the IOPS performance, there’s no better alternative than IOMeter. This utility comes with a lot of features. And, it is also highly customizable, so that it can push a system to its limit for a high-quality performance test.

We have run a number of tests using the IOMeter. First of all, we have run multiple sequential tests with deferred read/write loads. We have also run tests on every block size. This is not compulsory as the most common block sizes are 8 K, 4 K, and 64 K. The results will be in IOPS.

Check Image: IOMeter Benchmark Results

If you compare the results, you can see that we are getting improved performance with the RAM caching enabled. As we are using the deferred write feature, the best performance was in the random writes. Overall, we would have to admit that RAM caching has brought a major improvement in the performance.

About the ATTO Benchmark Results:

This is another popular benchmarking tool. It helps to get an idea about the transfer speeds of your system in various test sizes. However, in our case, we have noticed some anomaly with this tool (ATTO) in the test size of 4096 KB. We still don’t know about the reasons behind this inconsistency.

The problem is that the results range from good to bad randomly. Thus, we have run multiple tests, however, with no luck, we have got different results each time. So, we are not happy about this benchmark utility. Nevertheless, we will be sharing the results with you down below.

Wrapping it up:

From the above discussion, it is evident that RAM caching is indeed a good way to improve the performance of the drives of your system. In this instance, we are using just 16 GB of RAM for caching. But we have seen that the more the better.

So, you can actually use this technology without any problems on enterprise-class servers if you want to. However, even for personal use in home lab servers, it can dramatically increase the performance easily. So, we would highly recommend it to our readers.

Reference: Donald Fountain, TheHomeServerBlog