Storage networking is a topic that could easily descent into deep religious debate, but I often get questions from customers and partners such as what does it matter in a virtualized environment, if we are virtualizing, why should we care what the storage network looks like. The specific question more recently was around 1GbE iSCSI versus SAS, so I want to specifically address the SMB market space, but the decision points are not dissimilar.

To start with a quick look at the SAS protocol. SNIA has a great presentation on the differences between the various storage networking protocols, http://www.snia.org/sites/default/education/tutorials/2011/spring/storman/GibbonsTerry_Shareable_Storage_With_Switched_SASv2.pdf. SAS, as it points out, is not a standard, but rather a way of conveying unique sas attributes. This is yet another way, primarily in highly dense server scenarios, to present shared storage. Essentially it is a way of sharing out direct attached storage. The main draw here is the speed over the 1GbE iSCSI. Since SAS is generally at 6Gbps, and can run over 4 channels for 24Gbps.

The main challenges for SAS focus on deployment and cost. It is often looked at as a cost saving measure over Fibre Channel, high speed and a lower cost. The challange here is that it is fairly limited in it’s scalability. It also introduces some complexity not found in iSCSI. Bringing in new switches, and zoning them is reminiscent of fibre channel, which is far more scalable.

iSCSI is not without it’s challenges of course. There is the consideration of using separate physically isolated switches from the remainder of the network, or using VLAN tagging on existing switches. 1GbE iSCSI can be saturated given enough utilization, and proper design is critical to minimize latency.

So to answer the question, what does it matter, the first response is is it supported. VMware publishes an exceptional Hardware Compatibility List, http://www.vmware.com/resources/compatibility/search.php, which should always be the first stop on these decisions. Secondarily to that, know your environment. While Switches SAS does have it’s place, at this point, in the SMB environment, it often makes sense to stick with what is a known quantity. Every environment already has an IP network, so leveraging that, or extending it is the simplest way of moving forward. This keeps the environment standards based, and does not require sticking with a specific solution. At the end of the day, beyond what is supported, the best design principle is to keep everything simple. While it may not matter as long as it is supported, generally speaking, then best designs are the ones which are well documented, easily repeated, and simple.

As always, there are exceptions to every rule, but I would say that using iSCSI is preferable over SAS for all those reasons, why make things more difficult than they need to be.

Object storage is starting to come back around recently with services like Box.net, and Dropbox to name a few. It is really a simple concept, which can be important in a virtualized environment, especially when you look at the industry trends.

Object storage is really quite simple. You store a file, not as blocks on a file system, but rather as an “Object” on a raw device. This is not so much different than when a database is given a raw mount point and manages the storage internally. The concept is that I place an “object” in the store. I define a policy that I want it protected in R1/R10. This means when I create the object, or modify it, the system needs to keep two copies, generally on separate physical hardware, and in some cases in separate geographies.

This sounds very much like the old Lefthand, now StoreVirtual concept, where data is constantly replicated to keep multiple copies. The major difference is that the storage is accessed via an API, application programming interface. Think about drop box for a second. I place a file in my drop box folder on my laptop. It is replicated up to the “cloud”, and is available on my iPad, iPhone, web browser, etc. I didn’t tell it what do do, I just placed a file in the folder and it was suddenly available everywhere I am. I can have the same object on a Mac as a Windows PC without worrying about incompatible file systems, permissions, many of the things which are inherent to the file storage models. The great part is I don’t worry about backing up the file when I image my laptop because it comes right back when I link my freshly imaged laptop to the application which is calling the API.

What is happening here is that programatically, storage is being controlled, the object, my file, is being replicated. I don’t think about anything going on in the back end, it is simple, and it is cost effective. I am using the object storage as a repository, so there is not a performance expectation.

The performance in and of itself is an interesting discussion point. This is not something I am going to run my virtual desktop environment on, at least not just yet, but it is a great way to throw some large inexpensive drives out and present them to my end user community as a way to store data. This enables me to archive more effeciently, and share files across heterogeneous environments.

So what does this have to do with VMware? When we design a VMware environment, we often look at just the virtual infrastructure. We think about todays statefull applications, and we assume virtualization will just simplify our lives. It is critical to take a step back and think about what changes with a virtual environment. If we are simplifying and consolidating, we are also introducing new complexities, and potentially opening the door to a whole new series of challanges. Using our expensive primary VMware storage as an archive platform may not be the wisest solution, having a windows VM chugging away on our virtual environment may be much more costly.

One of my favorite sayings is, there are nine ways to skin a cat, no offense to cat lovers, and as we design VMware storage solutions, it is critical to look at the project on a larger level. As I learn more about Openstack, I will be posting more on the Software Defined Datacenter and what I believe that really means.

Storage networking is an interesting discussion. This is usually pretty dependent on your storage vendor, or your personal preferences, but this is often also a misunderstood issue. When I was new to storage, we used to joke about iSCSI, real men used Fibre Channel. File storage like NFS was someone else’s problem, I was only interested in block, fibre channel storage. In a VMware environment, it becomes increasingly clear that we need to examine all types of storage networking to make sure that we have the right fit. Now I did talk a bit about this topic in previous posts on SAN and NAS, but I feel it is worth discussing in greater detail.

To start, VMware fully supports NFS, linux Network File System, and this is a perfectly logical way to handle storage in this environment. It is simple, it works much like a traditional file system, it is easy to grow, and is generally thin provisioned by default. There is no VMware File System, VMFS, to manage, the NFS server provides the file system. In cases where datastore sizes need to change often, and cases where redundancy is less important this might work. The main draw here at this point is simplicity of provisioning, and management. This can happen over 1GbE and 10GbE, and this really has to be decided based on network bandwith needs. It can be done using a converged network, but should always be on a separate subnet from the rest of the network traffic.

The major downside of NFS is that it is sometimes treated by VMware as a second class citizen. Most times when VMware releases new features from a storage perspective they are released on block storage. NFS is usually not far behind, but it does take time. Another downside is a lack of multipathing. Now I will say there are ways to do multipathing with NFS in a VMware environment, but it is more complex and not always a standard. Finally NFS is heavily reliant on the NFS server. While there are some good systems out there, there is also a reason a majority of the largest deployments have opted to use block storage for VMware. It is more trusted, and gives more options for storage vendors.

iSCSI is another IP based protocol, supported in both 1GbE and 10GbE. This is a great protocol for small to mid sized environments. It is simple to configure, and runs on the traditional IP network, using traditional IP switches with a few minor modifications. iSCSI is attractive because it generally keeps costs down. It also can run over a converged network, but should be isolated by VLAN’s at a minimum, and dependent upon your storage vendor may take advantage of multiple paths for redundancy or performance. Modern storage arrays are moving away from 1GbE in favor of 10GbE for iSCSI, which is something to consider if looking at this as an option.

The major downside of iSCSI is that it is so easy to deploy improperly, and is seldom designed correctly. Being an IP based protocol it is not purpose built for storage, so there is inherent latency. Jumbo frames and flow control, though widely debated, often time can have tremendous impact on the storage in a VMware environment. With 1GbE networking in larger environments, or 10GbE in a converged network, speed and complexity can be a factor. In a future post I will discuss storage network design in more details, but this should give a high level idea of some of the challanges.

Fibre Channel Networking, in the enterprise at least, is probably the most common protocol. The major advantage is this is dedicated. A fibre channel network is designed to transmit the SCSI commands between the storage and the host server. This becomes particularly important in a virtualized environment. Again more on this in a future post on storage network design. The major advantage here is that the latency is very low, and the switches are not having to handle any other traffic. In a virtualized environment, we also want to consider the number of servers per host we are using. In a traditional physical server if we have 2 fibre channel hba ports on the servers, and we virtualize 20 servers per host, we now have 10 times as many servers using the same bandwith. In this context, the low latency and lack of resource contention on the storage network is much more important.

The downside for Fibre Channel is cost and complexity. This requires dedicated specialized switches that cannot be used for anything else outside of storage. They are costly to deploy and to manage when compared to a typical IP network. They are also complex. They are often times outside the traditional network teams expertise so they will leave management to the storage team.

Finally there are of course Fibre Channel over Ethernet, FCOE, and Serial Attached SCSI, SAS, but these are less common in most environments and have not as of yet gained wide adoption.

At the end of the day these all get us to the same place, and all have their merrits. There are pros and cons to each, and I will talk more about the actual network design soon. It is good to know about each and where it may or may not be a good fit in a particular environment.

No sooner do I get a post up about VSAN and how I don’t think this is a major storage play, but rather a lower end play, than I see this on twitter…

Now I want to say, if your not reading guys like Duncan and Scott to name just a very few, then you are missing a majority of what is happening in the virtualization world. With the recent hires at VMware technical marketing, the future is anything but boring. That being said, I love speculating, and I do realize this is typical for VMware to push the envelope, release new and innovative products and then make them even more awesome.

I would have to say though this is significant if it means what it seems to. There are so many amazing products on the market now from a storage perspective alone, that VMware would be missing a huge segment if they didn’t work on this. I do stand by my thoughts though, this doesn’t kill competition, it just makes the rest of us work harder to innovate.

Competition is never a bad thing, it can bring out the best in us. I look forward to seeing what this means for those of us who have invested heavily in the future of VMware, from a career standpoint, and those who are using the products. What an exciting time to be alive and in technology.