06/06/2017 |

S3 compatibility? It's just a feature


This month we closed 3 major multi-petabyte deals, for very large clusters, in three different market segments. Size apart, the only thing they have in common is the requirement for AWS S3 compatibility. And this is also true for the smallest installations we are working on.

S3, the de facto standard

Today, if you want to use object storage, there are a number of very important characteristics that end users, partners, and the industry in general consider fundamental. The list has gotten long, but one thing that stands out is S3 compatibility.
S3, a protocol and service introduced by Amazon AWS in 2006, has been adopted by developers, software products, and an increasing number of competing services. It's not the best API, and it doesn't take advantage of all the features available from the object store (for example, in the case of SDS, there is an entire set of bulk operations optimized for large data movements that are not available on S3), but almost nobody cares. It would be like questioning iSCSI and FC for a storage array, or NFS and SMB for a NAS. Can't you have something better? Yes, probably, but do the vast majority of end users care? Not at all.

S3 is a feature, not a productObject Storage on ARM: Cost Effective, Efficient, and Easy!

Several times in the storage industry, vendors have associated their products with a single feature because they invested a lot in it, and because, at that time, it was a unique differentiator.

  • Data domain was a synonym of deduplication. (Do you know a VTL today that doesn't do that?)
  • 3PAR was a synonym of thin provisioning. (Do you know an array today that doesn't do that?)
  • ROW Snasphot was a synonym of NetApp. (Do you know a NAS today that doesn't do that?)

And these are only three examples; the list is very long.
In the short term, associating your name to an innovative feature and making it the core of your product differentiation is a good strategy, and it can pay off. But this kind of strategy works for a limited time, until the rest of the industry catches up. At that point you must understand that even if you are still better than others, it no longer makes a difference; you must be ready to change your product and message accordingly.

Strategy and tactics around S3

The OpenIO team started working on SDS in 2006, and its first 1PB cluster has been in production since 2009. This is why SDS is mature and robust, even though the company was founded only 3 years ago.
The team has been working since then to build a strong core architecture and a long-term strategy based on a lightweight design, flexibility, and ease of use at scale.

To be fair, in 2006, S3 wasn't around. The set of APIs built to manage objects is comprehensive, and it made it possible to build front-end connectors for several protocols and services (S3, Swift, email, files, and so on), as well as unique features like Grid for Apps (an integrated event-driven data processing framework).
This long-term strategy allowed us to build a product that:

  • Supports different hardware architectures (ARM, X86),
  • Runs in a very small footprint (1 CPU core and 512MB RAM),
  • Scales like crazy (our largest customer has 650 nodes in production, but small customers can start small and grow without any impact on performance),
  • Is particularly efficient (by taking advantage of all the resources available in the cluster with dynamic load balancing),
  • Has incredible performance (thanks to the ability to work with flash storage and HDDs efficiently),
  • Is future proof (allowing customers to change the configuration over time without impact),
  • And it can run applications directly into the storage infrastructure (enabling end users to further improve the efficiency of the infrastructure, and get faster results).

In short, our strategy is built on flexibility and freedom of choice. This is why end users like us, and why we win! (BTW, how many object storage vendors can say this today?)
And, I won't lie; in the past, our S3 interface was poor and supported only a subset of the available APIs. Now, thanks to the effort of our development team, and the help of some end users, we are on par with the best implementations on the market, and our customers have proven this in the field. This enables us to move on to discuss what really differentiates us.
S3 is a feature, a very important feature indeed, even for us. But, again, it’s just a feature. It's like saying that a NAS supports SMB. Customers take for granted that when you mention SMB for an enterprise product it supports AD, SMB3, and so on, don't they?
S3 compatibility is very important for our customers, and I'm really glad we now have good S3 compatibility; but again it's just a feature, isn't it? Would you even think of talking to an object storage vendor today who doesn’t have good S3 compatibility?


OpenIO SDS’s strategy is to build the most flexible and efficient object storage on the market with unique features that go beyond data storage, enabling our customers to process data directly on the storage infrastructure in a serverless fashion.
At the same time, we constantly listen to our customers to understand their needs and add features accordingly.

This is why OpenIO SDS is leading in traditional object storage use cases and is ready to help you with the challenges posed by next-generation applications in Industrial IoT, machine learning, and artificial intelligence. How many object stores can do all this?

Want to know more?

OpenIO SDS is available for testing in four different flavors: Linux packages, the Docker image, and Raspberry Pi.

Stay in touch with us and our community through Twitter and our Slack community channel, to receive the latest info, support, and to chat with other users.

Object Storage on ARM: Cost Efficient, Effective, and Easy with OpenIO on Ambedded Mars200