Cloud Files Feature Release

// August 12th, 2009 // 3 Comments » // Development

By EJ Johnson

Today marks the release of a new feature to Cloud Files as well as a couple of other changes we wanted to highlight.

CDN LOG RETENTION

This new feature gives customers the ability to save their raw CDN web logs to their Cloud Files storage account.

One of the biggest attractions of Cloud Files for our customers is the ability to serve files over Limelight Networks. By partnering with Limelight, a tier one CDN provider, we’ve put real CDN power into the hands of everyone so they don’t have to worry about signing contracts or setting up additional infrastructure.  One drawback we’ve had was having visibility into their CDN usage.  Now, by offering the ability to store your raw CDN web logs, we are taking our first step at providing more insight into your usage of Cloud Files and CDN.

A couple of details about this new feature:

• This is an “opt-in” feature. Log files will only be saved if this new log-retention setting has been set on a CDN-enabled Container.  We recognize that most users will want access to these logs; however, since the data will be stored in their Cloud Files storage account, we didn’t want everyone to start paying for additional storage unless they choose to.

• There is no automatic purge of saved logs. Once logs are saved to Cloud Files, they will stay there unless explicitly removed.  As we’ll discuss below, the Object name of each log file includes a date/timestamp so it would be fairly straightforward to set up a weekly or monthly script that purges old logs.

• Although logs will be periodically saved to the storage account, there is no predictability as to when logs will be processed and saved. It depends greatly on the accumulation and transfer time from Limelight and also on the processing time of Rackspace systems.  The time between a “hit” of a CDN-enabled Container/Object and the time for that web log entry to appear in the storage Container could be anywhere from 8 to 24 hours.

• Once enabled, logs will be saved to a Container named “.CDN_ACCESS_LOGS“in the customer’s storage account. The Object names for each log file will have a unique name like: container_name.YYYYMMDDHH-XXXX.log.gz. The “container_name” will be the name of the CDN-enabled Container. The next part of the name represents the year, month, day, and hour that the log file was processed.  The XXXX part is a zero-padded 4 digit number that prevents name collisions if multiple logs filtering in from various Limelight geographic locations during the same hour period.

• The log files are stored in [gzip - http://www.gzip.org] compressed format to save on storage and bandwidth costs. The naming convention allows the logs to be sorted lexicographically by Container name and then by date.

• This feature can be enabled via the ReST API of the CDN management system or via any of our supported language API’s hosted on github.com.

• We expect third party tools and other language bindings to add support for this feature shortly. Please see the Developer Guide for the specifics regarding the ReST API or the documentation included in each language binding.

[API Dev Guide and Documentation]

NEW CDN URI FORMAT

There has also been a change to the URL format of a CDN-enabled Container.  Previously, when a Container was CDN-enabled, the URL looked like this: “http://cdn.cloudfiles.mosso.com/c10171″. The new format is: “http://c00010171.cdn.cloudfiles.rackspacecloud.com”. Of course, one change is the hostname part of the URL to align with our re-branding. The other change shifts the “unique” identifier from a path element to a prefix of the hostname.  The benefit to customers is mostly edge-case.  It provides the ability for customers to serve CDN content at the root-level of the URL rather than dangling off a path element.  This is critical for Flash players needing to specify a cross-domain policy that enables serving Flash content from their site via the Cloud Files CDN.

Both URL formats will work.  We realize that many of our existing customers have probably directly referenced the old CDN URL format in web sites, blog posts, etc.  Just be aware that the new format will be returned via the API even on Containers that were CDN-enabled prior to this change.

MULTI-FILE UPLOADER IN CONTROL PANEL

There has also been a recent update to the Cloud Files web interface in the Control Panel that allows users to upload more than one file at a time.  Users also have the ability to adjust the TTL setting on a CDN-enabled Container.  Valid values for the TTL setting are between 1 and 3 days.

There have been a few bug fixes included in this release, most notably to resolve issues with persisting/viewing Object metadata.

We will be issuing a follow-up release very shortly that will also allow users to enable CDN access log retention from the control panel.

Currently this new feature is only configurable with the ReST and language APIs.

WHAT’S NEXT FOR CLOUD FILES?
============================

We are always listening to what our customers’ needs are.  Having access to CDN logs was one of the most highly requested features for Cloud Files. There are still a few remaining “most requested” features that we are working on.  The new CDN URI format change is a big step as it sets us up to provide true CNAME support for CDN-enabled Containers. Rest assured, we’re still working to make that available as soon as we can.

We’ll also be coming out with another feature with respect to the CDN component of Cloud Files.  I don’t want to reveal the details yet, but we expect this to be a very important feature to Cloud Files customers using the CDN. Stay tuned.

Have more questions?
====================

Feel free to hit us up on chat or give us a call if you have other questions.  You are also welcome to join us on IRC for “unofficial” support and talk tech at irc.freenode.net on #cloudfiles.

Cloud Files Emerges!

// March 13th, 2009 // 3 Comments » // Community, In the news

Yesterday Cloud Files exited our public Beta period and entered General Availability.

We launched Cloud Files in private Beta almost a year ago, and it’s been in public Beta since last October. Since then, we’ve been solidifying the infrastructure and making sure we’re wired up with Limelight Networks to power our low-latency, globally-available content delivery network. We’ve also been working on another round of features that we know customers will be excited about - all of which I’ll cover in more detail below.

But for those of you who are not already familiar with Cloud Files, the two most popular use-cases are:

Content Hosting and Delivery:

Cloud Files is the first and only cloud service that leverages a tier one “content delivery network (CDN)” provider to create a complete, pain-free, hosting solution for storing and delivering media content.

· Serve media to your users around the globe – fast!

· Utilize CDN bandwidth starting at 22 cents a GB

· Sign-up with no minimum commitments or contracts.

· Easy online and programmatic interfaces for publishing to the CDN.

Storage and Backup:

Cloud Files is a reliable, scalable, and affordable web-based storage for backing up and archiving.

· High performance, redundant storage, starting at 15 cents a GB.

· All data is replicated across three separate locations.

· Use as much or little as you want and pay only for what you use.

· Store individual files as large as 5GB.

· Manage your files programmatically or through our online interface.

Pricing

* When you access Cloud Files using the Mosso | The Rackspace Cloud online control panel, the bandwidth you use (both incoming and outgoing) and the requests you generate are free of charge.

Other CDN costs:

No request/operation charges

No bandwidth charge from Cloud Files to CDN

How do we compare to the competition?

· Pricing: for many use-cases, Cloud Files is cheaper than other cloud storage offerings on the market today. For a detailed breakdown, you can review one of our previous posts on http://www.rackspacecloud.com/blog that also includes an Excel spreadsheet to run your own what-if scenarios.

· Support: We’re Rackspace. If you know us, you know what that means – we’ll do everything we can to make sure you get answers to your questions to help get you up and running. The Rackspace Cloud is powered by humans that are there for you 24×7x365 at no additional cost.

What’s new?

We have a long list of new features; too many, in fact, to cover in any great technical detail in this blog post. If you’re interested in more detail about these features, you should review our documentation and “coming soon” wiki site.

· Increased name limits: Container names can now be 256 bytes long and Object names can be up to 1024 bytes long.

· JSON or XML list results: When listing Containers or Objects, users can request that the output be formatted in JSON or XML. This format also provides additional detail about the items such as total bytes stored in a Container, the Object’s content-type, and size to name a few. This is a great way to quickly grab detailed information about your data without having to iterate over each item and issue a separate request for this detail.

· Pseudo Folder/Directory Nesting: In response to numerous requests to provide a means to create a nested-folder/directory structure, we have a technique that users can utilize to mimic this behavior. Users can create Object names with the ‘/’ path separator and “directory marker” Objects to seed their storage account for this support. To traverse these pseudo folders/directories, users can specify a “path” query parameter to only retrieve the Objects that appear to reside under that path.

· Chunked PUT Requests: Users also frequently request the ability to stream data into Cloud Files without having to buffer it to a disk to calculate the file’s size for the Content-Length header. Now users can use the Transfer-Encoding: chunked header on PUT requests. Users will need to remember that our upper limit is 5GB and anything larger than that will cause the server to disconnect the client.

· Replacing “offset” with “marker”: Listing Containers or Objects used to support limit/offset integer parameters to “page” through their items. We’ve replaced the offset parameter with a new string parameter called “marker.” It works functionally the same way but will return item names “greater than” the marker value. Generally, the name of the last item in the current result set would be used as the next marker value to continue iterating through the full set of items.

· A Ruby API: We’re continuing to add new language API’s to make it easier for developers to integrate Cloud Files into their applications. In addition to PHP, Python, Java, and C#/.NET, we are adding Ruby to the list. All of our language API’s support the full feature list of Cloud Files. When we say we “support” an API we mean that we keep it in lock-step with our features so that you’re not dependent on an external community to adapt to upstream vendor changes.

· Jungle Disk on Cloud Files: Our sister company, Jungle Disk, providing online backup software, now offers the option to store data on Cloud Files paving the way for a unified buying experience and new simple pricing.

What’s next?

Are there other things you want? Yes, of course there are! Here is a sneak peek of a few of the things you can expect to see this year, so be sure to keep a look-out!

· CNAME Support for CDN

· Restrictions on CDN content access

· Access to CDN web logs

· Build-outs of Cloud Files in other regions

Cloud Files with Limelight Networks

// December 1st, 2008 // 4 Comments » // Development

Cloud Files is a new product offering from Rackspace/Mosso that is an inexpensive, scalable, dyanmic storage system.  We’ve partnered with Limelight Networks’ Content Delivery Service to bring you an incredibly easy way to publish your content over a world-class, industry leading CDN.

The core storage system is designed to provide a safe, secure, automatically re-sizing, and network accessible way for you to store your data.  You can store files ranging in size from a single byte up to 5 gigabytes and you can store as much as you want and only pay for what you use.  With Cloud Files there is no over-buying/under-utilizing storage space.  All data is secure and private and only accessible to that user account.  This is a great solution for storing system backups, copies of your digital media such as photos and videos, or as a storage system available to multiple computers.

By combining the core storage system with Limelight Networks’ CDN, you now have an easy way to distribute your data to everyone.  Limelight Networks has 18 edge locations over the globe with media-grade optical networks connecting them.  They provide CDN service to over 1300 businesses(1) who rely on their network to be performant and continuously available.  Our partnership with them is not a one-off solution; when you publish content through them it is distributed across their entire infrastructure just like all of their other customers.

• Effortless: Non-developers have a simple web-interface that can be used to quickly and easily upload data and enable CDN access.  Users can, within minutes, sign up for Cloud Files, create a Container, upload a file and publish that Container’s content through the CDN.  This is great way to share out large video files with friends and family rather than trying to email large files to multiple recipients.  If you are a developer, we have a simple ReST web service and language-speicifc API’s in PHP, Python, Java, and C#/.NET.  All of the API’s are easy to use and provide documented examples to help you get started quickly.  They provide full support for managing your content in Cloud Files and publishing that content over the CDN.

• Affordable: Storage starts at .15 cents/gigabyte and bandwidth costs start at .22 cents/gigabyte.  You only pay for what you use and the cost decreases as your usage increases!  There are no additional costs to publish your content behind the CDN either.  In addition to storage and bandwidth, there are also some nominal transaction costs.  Check out our pricing page for more information [http://www.mosso.com/pricingfiles.jsp]

• Redundant: The storage system has been designed to be highly available and fault tolerant.  We store 3 full copies of your data on 3 separate storage nodes, but we only charge you for one copy ;-).  Storage nodes are grouped in logical Zones within our datacenters.  Zones are connected to redundant internet backbone providers and reside on redundant power supplies and generators.  If you publish your content behind the CDN, that also provides an additional layer of data redundancy.

• Secure: All uploads/downloads of data to the Cloud Files storage system is performed over SSL.  Requests against the storage system are only allowed if they contain a valid authentication token.  Authorization tokens are granted when a user successfully authenticates to the Mosso Authentication Service.  Authorization tokens are “session” tokens and expire over time requiring the user to re-authenticate periodically.

How it works
In the Mosso web-interface, it’s as simple as creating a Container (the storage compartment for your data), upload your Objects (the files you want to serve over CDN), and mark the Container as “public”.  The Container is then assigned a unique URL that you can combine with your Object names to embed in web pages, email messages, blog posts, etc.  For example, you could upload all of your family photos to a Container called “images”.  When you publish that Container, it will be assigned a unique URL like, “http://cdn.cloudfiles.mosso.com/c1234″.  You could then share out a link to one of your photos like “http://cdn.cloudfiles.mosso.com/c1234/IMG_3432.jpg”.  When that link is accessed, the photo is served from the CDN; it’s that simple!

When that URL is accessed, the FQDN “cdn.cloudfiles.mosso.com” resolves to Limelight Networks’ CDN.  When they see the request for “c1234/IMG_3432.jpg”, they immediately serve the content from their cache if it exists.  If the photo doesn’t exist in their cache, the call back to Cloud Files for that Object and then serve it and cache it for the next request.  Data is cached in the CDN for a configurable amount of time (the default is 1 day).

When the CDN fetches the object for the first time, it is cached at an edge location that is geographically the closest to the requester.  As that file is requested from other geographic locations, the CDN propagates that file to other edge locations so that users are always accessing the file from the closest geographic location to them.  This is the magic that a true CDN provides.  Content is served to users from the closest edge location reducing latency and speeding up the end-user’s experience.

Once you CDN-enable a Container, any new Objects you upload to that Container are also available for access over the CDN just like the example above.  There is no complicated “access control list” that needs to be set on each Object you upload or re-setting the “public” attribute.

If at some point you no longer want your content accessible over the CDN, you can mark the Container as “private”.  Any data currently cached in the CDN will continue to be served, but when the CDN cache expires, the data will not be re-fetched/cached from Cloud Files.

What about CNAMES?  I don’t want to use http://cdn.cloudfiles.mosso.com/cXYZ!
A request we anticipate some users will want will be the ability to provide a custom name for the CND URL.  For instance, you might not want to use “http://cdn.cloudfiles.mosso.com/c1234/image.jpg” but would rather use a name in your own domain such as “http://img.me.com/image.jpg” in your web sites.   Since this is not currently a supported feature of Cloud Files, we’d like to suggest an alternate way for you to use your own custom domain with Cloud Files.
You can use a custom URL in your own domain by using a concept called URL re-writing.  So let’s say you CDN-enable a Cloud Files Container and it is assigned “http://cdn.cloudfiles.mosso.com/c1234″.  Most popular web servers support a feature called URL re-writing that can be configured so that a request for “http://img.me.com” is actually redirected to “http://cdn.cloudfiles.mosso.com/c1234″.  With this re-writing scheme, a request to an image would be automatically redirected to the Container’s CDN URL which in turn is served by the CDN.  Anyone familiar with the Apache mod_rewrite module could use something like this:

RewriteEngine On
RewriteRule ^http://img\.me\.com/(.*)$ http://cdn.cloudfiles.mosso.com/c1234/$1 [R,L]

So with this basic technique, you are in full control of how visitors access the content you publish with the Cloud Files CDN.  This technique can be used to capture web log hits, restrict access to specific users, or direct hits back to your local web server if you decided not to use Cloud Files CDN.

1) [http://www.limelightnetworks.com/network.htm]