Home Articles About Us Contact Us

The Mechanics of Using Azure Storage Tables for Logging

Article
Before reading this article, you may want to check our Logging Using Azure Storage Tables article first.

For the sites that we host, we use one Azure storage table for each site. Before going any further, it is important to understand a little bit more about how Azure storage tables work.

Azure tables are a non-relational structure with no indexing other than the default “partition key” and “row key”. The partition key segregates entities into different storage areas. Searches against items in the same partition key are much quicker than searches against items with different partition keys. For this reason, in our logging solution we use a fixed partition key for every entry in the table. The row key has to be unique across the whole table and is the only way of sorting the table. I.e. rows are always returned in the order dictated by the row key. Each row also includes a timestamp column. This is for internal purposes only and should not be used directly.
Choosing a row key for our logging purposes is tricky as we would ideally like the rows to be in order of when they were logged, but we could realistically have two Azure instances (or two threads on the same multi-processor instance) writing at exactly the same time. This precludes using the time as our row key. The solution we came up with was to use a time stamp, formatted as YYYYMMDD HH:MM:SS, suffixed by a GUID. This gives us an ordered list of entries but also guarantees uniqueness across the system.

Start by defining a class representing your Azure table structure. You can add columns to this class at a later date and they will be automatically added to the table. Azure tables are very loosely bound and may be considered a super-set of all the record entries stored in them. Below is an example message class that can be used to store log messages. This is a very simple class that just stores a message. You will probably want to include many other fields, such as the date/time, type of message, etc. You will need to include the Microsoft.WindowsAzure.StorageClient library for the TableServiceEntity class.

public class Message : TableServiceEntity
{
    private const string MessagePartionKey = "LogEntry";
    private const string DateFormat = "yyyyMMdd ; HH:mm:ss:fffffff";
    private const string RowKeyFormat = "{0} - {1}";

    public string LogMessage { get; set; }

    public Message()
    {
    }

    public Message(string logMessage)
    {
        PartitionKey = MessagePartionKey;
        string date = DateTime.Now.ToUniversalTime().ToString(DateFormat);
        RowKey = string.Format(RowKeyFormat, date, Guid.NewGuid().ToString());
        LogMessage = logMessage;
    }
}


You can see that we have used “LogEntry” for the partition key and a date format similar to the one discussed earlier for the row key. Also note that we convert all date formats to universal time for consistency across all the servers that your application may be deployed on. The default constructor is included to allow instances to be created when the data is read back in, as it is advisable to keep this class in a shared class library for both reading and writing.

To store a log message, we can use something similar to the following.

public void StoreNewLogMessage(Message logMessage)
{
    CloudStorageAccount account;
    if (CloudStorageAccount.TryParse(CloudStorageAccountName, out account))
    {
        CloudTableClient tableClient = account.CreateCloudTableClient();
        tableClient.CreateTableIfNotExist(LogName);
        TableServiceContext tableContext = tableClient.GetDataServiceContext();
        tableContext.AddObject(LogName, logMessage);
        tableContext.SaveChangesWithRetries();
    }
    else
        HandleInternalError(“Cloud storage could not be opened.”, logMessage);
}


LogName is a property elsewhere in your class that returns the name of the Azure storage table to use. CloudStorageAccountName is a property elsewhere that returns an Azure storage connection string. This would usually be retrieved from a configuration file. To find your storage account credentials, open the Azure Management Portal and locate your storage account. When you select your account, you will see an option to view the Primary Access Key. You can construct a connection string in the format below:

“AccountName=[Your Storage Account Name];AccountKey=[Your Primary Access Key];DefaultEndpointsProtocol=https”

The final part of the function checks if opening the storage account has been successful. If it hasn’t, you will need to find an alternative way of sending the information to alert that the logging is not working.

Comments

jim 3/14/2013 2:48:48 PM
Nice approach!
Do you know what is the best solution for analyzing the log data afterwards when you are storing your logs on Azure Storage Tables?
Symatix 3/14/2013 3:24:08 PM
Hi Jim

What we do here, is that we have a local web server which runs our bespoke analytical software. When anyone looks at the logs, analyses trends, etc., using the website on our local intranet, the latest logs from the Azure tables are retrieved, parsed and written to a local database. The Azure tables are then cleared down to minimise the use of Azure storage. Our analytics website then uses the data from the local database to provide the information to the user.

Hope this helps.
Add a Comment




Your email address will not be displayed or used for marketing and is for verification purposes only.


Nobody likes those difficult to read captcha systems that are used to prevent spammers, so we've tried to make life easier. Simply add together the following two numbers to show that you're a real person.
9         16

Related Articles

Azure Deployment

When performing an Azure deployment, there are several factors that should be taken into consideration. Microsoft Windows Azu.....

Azure Table Storage

Azure table storage is an extremely useful feature of the Microsoft Windows Azure offering. The cost per transaction is extre.....

Microsoft Windows Azure - Logging Using Azure Table Storage

Azure logging can be a thorny issue. Embracing the move away from traditional server based deployments to hosted services bri.....

Microsoft Windows Azure - Definitions

The following is a list of definitions for Microsoft Windows Azure topics.....

Follow us on Facebook to get instant access when a new article is added.

To get instant access to new articles that are published, follow us on Facebook.