SQLiteWinRT: Now BLOB columns and Windows 8.1 supported

I’ve posted some updates to the SQLite WinRT API at http://sqlwinrt.codeplex.com. The SQLite WinRT API offers lightweight Windows Runtime (WinRT) APIs you can use to access the most common SQLite database functionality by using SQL statements, rather than a LINQ-style API.

Support for Windows 8.1

One change is to add an implementation for Windows 8.1. So if you download the source code now (there’s no msi or NUGet package for this as yet), you’ll get three projects which are the WinRT components for Windows 8.0, Windows 8.1 and Windows Phone 8.0. Make sure you have installed the SQLite SDK for your target platform on the Tools – Extensions and Updates menu and then Just include the appropriate project in your own solution, and you’re good to go. See my earlier post for an introduction to programming with the API: http://andywigley.com/2013/06/06/sqlite-winrt-database-programming-on-windows-phone-and-windows-8/

Namespaces Standardised

The other relatively minor change is that I’ve standardised the namespace used to SQLWinRT for all platforms. It just makes it a little easier to share code across platforms that way. That does mean that the blog post referenced in the previous paragraph shows code using the old SQLiteWinRTPhone namespace – though the sample accompanying that post has been updated.

Support for BLOB columns

There is a major update in functionality though – support for BLOB columns, which was requested to help store byte arrays – or Images – directly in the database.

You can define a table in the database with a BLOB column using code similar to this:

// Get a reference to the SQLite database
db = new SQLiteWinRT.Database(ApplicationData.Current.LocalFolder, "sqliteblobdemo.db");

await db.OpenAsync();

string sql = @"
    CREATE TABLE IF NOT EXISTS
        Pictures(Id     INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
                 Name   VARCHAR(140),
                 Image  BLOB);";
await db.ExecuteStatementAsync(sql);

Then, to store a record in this table, you could do something like:

private async void Insert_RecordAsync(string description, BitmapImage image)
{
    string sql = @"INSERT INTO Pictures (Name, Image) VALUES (@name, @image)";
    var db = await App.GetDatabaseAsync();
    using (var statement = await db.PrepareStatementAsync(sql))
    {
        // Convert BitmapImage to byte array
        byte[] imagebytes = ConvertToBytes(image);
        // .. and from that to an IBuffer
        Windows.Storage.Streams.IBuffer imagebuffer = imagebytes.AsBuffer();

        // set the statement parameters
        statement.BindTextParameterWithName("@name", description);
        statement.BindBlobParameterWithName("@image", imagebuffer);

        await statement.StepAsync();
    }
}

private byte[] ConvertToBytes(BitmapImage bitmapImage)
{
    byte[] data = null;
    using (MemoryStream stream = new MemoryStream())
    {
        WriteableBitmap wBitmap = new WriteableBitmap(bitmapImage);
        wBitmap.SaveJpeg(stream, wBitmap.PixelWidth, wBitmap.PixelHeight, 0, 100);
        stream.Seek(0, SeekOrigin.Begin);
        data = stream.GetBuffer();
    }

    return data;
}

Here’s an example to read data out again:

    protected override PictureViewModel CreateItem(SQLiteWinRT.Statement statement)
    {
      // Read the image from the BLOB column
        Windows.Storage.Streams.IBuffer blobBuffer = null;
        if (statement.GetColumnType(2) != ColumnType.Null)
        {
            blobBuffer = statement.GetBlobAt(2);
        }

        // Convert IBuffer back to a BitmapImage
        byte[] pictureBytes = blobBuffer.ToArray();

        var bitmapSource = new BitmapImage();

        using (MemoryStream ms = new MemoryStream(pictureBytes))
        {
            bitmapSource.CreateOptions = BitmapCreateOptions.None;
            ms.Seek(0, SeekOrigin.Begin);
            bitmapSource.SetSource(ms);
        };

      var c = new PictureViewModel(
        statement.GetIntAt(0),
        statement.GetTextAt(1),
        bitmapSource);

      return c;
    }

Here’s a complete sample app you can study to help implement this functionality in your own apps:

Sample code SQLiteBlobSample.zip

Push Notifications made easy: Using Windows Azure Notification Hubs with Windows Phone

When I built the Windows Phone 8 Jump Start training, one of the sessions was on Push Notifications. The demo app I used was an update of an old favourite, a Weather app which first saw the light of day a few years ago with the initial demos built for Windows Phone 7.0. In the demo, you have a phone app and there is also a WPF desktop app.

The WPF desktop app allows the ‘operator’ to select a city, temperature and weather conditions and then generate tile, toast and raw notification messages to communicate the weather forecast to the phone, but the WPF app is actually more complex than that and actually fulfils two key roles in a push notifications scenario, as it also hosts a WCF service and maintains a list of subscribers, simulating the cloud service that you would build for a real Push Notifications solution.

PushBasics

In summary, the phone registers for push notifications1 and receives a unique URI (the channel URI) which it then sends off to your cloud service2 (the WCF service) and the cloud service maintains a list of all subscribers for this app. Then the backend logic (the WPF UI in this demo) reads the list of subscriber channel URIs when it has something to communicate to the phone and sends the tile/toast/raw payloads to the subscriber URIs3. The subscriber URIs are actually endpoints hosted on MPNS (Microsoft Push Notification Services), so MPNS actually takes care of delivering the payloads to the phone4.

Fine – except that sample is difficult to setup and run, since you have to run Visual Studio as administrator otherwise the WCF self-hosting doesn’t work, you have to open port 8000 in your firewall because that is the port the WCF service listens on, and you have to edit the phone app code to set the correct hostname of the PC where the WCF service is running. Those are all implementation difficulties, but the real crime of this demo is that it’s too simplistic a demo and gives the impression that writing a Push Notifications solution is easier than it really is. In a real world app, when you post to MPNS you get back a response that gives you information about the state of the phone app subscription. There’s a whole page in the documentation about what you might get back: Push Notification Service response codes for Windows Phone and even if you ignore most of that, at the very least you should be removing subscribers from your list if you get a 404 Not Found response back, indicating that the subscription channel URI you posted to is no longer valid.

In fact, writing backend logic to send push notifications and correctly react to the Push Notification Service response codes is hard. And if you want to send push notifications to more than one client platform, such as Windows 8, iOS and Android, it’s incredibly hard.

That is – unless you use Windows Azure Notification Hubs Smile

Windows Azure Notification Hubs

Windows Azure Notification Hubs is a service running in Windows Azure that makes it really easy to send push notifications to mobile clients. You can read an overview here, but in essence what it does is remove the need to manage the subscriber list and create your own subscription WCF service from your backend logic and provide an easy to program API for sending notifications – and not only to Windows Phone, but to Windows 8, iOS and Android clients too.

But the benefits of Notification Hubs are not only for those building a cross-platform solution. Even if you are building a push notifications app for a single type of client, using this technology will greatly ease the creation of your solution.

I thought it would be interesting to do a new version of the Weather sample, but using Windows Azure Notification Hubs to make a comparison between the two ways of doing it, so for the rest of this post, I will walk you through what I did (download the sample code from the link at the bottom of this post).

From an architectural point of view, the main thing is that the custom WCF service and subscriber list disappears completely. Great, no need to build and host a custom web service in order to build a push solution! Instead, the notification hub fulfils that purpose and acts as a gateway between your backend logic and MPNS.

PushNotificationsWithWANH

Configuring the Notification Hub

First stage in building this is to configure the notification hub in Windows Azure, which is all done using the portal. You need to have a Windows Azure account and then go to the portal at http://manage.windowsazure.com.

  1. Click +NEW at the bottom of the screen.
  2. Click App Services, then Service Bus, then Notification Hub, then Quick Create.
  3. Type a name for your notification hub – I chose weathernotificationhub, select your desired region – West Europe for me, then click Create a new Notification Hub.
    WANHcreate
  4. You will now see the Service Bus namespace created screen.
    NotificationHubCreated

    Now, before you leave this screen, click on the Install the Windows Azure SDK (includes Service Bus client libraries) link to make sure you’ve got the client libraries installed in Visual Studio that you will need in both the phone app and also the desktop app.

  5. Click Connection Information at the bottom. Take note of the two connection strings – you will need them in a moment.
    AccessConnectionInformation

That’s it. Your Windows Azure Notification Hub is configured and ready to go.

Connecting the Phone App to Windows Azure Notification Hubs

Now let’s turn to the phone app. There aren’t many changes we need to make from the phone app we used in pre-notification hubs days. It still needs to register for push notifications using Microsoft.Phone.Notifications.HttpNotificationChannel and register for toast, tile and/or raw notifications. And when the app opens the channel, it will get a new Channel Uri reported to it when the ChannelUriUpdated event fires.

In the original app, it was at this point that you sent the channel Uri off to the WCF service hosted in the Weather Service desktop app, but instead we now send that off to our Windows Azure Notification Hub. To do this, you need to add a reference to the client libraries using the WindowsAzure.Messaging.Managed NuGet package.

In the Visual Studio menu, click Tools, then Library Package Manager, then Package Manager Console. Then, in the console window type:

Install-Package WindowsAzure.Messaging.Managed

and press Enter.

Now in the phone app, in MainPage.xaml.cs, we add the using Microsoft.WindowsAzure.Messaging using statement at the top, and then modify the ChennelUriUpdated event handler as follows:

async void httpChannel_ChannelUriUpdated(object sender, NotificationChannelUriEventArgs e)
{
    Trace("Channel opened. Got Uri:\n" + httpChannel.ChannelUri.ToString());
    Dispatcher.BeginInvoke(() => SaveChannelInfo());

    Trace("Subscribing to channel events");
    await SubscribeToServiceAsync();
    SubscribeToNotifications();

    Dispatcher.BeginInvoke(() => UpdateStatus("Channel created successfully"));
}

The SubscribeToServiceAsync() method is a new version of SubscribeToService() in the original demo. Instead of calling the WCF service (the commented out code), it now instantiates a Microsoft.WindowsAzure.Messaging.NotificationHub instance, which takes two arguments: the name of my notification hub (weathernotificationhub) and the connection string called DefaultListenSharedAccessSignature that you get from the Connection Strings display in the portal (step 5 in the previous section).

private async Task SubscribeToServiceAsync()
{
    try
    {
        var hub = new NotificationHub(
            "weathernotificationhub",
            "Endpoint=sb://weathernotificationhub-ns.servicebus.windows.net...[your value] ...");

        Registration registration = 
            await hub.RegisterNativeAsync(httpChannel.ChannelUri.ToString());
        Trace("Registered with Azure Notification Hub, Registration Id:" 
            + registration.RegistrationId);

        Dispatcher.BeginInvoke(() => 
            UpdateStatus("Registered with Windows Azure Notification Hub"));
    }
    catch (RegistrationAuthorizationException rEx)
    {
        Trace(rEx.Message);
        throw;
    }
}

That’s it as far as the phone client goes. Everything else stays the same: push notifications are delivered by the notifications infrastructure and handled in exactly the same way. Windows Azure Notification Hubs simply replaces your own subscription service, as far as the phone client goes.

Sending Notifications From Your Backend Logic

This is where the true benefit of Windows Azure Notification Hubs shines through. You no longer have to create your own web service with table storage to receive the channel URIs from your phone client subscribers, nor do you have to write logic to examine the status returns you get back from MPNS when you post notification messages and prune out stale registrations from the subscriber lists. You don’t need to write logic to post to each subscriber channel URI individually, just one call to the notification hub will cause the message to be sent out to all your clients. And if your backend logic needs to support Windows 8, iOS or Android clients as well as Windows Phone, the benefits of  Windows Azure Notification Hubs multiplies a hundred-fold.

Getting Registration Details from the Hub

In the Weather app, I started by deleting everything to do with the WCF registration service that we used in the previous version. Then fixed up the UI where it used to display things like “Waiting for connection…”. One thing the original did, was show the current count of registrations to the registration service. To do the same with Notification Hubs, you can call the NotificationHubClient.GetAllRegistrationsAsync(int32 top) method which returns details of all registrations limited to the top ‘n’ by the value you pass in the argument. In order to use the NotificationHubClient object, you need to add a reference to the client libraries using the WindowsAzure.Messaging.Managed NuGet package just the same as we did with the phone app.

After adding the NuGet package to my sample app, we show the number of clients on the screen using the following method:

private async Task UpdateRegistrationsCountAsync() { NotificationHubClient hub = NotificationHubClient.CreateClientFromConnectionString( "Endpoint=sb://weathernotificationhub-ns.servicebus.windows.net/;…", "weathernotificationhub"); var registrationsCount = await hub.GetAllRegistrationsAsync(Int32.MaxValue);

await Dispatcher.BeginInvoke((Action)(() =>
{ txtRegistrationsCount.Text = registrationsCount.Count().ToString(); })
);
}

The NotificationHubClient constructor takes two arguments: the first is the connection string you get from the Notification Hub Connection Strings page in the Windows Azure portal, but unlike the phone client which used the DefaultListenSharedAccessSignature, this time you need the DefaultFullSharedAccessSignature as this app is doing full interaction with the notification hub, such as posting notifications. The second arg is the name of your notification hub.

Sending Toast and Tile Push Notifications

Now to the main purpose of the app. Most of the logic around formatting and sending notification messages is encapsulated into the NotificationSenderUtility class in my sample. For example, this contains the following method to format the XML to send for a Windows Phone Toast notification:

private static string prepareToastPayload(string text1, string text2)
{
    // Create encoding manually in order to prevent
    // creation of leading BOM (Byte Order Mark) xFEFF at start
    // of string created from the XML
    Encoding Utf8 = new UTF8Encoding(false); // Prevents creation of BOM
    MemoryStream stream = new MemoryStream();
    XmlWriterSettings settings = new XmlWriterSettings() 
                { 
                    Indent = false,
                //   Encoding = Encoding.UTF8    !!NO-> adds Unicode BOM to start
                    Encoding = Utf8,    // Use manually created UTF8 encoding
                };
    XmlWriter writer = XmlWriter.Create(stream, settings);
    writer.WriteStartDocument();
    writer.WriteStartElement("wp", "Notification", "WPNotification");
    writer.WriteStartElement("wp", "Toast", "WPNotification");
    writer.WriteStartElement("wp", "Text1", "WPNotification");
    writer.WriteValue(text1);
    writer.WriteEndElement();
    writer.WriteStartElement("wp", "Text2", "WPNotification");
    writer.WriteValue(text2);
    writer.WriteEndElement();
    writer.WriteEndElement();
    writer.WriteEndDocument();
    writer.Close();

    return  Encoding.UTF8.GetString(stream.ToArray());
}

WARNING: While building this, I discovered that Windows Azure Notification Hubs are quite a bit fussier about the XML you send it than MPNS actually is. The original version of the code used the standard Encoding.UTF8 in the XmlWriterSettings (commented out in the code above) which results in an invisible Byte Order Mark (BOM) character being inserted at the front of the string, which causes Windows Azure Notification Hubs to barf when you send it that. The code above creates the XML without the BOM at the front which works fine. To read more about this, see my blog post BOMbin’ the L (aka Wierd Errors with XmlWriter).

Having prepared the payload, to send a Toast or Tile is simply a case of calling the NotificationHubClient.SendMpnsNativeNotificationAsync(string payload) method. The implementation in the attached sample is slightly more complex than this, but essentially it’s the following:

    NotificationHubClient hub = NotificationHubClient.CreateClientFromConnectionString(
        "Endpoint=sb://weathernotificationhub-ns.servicebus.windows.net/;...",
        "weathernotificationhub");

    NotificationOutcome outcome = await hub.SendMpnsNativeNotificationAsync(payload);

Sending Raw Notifications

The other thing the original sample does is send raw notifications directly to the app (if it happens to be running of course). The body of a raw payload is entirely up to the app developer, since unlike toast and tile notifications that may be processed by the Push Notifications handler in the phone OS, raw notifications are only ever handled by the app. So in the new version of this sample, the code building the raw notification payload is almost the same as before:

private static string prepareRAWPayload(string location, string temperature, string weatherType)
{
    // Create encoding manually in order to prevent
    // creation of leading BOM (Byte Order Mark) xFEFF at start
    // of string created from the XML
    Encoding Utf8 = new UTF8Encoding(false); // Prevents creation of BOM
    MemoryStream stream = new MemoryStream();
    XmlWriterSettings settings = new XmlWriterSettings()
    {
        Indent = false,
        //   Encoding = Encoding.UTF8    !!NO-> adds Unicode BOM to start
        Encoding = Utf8,    // Use manually created UTF8 encoding
    };
    XmlWriter writer = XmlTextWriter.Create(stream, settings);

    writer.WriteStartDocument();
    writer.WriteStartElement("WeatherUpdate");

    writer.WriteStartElement("Location");
    writer.WriteValue(location);
    writer.WriteEndElement();

    writer.WriteStartElement("Temperature");
    writer.WriteValue(temperature);
    writer.WriteEndElement();

    writer.WriteStartElement("WeatherType");
    writer.WriteValue(weatherType);
    writer.WriteEndElement();

    writer.WriteStartElement("LastUpdated");
    writer.WriteValue(DateTime.Now.ToString());
    writer.WriteEndElement();

    writer.WriteEndElement();
    writer.WriteEndDocument();
    writer.Close();

    return Encoding.UTF8.GetString(stream.ToArray());
}

I then had to do a bit of digging to figure out how to send this raw notification via Windows Azure Notification Hubs. One of the many methods of NotificationHubClient is SendNotificationAsync(Notification notification) which is a more generic Send* method than SendMpnsNativeNotificationAsync that we used before. It has the advantage that we can add HTTP headers to the request we send to MPNS, so we can define the X-NOTIFICATION header that a raw notification requires:

        // For raw notifications, have to use the more generic SendNotificationAsync() method
        Notification notification = new MpnsNotification(payload);
        notification.Headers.Add("X-NotificationClass", "3"); // Required header for RAW
        outcome = await hub.SendNotificationAsync(notification);

push end

Summary

Hopefully this has given you an introduction to building a push notification solution using Windows Azure Notification Hubs. There is plenty more to discover as I have only scratched the surface. For example, this sample sends notifications to all subscribers, but you can send to only a subset of subscribers using Tags. And of course, you can also send to Windows 8, iOS and Android clients, formatting the payloads for each client family yourself and then calling the SendWNSNativeNotificationAsync, SendAPSNativeNotificationAsync or SendGCMNativeNotificationAsync methods. An even easier way to do this, is to use the Templates feature, which is a neat way of sending a single generic message from your backend logic and allowing Windows Azure Notification Hubs to format the native message payload for each platform for you.

Here are some useful resources to allow you to dig deeper into Windows Azure Notification Hubs:

You should also be aware that Windows Azure Mobile Services uses Notification Hubs to send push notifications.

You can download the sample code described here yourself using the link at the bottom of this post. You will have to create your own notification hub and plug in your own hub name and connection strings, both in MainPage.xaml.cs in the phone project, and in MainWindows.xaml.cs in the desktop app – you should be able to find where easily enough.

 

BOMbin’ the L (aka Wierd Errors with XmlWriter)

[Title from a track on Come Find Yourself– Fun Lovin’ Criminals]

Have you experienced strange errors when working with XML generated by an XMLWriter? If so, this post may be helpful to you. I just spent a few hours trying to figure out why I was getting a weird error when I was sending some XML to the Windows Azure Notification Hubs client api (post on that coming up, just as soon as I have my sample working).

There’s a page here Get Started with Notification Hubs with tutorials on using this technology with Windows Store C#, Windows Phone, Android and iOS clients. In the Windows Phone tutorial, it walks through creating a Windows Phone client (needless to say) and a console app which simulates your backend logic that you use to send Tile, Toast or Raw notification messages to the mobile clients. In the instructions for the console app it includes the following:

In your Program class add the following method:

private static async void SendNotificationAsync()
{
    NotificationHubClient hub = NotificationHubClient.CreateClientFromConnectionString("<connection string with full access>", "<hub name>");
    string toast = "<?xml version=\"1.0\" encoding=\"utf-8\"?>" +
        "<wp:Notification xmlns:wp=\"WPNotification\">" +
           "<wp:Toast>" +
                "<wp:Text1>Hello from a .NET App!</wp:Text1>" +
           "</wp:Toast> " +
        "</wp:Notification>";
    await hub.SendMpnsNativeNotificationAsync(toast);
}

Make sure to insert the name of your hub and the connection string called DefaultFullSharedAccessSignature that you obtained in the section “Configure your Notification Hub.” Note that this is the connection string with Full access, not Listen access.

Then add the following line in your Main method:

SendNotificationAsync();
 Console.ReadLine();

Easy enough! So, using this as my guide, I created my own app that followed these principles, the main difference being that instead of creating a string with the XML payload in it, I thought I’d use an XmlWriter instead, because that’s neater:

        private static string prepareToastPayload(string text1, string text2)
        {
            MemoryStream stream = new MemoryStream();
            XmlWriterSettings settings = new XmlWriterSettings() 
                                            { 
                                                Indent = false,
                                                Encoding = Encoding.UTF8   
                                            };
            XmlWriter writer = XmlWriter.Create(stream, settings);
            writer.WriteStartDocument();
            writer.WriteStartElement("wp", "Notification", "WPNotification");
            writer.WriteStartElement("wp", "Toast", "WPNotification");
            writer.WriteStartElement("wp", "Text1", "WPNotification");
            writer.WriteValue(text1);
            writer.WriteEndElement();
            writer.WriteStartElement("wp", "Text2", "WPNotification");
            writer.WriteValue(text2);
            writer.WriteEndElement();
            writer.WriteEndElement();
            writer.WriteEndDocument();
            writer.Close();

            return Encoding.UTF8.GetString(stream.ToArray());
        }

Wierd exception

When I ran this, I got the following exception:

System.ArgumentException: The payload is not in accepted XML format. The first node should be Tile/Toast. If want to send raw notification, please set \”X-NotificationClass\” to \”3\” in header.\r\n   at Microsoft.ServiceBus.Notifications.RegistrationSDKHelper.DetectMpnsTemplateRegistationType(String body, String errorMsg)\r\n   at Microsoft.ServiceBus.Notifications.MpnsNotification.OnValidateAndPopulateHeaders()\r\n   at Microsoft.ServiceBus.Notifications.Notification.ValidateAndPopulateHeaders()\r\n   at Microsoft.ServiceBus.Notifications.NotificationHubManager.BeginSendNotification(Notification notification, Boolean testSend, AsyncCallback callback, Object state)\r\n   at Microsoft.ServiceBus.Notifications.NotificationHubManager.<>c__DisplayClass1.<SendNotificationAsync>b__0(AsyncCallback c, Object s)\r\n   at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl(Func`3 beginMethod, Func`2 endFunction, Action`1 endAction, Object state, TaskCreationOptions creationOptions)\r\n   at System.Threading.Tasks.TaskF
actory`1.FromAsync(Func`3 beginMethod, Func`2 endMethod, Object state)\r\n   at Microsoft.ServiceBus.Common.Parallel.TaskHelpers.CreateTask[T](Func`3 begin, Func`2 end)… in c:\\Development\\AzurePushNotificationsDemo\\NotificationSenderUtility\\NotificationSenderUtility.cs:line 139

Now as it turns out, the error message here is rather misleading. “The first node should be Tile/Toast” seems to be referring to the expected XML format if you are sending toast messages using the (very useful) template capability of Notification Hubs, which allows you to send a generic message to Notification Hubs just once, and it will build the appropriate payloads for your actual clients, whether they are Windows Phone, Windows 8, Android or iOS, all of which have different payloads. But “The payload is not in accepted XML format” is a bit more interesting and forced me to take a look at the XML I was generating.

And here it gets really confusing. I swapped out the XmlWriter code and replaced it with just simply setting a string variable directly, similar to the original tutorial – that works! So what is different about the string generated by my prepareToastpayload method? I then put the two methods one after the other and stored the results in two different string variables which I then examined when stopped at a breakpoint – identical!! So, what gives?

Examining the string bytes

Of course, the strings WEREN’T identical, despite what the Visual Studio debugger was showing me! A bit of a web search led me to this useful post by Jon Skeet, author of the book “C# In Depth”. His post Strings in C# and .NET includes a warning about the Visual Studio debugger not showing you the whole story, and he includes some useful code for dumping the bytes of a string to output. I plugged in the suggested methods and the output revealed the following at the beginning of the XML string I was sending to SendMpnsNativeNotifiactionAsync():

image

Aha! What are these (Possibly non-printable) xFEFF bytes at the start? Next web search was for FEFF and I came upon a few posts explaining that hex FEFF is something called a BOM or Byte Order Mark, which is something that is usually inserted into UTF-8 encoded files that are stored on disk, which XML parsers can look for to figure out how to interpret the file contents. However, if you are formatting XML to send as payload in a web service call as we are here, then you don’t want it. Best post that explain this is a six year old blog by Rick Strahl, XmlWriter, Strings and Byte Order Marks .

Un-BOMbin’ the L

Rick gives the solution as well. It’s actually very difficult to stop the XmlWriter for outputting XML without a BOM, but you can get it to behave by creating a custom UTF-8 encoding where you can specify you don’t want a BOM by passing false into an override of the constructor, and then use that in your XmlWriterSettings object:

        private static string prepareToastPayload(string text1, string text2)
        {
            // Create encoding manually in order to prevent
            // creation of leading BOM (Byte Order Mark) xFEFF at start
            // of string created from the XML
            Encoding Utf8 = new UTF8Encoding(false); // Prevents creation of BOM
            MemoryStream stream = new MemoryStream();
            XmlWriterSettings settings = new XmlWriterSettings() 
                                            { 
                                                Indent = false,
                                           //   Encoding = Encoding.UTF8    !!NO-> adds Unicode BOM to start
                                                Encoding = Utf8,    // Use manually created UTF8 encoding
                                            };
            XmlWriter writer = XmlWriter.Create(stream, settings);
            writer.WriteStartDocument();
            writer.WriteStartElement("wp", "Notification", "WPNotification");
            writer.WriteStartElement("wp", "Toast", "WPNotification");
            writer.WriteStartElement("wp", "Text1", "WPNotification");
            writer.WriteValue(text1);
            writer.WriteEndElement();
            writer.WriteStartElement("wp", "Text2", "WPNotification");
            writer.WriteValue(text2);
            writer.WriteEndElement();
            writer.WriteEndElement();
            writer.WriteEndDocument();
            writer.Close();

            var payload =  Encoding.UTF8.GetString(stream.ToArray());
            DisplayString(payload); // Inspect bytes
            return payload;
        }

Conclusion

So there you have it. Once you know what the problem is – an unwanted BOM – it’s quite easy to find posts that discuss it and give you a solution. But many developers who are having difficulty with XML they have created in code will not know what the problem is they are trying to fix. No amount of studying the code, from inspecting strings in Visual Studio, or quite likely the exception message will tell you that maybe there is a hidden and unwanted character at the start of the generated XML. For you, hopefully this post will help you find a solution.

SQLite-WinRT: Support for NULL columns added

Small update to the SQLite WinRT wrapper we published at http://sqlwinrt.codeplex.com a few months ago. This update adds support for storing NULL values into a row through the new methods BindNullParameterAt(int index) and BindNullParameterWithName(string name) added to the Statement class.

The following code sample shows how to store a null value into a column in a row, and then how to check if a null value has been returned to you in the data returned from a SELECT:

var db = new SQLiteWinRTPhone.Database(
    ApplicationData.Current.LocalFolder, "sqlite.db");

await db.OpenAsync();

string sql = @"
    CREATE TABLE IF NOT EXISTS
        Customer(Id     INTEGER PRIMARY KEY NOT NULL,
                    Name   VARCHAR(140),
                    NullableInt INTEGER);";

await db.ExecuteStatementAsync(sql);

try
{
    sql = @"INSERT INTO Customer(Id, Name, NullableInt) 
            VALUES (@id, @name, @nullInt);";
    using (var custstmt = await db.PrepareStatementAsync(sql))
    {
        custstmt.BindIntParameterWithName("@id", 11);
        custstmt.BindNullParameterWithName("@name"); 
        custstmt.BindNullParameterAt(3);
        await custstmt.StepAsync();
    }
}
catch (Exception ex)
{
    var result = Database.GetSqliteErrorCode(ex.HResult);
    Debug.WriteLine("INSERT failed with error " + result);
} 

// Right now read the rows out again
try
{
    sql = "SELECT Id, Name, NullableInt FROM Customer;";
    using (var custstmt = await db.PrepareStatementAsync(sql))
    {
        // Use StepAsync to execute a prepared statement
        while (await custstmt.StepAsync())
        {
            var Id = custstmt.GetIntAt(0);
            var Name = "NULL";
            if (custstmt.GetColumnType(1) != ColumnType.Null)
            {
                Name = custstmt.GetTextAt(1);
            }
            int? NullableInt = null;
            if (custstmt.GetColumnType(2) != ColumnType.Null)
            {
                NullableInt = custstmt.GetIntAt(2);
            }
            Debug.WriteLine("Got Customer: " + Id + " " 
                + Name + " NullableInt: " 
                + (NullableInt.HasValue ? NullableInt.Value.ToString() : "null"));
        }
    }
}
catch (Exception ex)
{
    var result = Database.GetSqliteErrorCode(ex.HResult);
    MessageBox.Show("SELECT failed with error " + result);
} 

Notice that you use BindNullParameterxxx to store a null value – using BindTextParameterWithName(string name, string value) and passing a NULL for the value doesn’t work and causes an exception to be thrown.

Notice also that to check if a column returned to you contains NULL, you use the Statement.GetColumnType(int index) method to find out if the column type is ColumnType.Null.

So, please go and download the source code again from https://sqlwinrt.codeplex.com/SourceControl/latest – sorry, no NuGet package for this. Please give it a try and give us any feedback. For more on how to use the wrapper, see my earlier post SQLite-WinRT: Database programming on Windows Phone and Windows 8.

SQLite Using NULL Demo code: SQLiteNULLdemo.zip

Now easier than ever to developer unlock a handset to get started building apps for Windows Phone!

Great news reported in the latest Windows Phone Developer Blog post, Making it easier to get started with Windows Phone App Studio beta, simplified phone registration, support options & more payout markets. There you can read about a great new tool, Windows Phone App Studio, that can help people without coding skills to create a phone app using a free, web-based app creation tool. Even experienced devs will find it useful to build out the basic structure of an app, but since you can download the code for the project the tool creates, you can then modify it to add more functionality.

Interesting though Windows Phone App Studio is, the best news announced in that post is the new simplified phone registration. This is fantastic news, as it removes one blocker we’ve had with people trying to get started building for Windows Phone.

The dev tools have always been free (download the Windows Phone 8.0 SDK for free from http://dev.windowsphone.com), but if you wanted to test your app on the emulator, you had to have a PC with a Core i3/i5/i7 processor and have Windows 8 Pro or Enterprise, which are the editions that include the necessary Hyper-V support. So if you couldn’t meet those hardware and OS specs, your only option was to run your app on a real phone – but to use a real phone you had to have registered for a developer account at http://dev.windowsphone.com so you could unlock it, which is normally $99/year (although that is currently on special offer at $19 until August 26th). At this point, the ‘mildly curious developer’, faced with the prospect of having to part with money to try out their new app, can lose interest and of course we’ve lost a potential convert to our developer community.

Any dev can unlock one phone for testing without a registration fee

Now, anyone can unlock one phone to test their apps without needing a dev account, and can install up to two ‘work in progress’ apps onto the phone at any one time. Fantastic! So now you can get started with Windows Phone dev using *any* PC running Windows 8 (any edition) at no charge (well, you need a handset obviously ). Only caveat is that your PC and Windows 8 OS still needs to be 64 bit, as that is a requirement of the SDK.

Beware the perils of async/await in application lifecycle event handlers (in fact in *any* event handlers)

Many developers are, quite rightly, adopting the use of async and await when writing Windows Phone and Windows 8 Store apps. It’s the right thing to do, as it allows long-running operations such as network calls and file i/o to complete efficiently without any risk of inadvertently locking up the UI thread and creating a poor user experience by freezing the screen responsiveness. In fact, in Windows 8 Store apps, it’s the *only* way to do these operations because in the .NET API for Windows Store apps, the old synchronous methods have been removed and only the new async versions are available. On Windows Phone 8, the developer has the choice of using the old Window Phone 7-style APIs, or in many cases newer async alternatives, such as the WinRT Windows.Storage APIs for file operations.

However, we’re seeing a lot of postings in the forums saying that their app only works intermittently, or works fine in debug but then fails intermittently when deployed. Quite often, the reason for this is that the developer has inadvertently called some async methods in a ‘fire and forget’ fashion, and what they are seeing is a synchronization problem; they assume that some async code has completed, and some code elsewhere is trying to use the result value from the async method before the async method has completed. The result, a null reference exception or incorrect behaviour because some object is not fully populated.

The problem with async/await in Application_Launching and Application_Activated

I accidently encountered this problem when I was building a simple demo for the last Windows Phone 8 JumpStart video series. OK, it was a dumb demo, but demos are often dumb because you want to keep them simple so as not to distract from the main point you are trying to make. This demo was to show how to program with the new Windows.Storage apis, so in my App class Application_Launching event handler I had code that read some data out of a file and saved the string into a public property called LogData:

    public partial class App : Application
    {
        ...

        // This is shared amongst all the pages
        // It is the contents of the log itself, as a large string
        public string LogData;

        // Code to execute when the application is launching (eg, from Start)
        // This code will not execute when the application is reactivated
        private async void Application_Launching(object sender, LaunchingEventArgs e)
        {            
            // Warning - this async method may not complete before this event handler method completes and other
            // events fire. Be careful not to assume completion of the logic in this method in other event handler code.
            this.LogData = await LoadFromLocalFolderAsync();
        }

        public async static Task<string> LoadFromLocalFolderAsync()
        {
            string theData = string.Empty;

            // There's no FileExists method in WinRT, so have to try to get a reference to it
            // and catch the exception instead
            StorageFile storageFile = null;
            bool fileExists = false;
            try
            {
                // See if file exists
                storageFile = await Windows.Storage.StorageFile.GetFileFromApplicationUriAsync(
                    new Uri("ms-appdata:///local/CaptainsLog.store"));
                fileExists = true;
            }
            catch (FileNotFoundException)
            {
                // File doesn't exist
                fileExists = false;
            }

            if (!fileExists)
            {
                // Initialise the return data
                theData = string.Empty;
            }
            else
            {
                // File does exist, so open it and read the contents
                Stream readStream = await storageFile.OpenStreamForReadAsync();
                using (StreamReader reader = new StreamReader(readStream))
                {
                    theData = await reader.ReadToEndAsync();
                }
            }

            return theData;
        }

And in MainPage.xaml.cs, OnNavigatedTo, the code gets the value of LogData from the App class and sets the Text property of a TextBlock to the string to show the contents of the file on the screen. Logic elsewhere in the app writes content into the file, so on startup you should always see the current contents of the file shown on the screen on application launch.

        protected override void  OnNavigatedTo(System.Windows.Navigation.NavigationEventArgs e)
        {
             base.OnNavigatedTo(e);
            
            // When we navigate to the page - put the log text into the display
            completeLogTextBlock.Text = ((App)App.Current).LogData;
        }

What could possibly go wrong? Well, plenty as it turns out! The first time you run this app, the screen is empty which is to be expected because the file is empty. But then you use the app to enter some text which is written to the file, then close the app. Launch again, and where you would expect to see some text displayed, the screen is still empty! What gives?

Exploring execution of Async methods

To show why this went wrong, let’s look at a simple console app to explore the two ways you can execute an async method.

using System;
using System.Threading.Tasks;

namespace AsyncSynchronizationProblem
{
    class Program
    {
        static void Main(string[] args)
        {
            new TheClassThatDoesWork().DoTheWork();

            Console.ReadKey();
        }
    }

    class TheClassThatDoesWork
    {
        public async void DoTheWork()
        {
            Console.WriteLine("TRACE Calling SomeLongProcedure Async");
            await SomeLongProcedureAsync();
            Console.WriteLine("TRACE Returned from SomeLongProcedure Async");

            Console.WriteLine();

            Console.WriteLine("TRACE Calling SomeLongProcedure Fire and forget");
            SomeLongProcedureAsync();
            Console.WriteLine("TRACE Returned from SomeLongProcedure Fire and forget");
        }

        private Task SomeLongProcedureAsync()
        {
            var task = Task.Run(() =>
                {
                    Console.WriteLine("TRACE Starting SomeLongProcedure work");
                    System.Threading.Thread.Sleep(1000); // Simulate a long running operation
                    Console.WriteLine("TRACE Completed SomeLongProcedure work");
                });

            return task;
        }
    }
}

When you run this app, you see the following console output:

asyncdemo1

In this sample, the long running method, Task SomeLongProcedureAsync() simply sleeps for 1s to simulate executing some lengthy code. In the method DoTheWork(), we just call the async method twice, once using await and once without. We write to the console output to show what is happening. I’m sure it is no surprise to most of you, that when we call using await, the caller waits until the long-running work has completed before continuing execution itself:

TRACE Calling SomeLongProcedure Async
TRACE Starting SomeLongProcedure work
TRACE Completed SomeLongProcedure work
TRACE Returned from SomeLongProcedure Async

and when we call without using await, it works in a fire-and-forget fashion, and the caller continues execution without waiting for the task to complete:

TRACE Calling SomeLongProcedure Fire and forget
TRACE Starting SomeLongProcedure work
TRACE Returned from SomeLongProcedure Fire and forget

TRACE Completed SomeLongProcedure work

This is pretty basic stuff about calling async methods. So why is this relevant?

Dangers of making event handlers async

Let’s look at another example. This one is still a console app, but has four classes, Program which just hosts the Main method of the console app, App, PhoneApplicationService and MainPage – I have named the last three the same as key classes in a Windows Phone app for reasons which hopefully will become obvious.

using System;
using System.Threading.Tasks;

namespace AsyncSynchronizationProblemWithEvents
{
    class Program
    {
        static void Main(string[] args)
        {
            var phoneAppService = new PhoneApplicationService();
            var app = new App(phoneAppService);
            
            phoneAppService.DoTheWork();

            // When the PhoneApplicationService has completed, construct a MainPage instance
            var mainPage = new MainPage();
            // Call methods on it
            mainPage.OnNavigatedTo();

            Console.ReadKey();
        }
    }

    class App
    {
        private PhoneApplicationService appService;

        public App(PhoneApplicationService applicationService)
        {
            appService = applicationService;
            // Subscribe to the PhoneApplicationService lifetime event(s)
            appService.ApplicationLaunching += Application_Launching;
        }

        async void Application_Launching(object sender, EventArgs e)
        {
            Console.WriteLine("TRACE Event handler - Calling SomeLongProcedure Async");
            await SomeLongProcedureAsync();
            Console.WriteLine("TRACE Event handler - Returned from SomeLongProcedure Async");
        }

        private Task SomeLongProcedureAsync()
        {
            var task = Task.Run(() =>
            {
                Console.WriteLine("TRACE Starting SomeLongProcedure work");
                System.Threading.Thread.Sleep(1000); // Simulate a long running operation
                Console.WriteLine("TRACE Completed SomeLongProcedure work");
            });

            return task;
        }
    }

    class PhoneApplicationService
    {
        public event EventHandler ApplicationLaunching;

        public void DoTheWork()
        {
            // Here we do some stuff...

            // Call any subscribers to our event(s)
            if (ApplicationLaunching != null)
            {
                ApplicationLaunching(this, new EventArgs());
            }
        }
    }

    class MainPage
    {
        public void OnNavigatedTo()
        {
            Console.WriteLine("TRACE Starting MainPage OnNavigatedTo");

            // Do some work here too...

            Console.WriteLine("TRACE Completed MainPage OnNavigatedTo");
        }
    }
}

If you study this app, you will see that the Main method creates instances of PhoneApplicationService, creates an instance of App and passes the PhoneApplicationService instance into its constructor, and finally creates an instance of MainPage and calls its OnNavigatedTo method. This is, of course, exactly what happens when you start a Windows Phone app. When you run it, the output looks like this:

asyncdemo2

There is clearly something odd going on here: the trace messages from the execution of SomeLongProcedure are output, but right in the middle, the calls to MainPage OnNavigatedTo appear. What’s going on?

Inside the App constructor, we hook the ApplicationLaunching event of the PhoneApplicationService instance so that in the event handler we can do some setup work when the app launches, and this of course is where we call the long-running async method:

        async void Application_Launching(object sender, EventArgs e)
        {
            Console.WriteLine("TRACE Event handler - Calling SomeLongProcedure Async");
            await SomeLongProcedureAsync();
            Console.WriteLine("TRACE Event handler - Returned from SomeLongProcedure Async");
        }

We are using the await keyword inside this event handler, so of course we mark it with the async modifier as normal. So at first glance, the developer expects this code to execute and for the event handler code to wait for completion of SomeLongProcedureAsync() before continuing execution, which in this case is simply to write a message to the console. If we look at the output, that is indeed what happens:

TRACE Event handler – Calling SomeLongProcedure Async
TRACE Starting SomeLongProcedure work

TRACE Starting MainPage OnNavigatedTo
TRACE Completed MainPage OnNavigatedTo

TRACE Completed SomeLongProcedure work
TRACE Event handler – Returned from SomeLongProcedure Async

But as you can see from the trace output, the MainPage OnNavigatedTo method is executed before the Application_Launching event handler has completed, even though it was clearly invoked by the Main function after the PhoneApplicationService.DoWork() method was invoked, which is the method that fires the Application_Launching event:

        static void Main(string[] args)
        {
            var phoneAppService = new PhoneApplicationService();
            var app = new App(phoneAppService);
            
            phoneAppService.DoTheWork();

            // When the PhoneApplicationService has completed, construct a MainPage instance
            var mainPage = new MainPage();
            // Call methods on it
            mainPage.OnNavigatedTo();

            Console.ReadKey();
        }

The problem of course, is that there is no way of awaiting the completion of an event handler, so when the DoWork() method fires the Application_Launching event, it does so in a fire and forget fashion, just the same as in the calling of SomeLongProcedureAsync() without using the await keyword, as we demonstrated in the first example. During execution, code executes sequentially until we hit the await keyword inside the Application_Launching event handler, at which point the wonders of the Task Parallel Library ensure that execution of the Task continues without blocking the original context and control returns immediately to the caller (PhoneApplicationService.DoTheWork() in this case). 

If the Application_Launching event handler had returned async Task instead of async void: 

async Task Application_Launching(object sender, EventArgs e)

(which is not possible with event handlers) and if it had been called using the await keyword, then  DoWork() would have awaited completion of the Application_Launching event handler logic before continuing, in which case, MainPage.OnNavigatedTo() would not have been called before the Application_Launching logic had completed. You can only await completion of methods that return Task or Task<T> and since this is impossible with event handlers, you can never await their completion. Hence there is every chance that your MainPage OnNavigatedTo logic will execute before your Application_Launching or Application_Activated logic has completed, although it is actually hit or miss, which is why so many developers report intermittent failures.

The Solution?

Of course, this is not a new problem. Any time you execute code asynchronously from your application lifecycle event handlers, whether using async/await or other asynchronous programming techniques, there is a real chance that you will have to implement some kind of synchronisation logic to ensure that your page logic does not try to access objects before your startup logic has set them up. Using older async coding technologies, this was perhaps a little more obvious to the developer, but many devs use the new Task-based async methods without thinking too deeply about it. It just works – or at least seems to until you hit a timing-related problem such as the one I’ve described here.

In the simple demo I described at the beginning of this article, the solution was simple. Instead of exposing the data fro the app in a public property of the App class, I created a new ViewModel class to host it and used XAML databinding to bind the TextBlock on MainPage to a property of the ViewModel. When the Application_Launching logic completed, it updated the ViewModel instance, and through the wonders of INotifyPropertyChanged, the MainPage UI updated with the correct data.

For other solutions, you may need to implement some other mechanism. One example was shown me by Peter Torr of the Windows Phone Product Group and is shown in the sample app which you can download from my post on SQLite:  SQLite-WinRT: Database programming on Windows Phone and Windows 8 . That sample opens (and the first time, creates) the SQLite database in Application_Launching and when resuming from tombstoning in Application_Activated. That sample used an old-fashioned synchronisation primitive, a ManualResetEvent, as a flag that is only set when the application lifecycle logic has finished:

  public partial class App : Application
  {
    static SQLiteWinRTPhone.Database db;
    static ManualResetEvent DBLoaded = new ManualResetEvent(false);

    public static Task<SQLiteWinRTPhone.Database> GetDatabaseAsync()
    {
      return Task.Run(() =>
      {
        DBLoaded.WaitOne();
        return db;
      });
    }

    // Code to execute when the application is launching (eg, from Start)
    // This code will not execute when the application is reactivated
    private void Application_Launching(object sender, LaunchingEventArgs e)
    {
      LoadDatabase();
    }

    private async Task LoadDatabase()
    {
      // Get a reference to the SQLite database
      db = new SQLiteWinRTPhone.Database(ApplicationData.Current.LocalFolder, "sqlitedemo.db");

      await db.OpenAsync();

      string sql = @"CREATE TABLE IF NOT EXISTS
                                Customer (Id      INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
                                          Name    VARCHAR( 140 ),
                                          City    VARCHAR( 140 ),
                                          Contact VARCHAR( 140 ) 
                            );";
      string description = "Create Customer table";
      await ExecuteSQLStatement(db, sql, description);

      sql = @"CREATE TABLE IF NOT EXISTS
                                Project (Id          INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
                                         CustomerId  INTEGER,
                                         Name        VARCHAR( 140 ),
                                         Description VARCHAR( 140 ),
                                         DueDate     DATETIME,
                                         FOREIGN KEY(CustomerId) REFERENCES Customer(Id) ON DELETE CASCADE 
                            )";
      description = "Create Project table";
      await ExecuteSQLStatement(db, sql, description);

      // Turn on Foreign Key constraints
      sql = @"PRAGMA foreign_keys = ON";
      description = "Enable foreign key constraints";
      await ExecuteSQLStatement(db, sql, description);

      DBLoaded.Set();
    }

    private static async Task ExecuteSQLStatement(SQLiteWinRTPhone.Database db, string sql, string description)
    {
      try
      {
        await db.ExecuteStatementAsync(sql);
        Debug.WriteLine(description + " executed OK");
      }
      catch (Exception ex)
      {
        var result = SQLiteWinRTPhone.Database.GetSqliteErrorCode(ex.HResult);
        throw new ApplicationException(description + " Failed with error " + result);
      }
    }

The ManualResetEvent is declared unsignalled as a static field of the app class:

 static ManualResetEvent DBLoaded = new ManualResetEvent(false);

and is only set at the very end of the async Task LoadDatabase() method:

private async Task LoadDatabase()
{
  // Get a reference to the SQLite database
  db = new SQLiteWinRTPhone.Database(…);
 
await db.OpenAsync();
  …

  DBLoaded.Set();
}

The final key ingredient is a ‘gatekeeper’ method which must be called by every other method in the app that wants to access the database:

public static Task<SQLiteWinRTPhone.Database> GetDatabaseAsync()
{
  return Task.Run(() =>
  {
    DBLoaded.WaitOne();
    return db;
  });
}

This method waits for the DBLoaded semaphore to be set and does so on a Threadpool thread (through Task.Run()) so does not block the UI thread.

Calling async code from Application_Deactivated or Application_Closing

The guidance here is “don’t’”. If you write your apps carefully, you can be saving changes to persistent data as you go along, so you shouldn’t have anything to do in the application lifecycle events.

If you must, you can try doing something like this:

SomeAsyncMethod().AsTask().Wait()   

If the operation completes within the timeout period AND it doesn’t deadlock due to needing to pump the UI thread, it will work… but don’t count on it.