dev comments edit

HTTP/2 is a major update to the HTTP 1.x protocol and I decided to spare some time to have a general idea what it is all about:

Here are my findings:

  • It’s based on SPDY (a protocol developed by Google, currently deprecated)
  • It uses same methods, status codes etc. so it is backwards-compatible and the main focus is on performance
  • The problem it is addressing is HTTP requiring a TCP connection per request.
  • Key differences:
    • It is binary rather than text.
    • It can use one connection for multiple requests
    • Allows servers push responses to browser caches. This way it can automatically start sending assets before the browser parses the HTML and sends a request for each of them (images, JavaScript, CSS etc)
  • The protocol doesn’t have built-in encryption but currently Firefox, Internet Explorer, Safari, and Chrome agree that HTTPS is required.
  • There will be a negotiation process between the client and server to select which version to use
  • WireShark has support for it but Fiddler doesn’t.
  • As the speed is the main focus it’s especially important for CDNs to support it. In September 2016, AWS announced that they now support HTTP/2. For existing distributions it needs to enabled explicitly by updating the settings.

    AWS CloudFront HTTP/2 Support

  • On the client side looks like it’s been widely adopted and supported. CanIUse.com also confirms that it’s only allowed over HTTPS on all browsers that support it.

    HTTP/2 Browser Support

What Does It Look Like on the Wire

As it’s binary I was curious, as a developer, to see what the actual bits looked like. Normally it’s easy to inspect HTTP requests/responses because it’s just text.

Apparently the easiest way to do it is WireShark. First, I had to enable session logging by creating a user variable in Windows:

Windows environment variable to capture TLS session keys

and pointing the WireShark to use that log (Edit -> Preferences -> Protocols -> SSL)

This is a very neat trick and it can be used to analyse all encrypted traffic so it serves a broader purpose. After restarting the browser and WireShark I was able to see the captured session keys and by starting a new capture with WireShark I could see the decrypted HTTP/2 traffic.

WireShark HTTP/2 capture

It’s hard to make sense of everything in the packets but I guess it’s a good start to be able to inspect the wire format of the new protocol.

Resources

aws, route53, angularjs, development comments edit

Previously in this series:

So far I was using the console client but I thought I chould use a prettier web-based-UI and came up with this:

DomChk53 AngularJS Client

It’s using AngularJS and Bootstrap which significantly improved the development process.

API in the backend is AWS API Gateway on a custom domain (api.domchk53.com) and using Lambda functions to do the actual work. One great thing about API Gatewayis that it’s very easy to set requests rates:

Currently I set it to max 5 requests per second. I chose this value because of the limitation on AWS API as stated here:

All requests – Five requests per second per AWS account. If you submit more than five requests per second, Amazon Route 53 returns an HTTP 400 error (Bad request). The response header also includes a Code element with a value of Throttling and a Message element with a value of Rate exceeded.

Of course limiting this on the client side assumes a single client so you may still get “Rate exceeded” errors even if running single query at a time. I’m planning to implement a Node server using SQS to move the queue to server side but that’s not one of my priorities right now.

The Lambda function is straightforward enough. Just calls the checkDomainAvailability API method with the supplied parameters:

exports.handler = function(event, context) {
    var AWS = require('aws-sdk');
	var options = {
	    region: "us-east-1"
	}
	
	var route53domains = new AWS.Route53Domains(options);
    
    var params = {
        DomainName: event.domain + '.' + event.tld
    };

    route53domains.checkDomainAvailability(params, function (err, data) {

        if (err) {
            context.fail(err);
        } else {
            var result = {
                Domain: event.domain,
                Tld: event.tld,
                CheckDate: new Date().toISOString(),
                RequestResult: "OK",
                Availability: data.Availability
            };
            
            context.succeed(result);  
        }
    });
}

Usage

I wanted this tool as an improvement to AWS already provides. What you can do with Management Console is search a single domain and it searches it against the popular 13 TLDs. If you need anything outside these 13 you have to pick them manually.

In DomChk53 you can search multiple domain names at once against all supported TLDs (293 as of this writing).

Also you can group TLDs into lists so you can for example search the most common ones (com, net, co.uk etc.) and say finance related ones (money, cash, finance etc.). Depending on the domain name one group may be more relevant.

You can cancel a query at any time to avoid wasting precious requests if you change your mind about the domain.

What’s missing

For a while I’m planning to leave it as is but when I have it in me to revisit the project I will implement:

  • Server-side queueing of requests
  • The option to export/email the results in PDF format

I’m also open to other suggestions…

Resources

aws, route53, angularjs, development comments edit

I’ve been using my own dynamic DNS application (which I named DynDns53 and blogged about it here). So far it had a WPF application and I was happy with it but I thought if I could develop a web-based application I wouldn’t have to install anything (which is what I’m shooting for these days) and achieve the same results.

So I built a JavaScript client with AngularJS framework. The idea is exactly the same, the only difference is it’s all happening inside the browser.

DynDns53 web client

Ingredients

To have a dynamic DNS client you need to have the following

  1. A way to get your external IP address
  2. A way to update your DNS record
  3. An application that performs Step 1 & 2 perpetually

Step 1: Getting the IP Address

I have done and blogged about this several times now. (Feels like I’m repeating myself a bit, I guess I have to find something original to work with. But first I have to finish this project and have closure!)

Since it’s a simple GET request it sounds easy but I quickly hit the CORS wall when I tried the following bit:

app.factory('ExternalIP', function ($http) {
    return $http.get('http://checkip.amazonaws.com', { cache: false });
}); 

In my WPF client I can call whatever service I want whenever I want but when running inside the browser things are a bit different. So I decided to take a detour and create my own service that allowed cross-origin resource sharing.

AWS Lambda & API Gateway

First I thought I could do it even without Lambda function by using the HTTP proxy integration. I could return what the external site returns:

Unfortunately this didn’t work because it was returning the IP of the AWS machine that’s actually running the API gateway. So I had to get the client’s IP from the request and send it back in my own Lambda function.

Turns out in order to get HTTP headers you need to fiddle with some template mapping and assign the client’s IP address to a variable:

This can be later referred to in the Lambda function through event parameter:

exports.handler = function(event, context) {
    context.succeed({
        "ip": event.ip
    })
}

And now that we have our own service we can allow CORS and be able call it from our client inside the browser:

Step 2: Updating DNS

This bit is very similar to WPF version. Instead of using the AWS .NET SDK I just used the JavaScript SDK. AWS has a great SDK builder which lets you to select the pieces you need:

It also shows if the service supports CORS. It’s a relief that Route53 does so we can keep going.

The whole source code is on GitHub but here’s the gist of it: Loop through all the subdomains, get all the resource records in the zone, find the matching record and update it with the new IP:

  $scope.updateAllDomains = function() {
      angular.forEach($rootScope.domainList.domains, function(value, key) {
        $scope.updateDomainInfo(value.name, value.zoneId);
      });
  } 
  $scope.updateDomainInfo = function(domainName, zoneId) {
    var options = {
      'accessKeyId': $rootScope.accessKey,
      'secretAccessKey': $rootScope.secretKey
    };
    var route53 = new AWS.Route53(options);
    
    var params = {
      HostedZoneId: zoneId
    };

    route53.listResourceRecordSets(params, function(err, data) {
        if (err) { 
          $rootScope.$emit('rootScope:log', err.message);
          console.log(err.message);
        } else {
          angular.forEach(data.ResourceRecordSets, function(value, key) {
              if (value.Name.slice(0, -1) == domainName) {
                var externalIPAddress = "";
                ExternalIP.then(function(response){
                     externalIPAddress = response.data.ip;
                     $scope.changeIP(domainName, zoneId, externalIPAddress)
                 });
              }
          });
        }
    });
  }
  $scope.changeIP = function(domainName, zoneId, newIPAddress) {
    var options = {
      'accessKeyId': $rootScope.accessKey,
      'secretAccessKey': $rootScope.secretKey
    };

    var route53 = new AWS.Route53(options);
    var params = {
      ChangeBatch: {
        Changes: [
          {
            Action: 'UPSERT',
            ResourceRecordSet: {
              Name: domainName,
              Type: 'A',
              TTL: 300,
              ResourceRecords: [ {
                  Value: newIPAddress
                }
              ]
            }
          }
        ]
      },
      HostedZoneId: zoneId
    };

    route53.changeResourceRecordSets(params, function(err, data) {
      if (err) { 
        $rootScope.$emit('rootScope:log', err.message); 
      }
      else { 
        var logMessage = "Updated domain: " + domainName + " ZoneID: " + zoneId + " with IP Address: " + externalIPAddress;
        $rootScope.$emit('rootScope:log', logMessage);
      }
    });
  }

The only part that trippped me up was that I wasn’t setting the TTL in the changeResourceRecordSets parameters and I was getting an error but found a StackOverflow question that helped to get past the issue.

Step 3: A tool to bind them

Now the fun part: An AngularJS client to call these services. I guess the UI is straight-forward. Basically it just requires the user to enter AWS IAM keys and domains to update.

I didn’t want to deal with the hassle of sending the keys to a remote server and host them securely. Instead I thought it would be simpler just to use browser’s local storage with HTML5. This way the keys never leave the browser.

It also only updates the IP address if it has changed so saves unnecessary API calls.

Also it’s possible to view what’s going on in the event log area.

I guess I can have my closure now and move on!

Resources

c#, development, gadget, fitbit, aria comments edit
"What gets measured, gets managed." - Peter Drucker

It’s important to have goals, especially SMART goals. The “M” in S.M.A.R.T. stands for Measurable. Having enough data about a process helps tremendously to improve that process. To this effect, I started to collect exercise data from my Microsoft band which I blogged about here.

Weight tracking is also crucial for me. I used to record my weight manually on a piece of paper but, for the obvious reasons, I abandoned it quickly and decided to give Fitbit Aria a shot.

Fitbit Aria Wi-Fi Smart Scale

Aria is basically a scale that can connect to your Wi-Fi network and send your weight results to Fitbit automatically which can then be viewed via Fitbit web application.

Setup

Since it doesn’t have a keyboard or any other way to interact directly setup is carried out by running a program on your computer

It’s mostly just following the steps on the setup tool. You basically let it connect to your Wi-Fi network so that it can synchronize with Fitbit servers.

Putting the scale into setup mode proved to be tricky in the past though. Also it was not easy to change Wi-Fi so I had to reset the go back to factory settings and ran the setup tool again.

Getting the data via API

Here comes the fun part! Similar to my MS Band workout demo, I developed a WPF program to get my data from Fitbit’s API. Ultimately the goal is to combine all these data in one application and make sense of it.

Like MS Health API, FitBit uses OAuth 2.0 authorization and requires a registered application.

The endpoint that returns weight data accepts a few various formats depending on your needs. As I wanted a range instead of a single day I used the following format:

https://api.fitbit.com/1/user/{user ID}/body/log/weight/date/{startDate}/{endDate}.json

This call returns an array of the following JSON objects:

{
	"bmi": xx.xx,
	"date": "yyyy-mm-dd",
	"fat": xx.xxxxxxxxxxxxxxx,
	"logId": xxxxxxxxxxxxx,
	"source": "Aria",
	"time": "hh:mm:ss",
	"weight": xx.xx
}

Sample application

The bulk of the application is very similar to MS Band sample: It first opens an authorization window and once the client consents for the app to be granted some privileges it uses the access token to retrieve the actual data.

There are a few minor differences though:

  • Unlike MS Health API it requires Authorization header in the authorization code request calls which is basically Base64 encoded client ID and client secret
string base64String = Convert.ToBase64String(Encoding.UTF8.GetBytes($"{Settings.Default.ClientID}:{Settings.Default.ClientSecret}"));
request.AddHeader("Authorization", $"Basic {base64String}");
  • It requires a POST request to redeem URL. Apparently RestSharp has a weird behaviour. You’d think a method called AddBody could be used to send the request body, right? Not quite! It doesn’t transmit the header so I kept getting a missing field error. So instead I used AddParameter:
string requestBody = $"client_id={Settings.Default.ClientID}&grant_type=authorization_code&redirect_uri={_redirectUri}&code={code}";
request.AddParameter("application/x-www-form-urlencoded", requestBody, ParameterType.RequestBody);

I found a lot of SO questions and a hillarious blog post addressing the issue. It’s good to know I wasn’t alone in this!

The rest is very straightforward. Make the request, parse JSON and assign the list to the chart:

public void GetWeightData()
{
    var endDate = DateTime.Today.ToString("yyyy-MM-dd");
    var startDate = DateTime.Today.AddDays(-30).ToString("yyyy-MM-dd");
    var url = $"https://api.fitbit.com/1/user/XXXXXX/body/log/weight/date/{startDate}/{endDate}.json";
    var response = SendRequest(url);

    ParseWeightData(response);
}

public void ParseWeightData(string rawContent)
{
    var weightJsonArray = JObject.Parse(rawContent)["weight"].ToArray();
    foreach (var weightJson in weightJsonArray)
    {
        var weight = new FitbitWeightResult();
        weight.Weight = weightJson["weight"]?.Value<decimal>() ?? 0;
        weight.Date = weightJson["date"].Value<DateTime>();
        WeightResultList.Add(weight);
    }
}

And the output is:

Conclusion

So far I managed to collect walking data from MS Band, weight data from Fitbit Aria. In this demo I limited the scope with weight data only but Fitbit API can be used to track sleep, exercise and nutrition.

I currently use My Fitness Pal to log what I eat. They too have an API but even though I requested twice they haven’t given me a key yet! Good news is Fitbit has a key and I can get my MFP logs through Fitbit API. I also log my sleep on Fitbit manually so next step is to combine all these in one application to have a nice overview.

Resources

c#, development, gadget, band comments edit

I bought this about 6 months ago and in this post I’ll talk about my experiences so far. They released version 2 of it in last November so I thought I should write about it before it gets terribly outdated!

Choosing the correct size

It comes in 3 sizes: Small, Medium and Large and finding the correct size is the first challenge. They seem to have improved the sizing guide for version 2. In the original one they didn’t mention the appropriate size for wrist’s circumference.

To have the same effect I followed someone’s advice on a forum regarding the circumferences. Downloaded a printable ruler to measure mine. It was at the border of medium and laarge and I decided to go with medium but even at the largest setting it’s not comfortable and irritates my skin. Most of the time I have to wear it on top of a large plaster

Wearing notes

I hope they fixed it in v2 but the first generation Band is quite bulky and uncomfortable. To be honest most of the time I just kept wearing it because I had spent £170 and didn’t come to terms with making a terrible investment. I wear it when I’m walking but as soon as I arrive at home or work I take it off because it’s almost impossible to type something with it.

Band in action

For solely getting fitness data purposes you can use it without pairing with your phone but pairing is helpful as you can read your texts on it, see emails and answer calls.

I also installed the Microsoft Health app and started using Microsoft Health dashboard:

Troubleshooting

As soon as I started using it I noticed a discrepancy with the step count on the Microsoft Health dashboard. Turns out by default it was using phone’s motion tracker as well so it was doubling my steps. After I turned it off started getting the exact same results as on Band.

Turn off motion tracking to get accurate results

Developing with Band and Cloud API

Recording data about something helps tremendously to make it manageable. That’s why I like using these health & fitness gadgets. But of course it doesn’t mean much if you don’t make sense of that data.

In my sample application I used Microsoft Health Cloud API to get the Band’s data. In order this to work Band needs to sync with Microsoft Health app on my phone and the app syncs with my MS account.

The API has a great guide here that can be downloaded as a PDF. It outlines all the necessary steps very clearly and in detail.

Long story short, firstly you need to go to Microsoft Account Developer Center and register an application. This will give you a client ID and client secret that will be used for OAuth 2.0 authentication.

API uses OAuth 2.0 authentication. After the token has been acquired, using the actual API is quite simple, in my example app I used /Summaries endpoint to get the daily step counts.

Implementation

The sample application is a simple WPF desktop application. Upon launch it checks if the user has an access token stored, if not then it shows the OAuth window and the user need to login to their account.

To let the user login to their Microsoft account I added a web browser control to a window and navigated to authorization page:

string authUri = $"{baseUrl}/oauth20_authorize.srf?client_id={Settings.Default.ClientID}&scope={_scope}&response_type=code&redirect_uri={_redirectUri}";
webBrowser.Navigate(authUri);

Once the authorization is complete, the web browser is redirected to with a query parameter code. This is not the actual token we need. Now, we need to go to another URL (oauth20_token.srf) with this code and client secret as parameters and redeem the actual access token:

private void webBrowser_Navigated(object sender, System.Windows.Navigation.NavigationEventArgs e)
{
    if (e.Uri.Query.Contains("code=") && e.Uri.Query.Contains("lc="))
    {
        string code = e.Uri.Query.Substring(1).Split('&')[0].Split('=')[1];

        string authUriRedeem = $"/oauth20_token.srf?client_id={Settings.Default.ClientID}&redirect_uri={_redirectUri}&client_secret={Settings.Default.ClientSecret}&code={code}&grant_type=authorization_code";

        var client = new RestClient(baseUrl);
        var request = new RestRequest(authUriRedeem, Method.GET);
        var response = (RestResponse)client.Execute(request);
        var content = response.Content;

        // Parse content and get auth code
        Settings.Default.AccessToken = JObject.Parse(content)["access_token"].Value<string>();
        Settings.Default.Save();

        Close();
    }
}

After we get the authorization out of the way we can actually call the API and get some results. It’s a simple GET call (https://api.microsofthealth.net/v1/me/Summaries/daily) and the response JSON is pretty straightforward. The only thing to keep in mind is to add the access token to Authorization header:

request.AddHeader("Authorization", $"bearer {Settings.Default.AccessToken}");

Here’s a sample output for a daily summary:

{
	"userId": "67491ecc-c408-47b6-a3ad-041edb410524",
	"startTime": "2016-01-18T00:00:00.000+00:00",
	"endTime": "2016-01-19T00:00:00.000+00:00",
	"parentDay": "2016-01-18T00:00:00.000+00:00",
	"isTransitDay": false,
	"period": "Daily",
	"duration": "P1D",
	"stepsTaken": 2784,
	"caloriesBurnedSummary": {
		"period": "Daily",
		"totalCalories": 1119
	},
	"heartRateSummary": {
		"period": "Daily",
		"averageHeartRate": 77,
		"peakHeartRate": 88,
		"lowestHeartRate": 68
	},
	"distanceSummary": {
		"period": "Daily",
		"totalDistance": 232468,
		"totalDistanceOnFoot": 232468
}

Since we now have the data, we can visualize it:

If you want to play with the sample code don’t forget to register an app and update the settings with your client ID and secret

Next

I guess the most fun would be to develop something that actually runs on the device. My next goal with my Band is to develop a custom tile using its SDK. I hope I can finish it while a first-gen device is still fairly relevant.

Resources

aws, s3, ec2, eip comments edit

I’ve been using AWS for a few years now and over the years I noticed there some questions that keep popping up. I was confused by these issues at first and as they look like they are tripping everybody up at some point I decided to compile of a small list of common gotchas. I’ll update this or post another when if I come across more of these.

1. The S3 folder delusion

When you AWS console you can create folders to group objects but this is just a delusion deliberately created by AWS to simplify the usage. In reality, S3 has a flat structure and all the objects are on the same level. Here’s the excerpt from AWS documentation that states this fact:

In Amazon S3, buckets and objects are the primary resources, where objects are stored in buckets. Amazon S3 has a flat structure with no hierarchy like you would see in a typical file system. However, for the sake of organizational simplicity, the Amazon S3 console supports the folder concept as a means of grouping objects. Amazon S3 does this by using key name prefixes for objects.

So essentially AWS is just smart enough to recognize the standard folder notation we’ve been using for ages to make this things easier for us.

2. Reserved instance confusion

Reserved instances cost less but require some resource planning and paying some money up-front. Although there is now an option to buy reserved instances with no upfront payment they generally shine on long-term commitments with heavy usage (always-on machines). The confusing bit you don’t reserve actual instances. Unfortunately management console doesn’t do a great job in bridging that gap and when you buy a reserved instance you don’t even know which running instance it covers.

Basically you just buy a subscription for 1 or 3 years and you pay less for any machine that meets that criteria. For instance, say you reserved 1 Linux t1.small instance for 12 months and you are running 2 t1.small Linux instances at the moment. You will pay reserved instance prices for one of them and on-demand price for the other. From a financial point of view it doesn’t matter which one is which. If you shut down one of those instances, again regardless of the instance, you still pay for reserved instance price as it matches your reserved instance criteria.

So that’s all there is to it really. Reserved instance is just about billing and has nothing to do with the actual running instances.

3. Public/Elastic IP uncertainty

There are 3 types of IP addresses in AWS:

Private IPs are internal IP that every instance are assigned. They remain the same throughout the lifespan of the instance and as the name implies they are not addressable from the Internet.

Public IPs are optional. They remain the same as long as the instance is running but they are likely to change after a reboot. So they are not reliable for web-accessible applications.

Elastic IPs are basically static IPs that never change. By default AWS gives up to 5 EIPs. If you need more you have to contact their support. They come free of charge as long as they are associated with a running instance. It costs a small amount if you just want to keep them around without using them though.

Resources

csharp, wpf, syncfusion, pdf comments edit

Every few months I have to clean up my desktop computer as dust gets stuck in the CPU fan and it gets hot and slow and loud and annoying! A few days ago I snapped and decided to phase out the desktop and made my laptop as main machine. Even though I love making a fresh start on a new computer it comes with re-installing a bunch of stuff.

One missing thing that made itself obvious at the very start was a PDF reader. So far I’ve always been disappointed with PDF viewers. They are too bloated with unnecessary features and they always try to install a browser toolbar or an anti-virus trial.

My DIY PDF Reader

I started looking into my options to build my own PDF viewer and fortunately didn’t have to look too much. SyncFusion is offering a free license to their products for indie developers and small startups. I used their great wizard control in a past project (Image2PDF) so I first checked if they had something for me. Turns out they have exactly everything I needed wrapped in an easy to use control. Their WPF suite comes with a PdfViewerControl. It supports standard navigation and zooming functions which is pretty much what I need from a PDF viewer. So all I had to do was a start a new WPF project, drag & drop PdfViewerControl and run!

The whole XAML code looks like this:

<Window
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
        xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
        xmlns:local="clr-namespace:PdfViewer"
        xmlns:PdfViewer="clr-namespace:Syncfusion.Windows.PdfViewer;assembly=Syncfusion.PdfViewer.WPF" x:Class="PdfViewer.MainWindow"
        mc:Ignorable="d"
        Title="Easy PDF Viewer" 
        WindowState="Maximized">
    <Grid>
        <PdfViewer:PdfViewerControl  HorizontalAlignment="Stretch" VerticalAlignment="Stretch"/>
    </Grid>
</Window>

And for my 5 minutes, this is the application I got:

Conclusion

If I need more features in the future I think I’ll just build on this. I always have the open source PDF library iTextSharp which I like quite a lot and now have SyncFusion PDF components and libraries in my arsenal, I have no intention to deal with adware-ridden, bloated applications with lots of security flaws.

Resources

csharp, ios, swift, wpf, xamarin, tfl api comments edit

Recently I discovered that Transport for London (TFL) has some great APIs that I can play around with some familiar data. It’s very to use as an API key is not even mandatory. My main goal here is to discover what I can do with this data and build a few user interfaces consuming it. All source code is available on GitHub

Tube status

The API endpoint I will use returns the current status of tube lines, an array of the following JSON objects:

{
    "$type": "Tfl.Api.Presentation.Entities.Line, Tfl.Api.Presentation.Entities",
    "id": "central",
    "name": "Central",
    "modeName": "tube",
    "created": "2015-10-14T10:31:00.39",
    "modified": "2015-10-14T10:31:00.39",
    "lineStatuses": [
        {
            "$type": "Tfl.Api.Presentation.Entities.LineStatus, Tfl.Api.Presentation.Entities",
            "id": 0,
            "statusSeverity": 10,
            "statusSeverityDescription": "Good Service",
            "created": "0001-01-01T00:00:00",
            "validityPeriods": []
        }
    ],
    "routeSections": [],
    "serviceTypes": [
        {
            "$type": "Tfl.Api.Presentation.Entities.LineServiceTypeInfo, Tfl.Api.Presentation.Entities",
            "name": "Regular",
            "uri": "/Line/Route?ids=Central&serviceTypes=Regular"
        }
    ]
}

Visualizing the data - line colours

TFL have standard colours for tube lines which are documented here. So I created a small lookup json using that reference:

[
    { "id": "bakerloo", "CMYK": { "M": 58, "Y": 100, "K": 33 }, "RGB":  { "R": 137, "G": 78, "B": 36 } },
    { "id": "central", "CMYK": { "M": 95, "Y": 100 }, "RGB":  { "R": 220, "G": 36, "B": 31 } },
    { "id": "circle",  "CMYK": { "M": 16, "Y": 100 }, "RGB":  { "R": 255, "G": 206, "B": 0 } },
    { "id": "district", "CMYK": { "C":  95, "Y": 100, "K": 27 }, "RGB":  { "R": 0, "G": 114, "B": 41 } },
    { "id": "hammersmith-city", "CMYK": { "M": 45, "Y": 10 }, "RGB":  { "R": 215, "G": 153, "B": 175 } },
    { "id": "jubilee", "CMYK": { "C": 5, "K": 45 }, "RGB":  { "R": 134, "G": 143, "B": 152 } },
    { "id": "metropolitan", "CMYK": { "C": 5, "M": 100, "K": 40 }, "RGB":  { "R": 117, "G": 16, "B": 86 } },
    { "id": "northern", "CMYK": { "K": 100 }, "RGB":  { "R": 0, "G": 0, "B": 0 } },
    { "id": "piccadilly", "CMYK": { "C": 100, "M": 88, "K": 5 }, "RGB":  { "R": 0, "G": 25, "B": 168 } },
    { "id": "victoria", "CMYK": { "C": 85, "M": 19 }, "RGB":  { "R": 0, "G": 160, "B": 226 } },
    { "id": "waterloo-city", "CMYK": { "C": 47, "Y": 32 }, "RGB":  { "R": 118, "G": 208, "B": 189 } }
]

I was hoping to map status values to colours as well (i.e. “Severe delays” to red) but there is no official guide to that. The status codes and values can be retrieved from this endpoint: https://api.tfl.gov.uk/line/meta/severity which returns a collection of objects like this:

{
    "$type": "Tfl.Api.Presentation.Entities.StatusSeverity, Tfl.Api.Presentation.Entities",
    "modeName": "tube",
    "severityLevel": 2,
    "description": "Suspended"
}

I simplified it for my purposes (just the values for tube):

[
    { "severityLevel": 0, "description": "Special Service" },
    { "severityLevel": 1, "description": "Closed" },
    { "severityLevel": 2, "description": "Suspended" },
    { "severityLevel": 3, "description": "Part Suspended" },
    { "severityLevel": 4, "description": "Planned Closure" },
    { "severityLevel": 5, "description": "Part Closure" },
    { "severityLevel": 6, "description": "Severe Delays" },
    { "severityLevel": 7, "description": "Reduced Service" },
    { "severityLevel": 8, "description": "Bus Service" },
    { "severityLevel": 9, "description": "Minor Delays" },
    { "severityLevel": 10, "description": "Good Service" },
    { "severityLevel": 11, "description": "Part Closed" },
    { "severityLevel": 12, "description": "Exist Only" },
    { "severityLevel": 13, "description": "No Step Free Access" },
    { "severityLevel": 14, "description": "Change of frequency" },
    { "severityLevel": 15, "description": "Diverted" },
    { "severityLevel": 16, "description": "Not Running" },
    { "severityLevel": 17, "description": "Issues Reported" },
    { "severityLevel": 18, "description": "No Issues" },
    { "severityLevel": 19, "description": "Information" },
    { "severityLevel": 20, "description": "Service Closed" },
]

I will keep it around but in this initial version I won’t use it as description is returned with the status query anyway. But it was still a useful exercise to figure out there is no “official” colour for status values. After all what’s the colour of “No Step Free Access” or “Exist Only”? There is a also reason field that explains the effects of any delays etc. which should ne displayed along with the severity especially when there are some disruptions in the service.

‘Nuff said about the data! Let’s start building something with it!

Core library

As I will build several API call to retrieve tube status is encapsulated in the core library which basically has sends the HTTP request, parses the JSON and returns the LineInfo list:

public class Fetcher
{
    private readonly string _apiEndPoint = "https://api.tfl.gov.uk/line/mode/tube/status?detail=true";

    public List<LineInfo> GetTubeInfo()
    {
        var client = new RestClient(_apiEndPoint);
        var request = new RestRequest("/", Method.GET);
        request.AddHeader("Content-Type", "application/json");
        var response = (RestResponse)client.Execute(request);
        var content = response.Content;
        var tflResponse = JsonConvert.DeserializeObject<List<TflLineInfo>>(content);

        var lineInfoList = tflResponse.Select(t =>
            new LineInfo()
            {
                Id = t.id,
                Name = t.name,
                Reason = t.lineStatuses[0].reason,
                StatusSeverityDescription = t.lineStatuses[0].statusSeverityDescription,
                StatusSeverity = t.lineStatuses[0].statusSeverity
            }).ToList();

        return lineInfoList;
    }
}

LineInfo class contains the current status with the description. It also contains the colour defined by TFL for that tube line:

public class LineInfo
{
    public string Id { get; set; }
    public string Name { get; set; }
    public int StatusSeverity { get; set; }
    public string StatusSeverityDescription { get; set; }
    public string Reason { get; set; }
    public RGB LineColour
    {
        get
        {
            return TubeColourHelper.GetRGBColour(this.Id);
        }
    }
}

As the line colours aren’t returned by the service I have to populate it by a helper class:

public class TubeColourHelper
{
    private static Dictionary<string, RGB> _tubeColorRGBDictionary = new Dictionary<string, RGB>();

    static TubeColourHelper()
    {
        _tubeColorRGBDictionary = new Dictionary<string, RGB>();

        string json = File.ReadAllText("./data/colours.json");
        var tubeColors = JArray.Parse(json);
        foreach (var tubeColor in tubeColors)
        {
            _tubeColorRGBDictionary.Add(tubeColor["id"].Value<string>(), new RGB(
                tubeColor["RGB"]["R"]?.Value<int>() ?? 0,
                tubeColor["RGB"]["G"]?.Value<int>() ?? 0,
                tubeColor["RGB"]["B"]?.Value<int>() ?? 0));
        }
    }
    
    public static RGB GetRGBColour(string lineId)
    {
        if (!_tubeColorRGBDictionary.ContainsKey(lineId))
        {
            throw new ArgumentException($"Colour for line [{lineId}] could not be found in RGB colour map");
        }

        return _tubeColorRGBDictionary[lineId];
    }
}

The static constructor runs only the first time it is accessed, reads the colours.json and populates the dictionary. From then on it’s just a lookup in memory.

First client: C# Console Application on Windows

Time to develop our first client and see some actual results. As it’s generally the case with console applications this one is pretty simple and hassle-free. I decided to start with that one just to see the core library is working as expected.

class Program
{
    static void Main(string[] args)
    {
        var fetcher = new Fetcher();
        var viewer = new ConsoleViewer();

        bool exit = false;

        viewer.DisplayTubeStatus(fetcher.GetTubeInfo());

        do
        {
            ConsoleKeyInfo key = System.Console.ReadKey();
            switch (key.Key)
            {
                case ConsoleKey.F5:
                    viewer.DisplayTubeStatus(fetcher.GetTubeInfo());
                    break;
                case ConsoleKey.Q:
                    exit = true;
                    break;
                default:
                    System.Console.WriteLine("Unknown command");
                    break;
            }
        }
        while (!exit);
    }
}

Displays the results when it’s first run. You can refresh by pressing F5 or quit by pressing Q. The output looks like this:

The problem with console application is that I wasn’t able to use RGB values directly as the console only supports an enumeration called ConsoleColor.

Second client: WPF Application on Windows

Now let’s look at a more graphical UI, a WPF client:

Same idea, display the results upon first run then call the service again on Refresh button’s click event.

Third client: iOS App with Xamarin

I’ve recently subscribed to Xamarin and one of the main reasons for starting this project was to see it in action. What I was mostly curious about was if I could use my C# libraries using NuGet packages on an iOS application developed with Xamarin. This would allow me build apps significantly faster.

It didn’t work out of the box because I used C# 6.0 and .NET Framework 4.5.2 on the Windows side but it wasn’t available on the Mac.

But wasn’t too hard to change the framework and make some small modifications to make it work. Good news is that it supports NuGet and most common libraries have Mono support including RestSharp and Newtonsoft.Json which I used in this project

I had to remove and add them but finally they worked fine so I didn’t have to change anything in the code.

I won’t go into implementation details as there’s not much change. The app has one table view controller and it calls the core library to get the results and assigns them to the table’s data source. It’s a relief that I could have the same functionality as Windows with just minor changes.

public override void ViewDidLoad()
{
    base.ViewDidLoad();

    var fetcher = new Fetcher();
    var lineInfoList = fetcher.GetTubeInfo();
    TableView.Source = new TubeStatusTableViewControllerSource(lineInfoList.ToArray());
    TableView.ReloadData();
}

Anyway, more on Xamarin later after I cover the Swift version.

Fourth client: iOS App with Swift

Last but not least, here comes Swift client built with XCode. Naturally this one cannot use the core library that the first 3 clients shared (which is good because I was looking for a chance to practice handling HTTP requests and parsinng JSON with Swift anyway).

I didn’t use any external libraries so the implementation is a bit long but mainly it sends the request using NSURLSession and NSSessionDataTask.

func getTubeStatus(completionHandler: (result: [LineInfo]?, error: NSError?) -> Void) {
    let parameters = ["detail" : "true"]
    let mutableMethod : String = Methods.TubeStatus
    
    taskForGETMethod(mutableMethod, parameters: parameters) { JSONResult, error in
        if let error = error {
            completionHandler(result: nil, error: error)
        } else {
            if let results = JSONResult as? [AnyObject] {
                let lineStatus = LineInfo.lineStatusFromResults(results)
                completionHandler(result: lineStatus, error: nil)
            }
       }
    }
}

Then constructs the LineInfo objects by calling the static lineStatusFromResults method:

static func lineStatusFromResults(results: [AnyObject]) -> [LineInfo] {
    var lineStatus = [LineInfo]()
    for result in results {
        lineStatus.append(LineInfo(status: result))
    }
    
    return lineStatus
}

which creates a new LineInfo and adds to resultset:

init(status: AnyObject) {

    Id = status["id"] as! String
    Name = status["name"] as! String
    StatusSeverity = status["lineStatuses"]!![0]!["statusSeverity"] as! Int
    StatusSeverityDescription = status["lineStatuses"]!![0]!["statusSeverityDescription"] as! String
    LineColour = RGB(R: 0, G: 0, B: 0)
}

JSON parsing is a bit nasy because of unwrapping the optionals. I’ll look into SwiftyJSON later on which is a popular JSON library for Swift.

Finally the controller displays the results:

override func viewWillAppear(animated: Bool) {
    super.viewWillAppear(animated)
    
    TFLClient.sharedInstance().getTubeStatus { lineStatus, error in
        
        if let lineStatus = lineStatus {
            self.lineInfoList = lineStatus
            dispatch_async(dispatch_get_main_queue()) {
                self.tableView!.reloadData()
            }
        } else {
            print(error)
        }
    }
}

And the custom cells are created when data is loaded and the text and colours are set:

override func tableView(tableView: UITableView, cellForRowAtIndexPath indexPath: NSIndexPath) -> UITableViewCell {
    let cell = tableView.dequeueReusableCellWithIdentifier("TubeInfoCell", forIndexPath: indexPath) as! TubeInfoTableViewCell
    let lineStatus = lineInfoList[indexPath.row]
    
    cell.backgroundColor = colourHelper.getTubeColor(lineStatus.Id)
    
    cell.lineName?.text = lineStatus.Name
    cell.lineName?.textColor = UIColor.whiteColor()
    cell.severityDescription?.text = lineStatus.StatusSeverityDescription
    cell.severityDescription?.textColor = UIColor.whiteColor()

    return cell
}

And here’s the output:

Xamarin vs Swift

Here’s a quick overview and comparison of both platforms based on my (limited) experiences with this toy project:

  • XCode is much faster when building and deploying
  • XamarinStudio doesn’t seem to be very intuitive at times. For examples the code snippets use Java-notation
  • The more I use Swift the more I like it and it doesn’t slow me down terribly. Once you get used to it the difference is syntax more or less. For example the following two methods do the same thing:

    Xamarin:

    csharp public override nint RowsInSection (UITableView tableview, nint section) { return lineInfoList.Length; }

    Swift:

    swift override func tableView(tableView: UITableView, numberOfRowsInSection section: Int) -> Int { return lineInfoList.count }

    I can even argue I’d be more comfortable with the Swift version here as I have no idea what a “nint” is as it’s an input parameter in the Xamarin version.

  • The idea behind Xamarin subscription was to develop iOS apps quickly as I’m a seasoned C# developer and feel comfortable with it. But turns it, I can’t move as fast as I expected. With the Indie subscription you can only use Xamarin Studio. Enabling Visual Studio is only allowed with the business version which costs $1000/year. And Xamarin Studio is a brand new IDE for me so it definitely has a learning curve. Also I’m getting used to XCode now (besides the fact that it crashes hundred times a day on average!)

Conclusion

This was just a reconnaisance mission to explore the TFL API, iOS development with Xamarin and Swift. It was a fun exercise for me, I hope anyone who reads this can benefit from it too.

Resources

csharp, nancy comments edit

Recently I needed to simulate HTTP responses from a 3rd party. I decided to use Nancy to quickly build a local web server that would handle my test requests and return the responses I wanted.

Here’s the definition of Nancy from their official website:

Nancy is a lightweight, low-ceremony, framework for building HTTP based services on .Net and Mono.

It can handle DELETE, GET, HEAD, OPTIONS, POST, PUT and PATCH requests. It’s very easy to customize and extend as it’s module-based. In order to build our tiny web server we are going to need self-hosting package:

Install-Package Nancy.Hosting.Self

This would automatically install Nancy as it depends on that package.

Self-hosting in action

The container application can be anything as long it keeps running one way or another. A background service would be ideal for this task. Since all I need is testing I just created a console application and added Console.ReadKey() statement to keep it “alive”

class Program
{
    private string _url = "http://localhost";
    private int _port = 12345;
    private NancyHost _nancy;

    public Program()
    {
        var uri = new Uri( $"{_url}:{_port}/");
        _nancy = new NancyHost(uri);
    }

    private void Start()
    {
        _nancy.Start();
        Console.WriteLine($"Started listennig port {_port}");
        Console.ReadKey();
        _nancy.Stop();
    }

    static void Main(string[] args)
    {
        var p = new Program();
        p.Start();
    }
}

If you try this code, it’s likely that you’ll have an error (AutomaticUrlReservationCreationFailureException)

saying:

The Nancy self host was unable to start, as no namespace reservation existed for the provided url(s).

Please either enable UrlReservations.CreateAutomatically on the HostConfiguration provided to 
the NancyHost, or create the reservations manually with the (elevated) command(s):

netsh http add urlacl url="http://+:12345/" user="Everyone"

There are 3 ways to resolve this issue and two of which are already suggested in the error message:

  1. In an elevated command prompt (fancy way of saying run as administrator!), run

    netsh http add urlacl url="http://+:12345/" user="Everyone"

    What add urlacl does is

    Reserves the specified URL for non-administrator users and accounts

    If you want to delete it later on you can use the following command

    netsh http delete urlacl url=http://+:12345/

  2. Specify a host configuration to NancyHost like this:

    ```csharp var configuration = new HostConfiguration() { UrlReservations = new UrlReservations() { CreateAutomatically = true } };

    _nancy = new NancyHost(configuration, uri); ```

    This essentially does the same thing and a UAC prompt pops up so it’s not that automatical!

  3. Run the Visual Studio (and the standalone application when deployed) as administrator

After applying either one of the 3 solutions, let’s run the application and try the address http://localhost:12345 in a browser and we get …

Excellent! We are actually getting a response from the server even though it’s just a 404 error.

Now let’s add some functionality, otherwise it isn’t terribly useful.

Handling requests

Requests are handled by modules. Creating a module is as simple as creating a class deriving from NancyModule. Let’s create two handlers for the root, one for GET verbs and one for POST:

public class SimpleModule : Nancy.NancyModule
{
    public SimpleModule()
    {
        Get["/"] = _ => "Received GET request";

        Post["/"] = _ => "Received POST request";
    }
}

Nancy automatically discovers all modules so we don’t have to register them. If there are conflicting handlers the last one discovered overrides the previous ones. For example the following example would work fine and the second GET handler will be executed:

public class SimpleModule : Nancy.NancyModule
{
    public SimpleModule()
    {
        Get["/"] = _ => "Received GET request";

        Post["/"] = _ => "Received POST request";

        Get["/"] = _ => "Let me have the request!";
    }
}

Working with input data: Request parameters

In the simple we used underscore to represent input as didn’t care but most of the time we would. In that case we can get the request parameters as a DynamicDictionary (a type that comes with Nancy). For example let’s create a route for /user:

public SimpleModule()
{
    Get["/user/{id}"] = parameters =>
    {
        if (((int)parameters.id) == 666)
        {
            return $"All hail user #{parameters.id}! \\m/";
        }
        else
        {
            return "Just a regular user!";
        }
    };
}

And send the GET request:

GET http://localhost:12345/user/666 HTTP/1.1
User-Agent: Fiddler
Host: localhost:12345
Content-Length: 2

which would return the response:

HTTP/1.1 200 OK
Content-Type: text/html
Server: Microsoft-HTTPAPI/2.0
Date: Tue, 10 Nov 2015 11:40:08 GMT
Content-Length: 23

All hail user #666! \m/

Working with input data: Request body

Now let’s try to handle the data posted in the request body. Data posted in the body can be accessed though this.Request.Body property such as for the following request

POST http://localhost:12345/ HTTP/1.1
User-Agent: Fiddler
Host: localhost:12345
Content-Length: 55
Content-Type: application/json

{
    "username": "volkan",
    "isAdmin": "sure!"
}

this code would first convert the request stream to a string and deserialize it to a POCO:

Post["/"] = _ =>
{
    var id = this.Request.Body;
    var length = this.Request.Body.Length;
    var data = new byte[length];
    id.Read(data, 0, (int)length);
    var body = System.Text.Encoding.Default.GetString(data);

    var request = JsonConvert.DeserializeObject<SimpleRequest>(body);
    return 200;
};

If the was posted from a form for example and sent in the following format in the body

username=volkan&isAdmin=sure!

then we could simply convert it to a dictionary with a little bit of LINQ:

Post["/"] = parameters =>
{
    var id = this.Request.Body;
    long length = this.Request.Body.Length;
    byte[] data = new byte[length];
    id.Read(data, 0, (int)length);
    string body = System.Text.Encoding.Default.GetString(data);
    var p = body.Split('&')
        .Select(s => s.Split('='))
        .ToDictionary(k => k.ElementAt(0), v => v.ElementAt(1));

    if (p["username"] == "volkan")
        return "awesome!";
    else
        return "meh!";
};

This is nice but it’s a lot of work to read the whole and manually deserialize it! Fortunately Nancy supports model binding. First we need to add the using statement as the Bind extension method lives in Nancy.ModelBinding

using Nancy.ModelBinding;

Now we can simplify the code by the help of model binding:

Post["/"] = _ =>
{
    var request = this.Bind<SimpleRequest>();
    return request.username;
};

The important thing to note is to send the data with the appropriate content type. For the form data example the request should be like this:

POST http://localhost:12345/ HTTP/1.1
User-Agent: Fiddler
Host: localhost:12345
Content-Length: 29
Content-Type: application/x-www-form-urlencoded

username=volkan&isAdmin=sure!

It also works for binding JSON to the same POCO.

Preparing responses

Nancy is very flexible in terms of responses. As shown in the above examples you can return a string

Post["/"] = _ =>
{
    return "This is a valid response";
};

which would yield this HTTP message on the wire:

HTTP/1.1 200 OK
Content-Type: text/html
Server: Microsoft-HTTPAPI/2.0
Date: Tue, 10 Nov 2015 15:48:12 GMT
Content-Length: 20

This is a valid response

Response code is set to 200 - OK automatically and the text is sent in the response body.

We can just set the code and return a response with a simple one-liner:

Post["/"] = _ => 405;

which would produce:

HTTP/1.1 405 Method Not Allowed
Content-Type: text/html
Server: Microsoft-HTTPAPI/2.0
Date: Tue, 10 Nov 2015 15:51:36 GMT
Content-Length: 0

To prepare more complex responses with headers and everything we can construct a new Response object like this:

Post["/"] = _ =>
{
    string jsonString = "{ username: \"admin\", password: \"just kidding\" }";
    byte[] jsonBytes = Encoding.UTF8.GetBytes(jsonString);

    return new Response()
    {
        StatusCode = HttpStatusCode.OK,
        ContentType = "application/json",
        ReasonPhrase = "Because why not!",
        Headers = new Dictionary<string, string>()
        {
            { "Content-Type", "application/json" },
            { "X-Custom-Header", "Sup?" }
        },
        Contents = c => c.Write(jsonBytes, 0, jsonBytes.Length)
    };
};

and we would get this at the other end of the line:

HTTP/1.1 200 Because why not!
Content-Type: application/json
Server: Microsoft-HTTPAPI/2.0
X-Custom-Header: Sup?
Date: Tue, 10 Nov 2015 16:09:19 GMT
Content-Length: 47

{ username: "admin", password: "just kidding" }

Response also comes with a lot of useful methods like AsJson, AsXml and AsRedirect. For example we could simplify returning a JSON response like this:

Post["/"] = _ =>
{
    return Response.AsJson<SimpleResponse>(
        new SimpleResponse()
        {
            Status = "A-OK!", ErrorCode = 1, Description = "All systems are go!"
        });
};

and the result would contain the appropriate header and status code:

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Server: Microsoft-HTTPAPI/2.0
Date: Tue, 10 Nov 2015 16:19:18 GMT
Content-Length: 68

{"status":"A-OK!","errorCode":1,"description":"All systems are go!"}

One extension I like is the AsRedirect method. The following example would return Google search results for a given parameter:

Get["/search"] = parameters =>
{
    string s = this.Request.Query["q"];
    return Response.AsRedirect($"http://www.google.com/search?q={s}");
};

HTTPS

What if we needed to support HTTPS for our tests for some reason? Fear not, Nancy covers that too. By default, if we just try to use HTTPS by changing the protocol we would get this exception:

The connection to ‘localhost’ failed. System.Security.SecurityException Failed to negotiate HTTPS connection with server.fiddler.network.https HTTPS handshake to localhost (for #2) failed. System.IO.IOException Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.

The solution is to add create a self-signed certificate and add it using netsh http add command. Here’s the step-by-step process:

  1. Create a self-signed certificate: Open a Visual Studio command prompt and enter the following command:
makecert nancy.cer

You can provide more properties so that it would look with a name that makes sense. Here’s an MSD page to

  1. Run mmc and add Certificates snap-in. Make sure to select Computer Account.

I selected My User Account at first and it gave the following error:

SSL Certificate add failed, Error: 1312 A specified logon session does not exist. It may already have been terminated.

In that case the solution is just to drag and drop the certificate to the computer account as shown below:

  1. Right-click on Certificates (Local Computer) -> Personal -> Certificates and select All tasks -> Import and browse to nancy.cer file created in Step 1

  2. Double-click on the certificate, switch to Details tab and scroll to the bottom and copy the Thumbprint value (and remove the spaces after copied it)

  1. Now enter the following commands. The first one is the same as before, just with HTTPS as protocol. The second command add the certificate we’ve just created.
netsh http add urlacl url=https://+:12345/ user="Everyone"

netsh http add sslcert ipport=0.0.0.0:1234 ccerthash=653a1c60d4daaae00b2a103f242eac965ca21bec appid={A0DEC7A4-CF28-42FD-9B85-AFFDDD4FDD0F} clientcertnegotiation=enable

Here appid can be any GUID.

Let’s take it out for a test drive:

Get["/"] = parameters =>
{
    return "Response over HTTPS! Weeee!";
};

This request

GET https://localhost:12345 HTTP/1.1
Host: localhost:12345


returns this response

HTTP/1.1 200 OK
Content-Type: text/html
Server: Microsoft-HTTPAPI/2.0
Date: Wed, 11 Nov 2015 10:24:58 GMT
Content-Length: 27

Response over HTTPS! Weeee!

Conclusion

There are a few alternatives when you need a small web server to test something locally. Nancy is one of them. It’s easy to configure, use and it’s lightweight. Apparently you can even host in on a Raspberry Pi!

Resources

csharp, design, development, mef, managed extensibility framework comments edit

In this post I will try to cover some of the basic concepts and features of MEF over a working example. In the future I’ll post more articles demonstraint MEF usage with more complex applications.

Background

Many successful and popular applications, such as Visual Studio, Eclipse, Sublime Text, support a plug-in model. Adopting a plugin-based model, whenever possible, has quite a few advantages:

  • Helps to keep the core lightweight instead of cramming all features into the same code-base.
  • Helps to make the application more robust: New functionality can be added without changing any existing code.
  • Helps to make development easier as different modules can be developed by different people simultaneously
  • Allows plugin development without distributing the main the source code

Extensibility is based on composition and it is very helpful to build SOLID compliant applications as it adopts Open/closed and Dependency Inversion principles.

MEF is part of .NET framework as of version 4.0 and it lives inside System.ComponentModel.Composition namespace. This is also the standard extension model that has been used in Visual Studio. It is not meant to replace Invesion of Control (IoC) frameworks. It is rather meant to simplify building extensible applications using dependency injection based on component composition.

Some terminology

Before diving into the sample, let’s look at some MEF terminology and core terms:

  • Part: Basic elements in MEF are called parts. Parts can provide services to other parts (exporting) and can consume other parts’ services (importing).

  • Container: This is the part that performs the composition. Most common one is CompositionContainer class.

  • Catalog: In order to discover the parts, containers use catalogs. There are various catelogs suplied by MEF such as

    • AssemblyCatalog: Discovers attributed parts in a managed code assembly.
    • DirectoryCatalog: Discovers attributed parts in the assemblies in a specified directory.
    • AggregateCatalog: A catalog that combines the elements of ComposablePartCatalog objects.
    • ApplicationCatalog: Discovers attributed parts in the dynamic link library (DLL) and EXE files in an application’s directory and path
  • Export / import: The way the plugins make themselves discoverable is by exporting their implementation of a contract. A contract is simply a common interface that the application and the plugins understand so they can speak the same language so to speak.

Sample Project

As I learn best by playing around, I decided to start with a simple project. I’ve recently published a sample project for Strategy design pattern which I blogged here. In this post I will use the same project and convert it into a plugin-based version.

IP Checker with MEF v1: Bare Essentials

At this point we have everything we need for the first version of the plugin-based IP checker. Firstly, I divided my project into 5 parts:

  • IPCheckerWithMEF.Lab: The consumer application
  • IPCheckerWithMEF.Contract: Project containing the common interface
  • Plugins: Extensions for the main application
    • IPCheckerWithMEF.Plugins.AwsIPChecker
    • IPCheckerWithMEF.Plugins.CustomIPChecker
    • IPCheckerWithMEF.Plugins.DynDnsIPChecker

I set the output folder of the plugins to a directory called Plugins at the project level.

Let’s see some code!

For this basic version we need 3 things:

  • A container to handle the composition.
  • A catalog that the container can use to discover the plugins.
  • A way to tell which classes should be discovered and imported

In this sample I used a DirectoryCatalog that points to the output folder of the plugin projects. So after adding the required parts above the main application shaped up to be something like this:

public class MainApplication
{
    private CompositionContainer _container;

    [ImportMany(typeof(IIpChecker))]
    public List<IIpChecker> IpCheckerList { get; set; }

    public MainApplication(string pluginFolder)
    {
        var catalog = new DirectoryCatalog(pluginFolder);
        _container = new CompositionContainer(catalog);

        LoadPlugins();
    }

    public void LoadPlugins()
    {
        try
        {
            _container.ComposeParts(this);
        }
        catch (CompositionException compositionException)
        {
            Console.WriteLine(compositionException.ToString());
        }
    }
}

In the constructor, it instantiates a DirectoryCatalog with the given path and passes it to the container. The container imports IIpChecker type objects found in the assemblies inside that folder. Note that we didn’t do anything about IpCheckerList. By decorating it with ImportMany attribute we declared that it’s to be filled by the composition engine. In this example we could only use ImportMany as opposed to Import which would look for a single part to compose. If we used Import we would get the following exception:

Now to complete the circle we need to export our plugins with Export attribute such as:

[Export(typeof(IIpChecker))]
public class AwsIPChecker : IIpChecker
{
    public string GetExternalIp()
    {
        // ...
    }
}

Alternatively we can use InheritedExport attribute on the interface to export any class that implements the IIpChecker interface.

[InheritedExport(typeof(IIpChecker))]
public interface IIpChecker
{
    string GetExternalIp();
}

This way the plugins would still be discovered even if they weren’t decorated with Export attribute because of this inheritance model.

Putting it together

Now that we’ve seen the plugins that export the implementation and part that discovers and imports them let’s see them all in action:

class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine("Starting the main application");

        string pluginFolder = @"..\..\..\Plugins\";
        var app = new MainApplication(pluginFolder);

        Console.WriteLine($"{app.IpCheckerList.Count} plugin(s) loaded..");
        Console.WriteLine("Executing all plugins...");

        foreach (var ipChecker in app.IpCheckerList)
        {
            Console.WriteLine(ObfuscateIP(ipChecker.GetExternalIp()));
        }
    }

    private static string ObfuscateIP(string actualIp)
    {
        return Regex.Replace(actualIp, "[0-9]", "*");
    }
}

We create the consumer application that loads all the plugins in the directory we specify. Then we can loop over and execute all of them:

So far so good. Now, let’s try to export some metadata about our plugins so that we can display the loaded plugins to the user.

IP Checker with MEF v2: Metadata comes into play

In almost all applications plugins come with some sort of information so that the user can identify which ones have been installed and what they do. To export the extra data let’s add a new interface:

public interface IPluginInfo
{
    string DisplayName { get; }
    string Description { get; }
    string Version { get; }
}

And on the plugins we fill that data and export it using the ExportMetadata attribute:

[Export(typeof(IIpChecker))]
[ExportMetadata("DisplayName", "Custom IP Checker")]
[ExportMetadata("Description", "Uses homebrew service developed with Node.js and hosted on Heroku")]
[ExportMetadata("Version", "2.1")]
public class CustomIpChecker : IIpChecker
{
    // ...
}

In v1, we only imported a list of objects implementing IIpChecker. So how do we accommodate this new piece of information? In order to do that we have to change the way we import the plugins and use the Lazy construct:

[ImportMany]
public List<Lazy<IIpChecker, IPluginInfo>> Plugins { get; set; }

According to MSDN this is mandatory to get metadata out of plugins:

The importing part can use this data to decide which exports to use, or to gather information about an export without having to construct it. For this reason, an import must be lazy to use metadata

So let’s load and display this new plugin information:

private static void PrintPluginInfo()
{
    Console.WriteLine($"{_app.Plugins.Count} plugin(s) loaded..");
    Console.WriteLine("Displaying plugin info...");
    Console.WriteLine();

    foreach (var ipChecker in _app.Plugins)
    {
        Console.WriteLine("----------------------------------------");
        Console.WriteLine($"Name: {ipChecker.Metadata.DisplayName}");
        Console.WriteLine($"Description: {ipChecker.Metadata.Description}");
        Console.WriteLine($"Version: {ipChecker.Metadata.Version}");
    }
}

Notice that we access the metadata through [PluginName].Metadata.[PropertyName] properties. To access the actual plugin and call the exported methods we have to use [PluginName].Value such as:

foreach (var ipChecker in _app.Plugins)
{
    ipChecker.Value.GetExternalIp();
}

Managing the plugins

What if we want to add or remove plugins at runtime? We can do it without restarting the application but refreshing the catalog and calling the container’s ComposeParts method again.

In this sample application I added a FileSystemWatcher that listens to the Created and Deleted events on the Plugins folder and calls the LoadPlugins method of the application when an event fires. LoadPlugins first refreshes the catalog and composes the parts:

public void LoadPlugins()
{
    try
    {
        _catalog.Refresh();
        _container.ComposeParts(this);
    }
    catch (CompositionException compositionException)
    {
        Console.WriteLine(compositionException.ToString());
    }
}

But making this change alone isn’t sufficient and we would end up getting a CompositionException:

By default recomposition is disabled so we have to specify it explicitly while importing parts:

[ImportMany(AllowRecomposition = true)]
public List<Lazy<IIpChecker, IPluginInfo>> Plugins { get; set; }

After these changes the final version of composing class looks like this:

public class MainApplication
{
    private CompositionContainer _container;
    private DirectoryCatalog _catalog;

    [ImportMany(AllowRecomposition = true)]
    public List<Lazy<IIpChecker, IPluginInfo>> Plugins { get; set; }

    public MainApplication(string pluginFolder)
    {
        _catalog = new DirectoryCatalog(pluginFolder);
        _container = new CompositionContainer(_catalog);

        LoadPlugins();
    }

    public void LoadPlugins()
    {
        try
        {
            _catalog.Refresh();
            _container.ComposeParts(this);
        }
        catch (CompositionException compositionException)
        {
            Console.WriteLine(compositionException.ToString());
        }
    }
}

and the client app:

class Program
{
    private static readonly string _pluginFolder = @"..\..\..\Plugins\";
    private static FileSystemWatcher _pluginWatcher;
    private static MainApplication _app;

    static void Main(string[] args)
    {
        Console.WriteLine("Starting the main application");

        _pluginWatcher = new FileSystemWatcher(_pluginFolder);
        _pluginWatcher.Created += PluginWatcher_FolderUpdated;
        _pluginWatcher.Deleted += PluginWatcher_FolderUpdated;
        _pluginWatcher.EnableRaisingEvents = true;

        _app = new MainApplication(_pluginFolder);

        PrintPluginInfo();

        Console.ReadLine();
    }

    private static void PrintPluginInfo()
    {
        Console.WriteLine($"{_app.Plugins.Count} plugin(s) loaded..");
        Console.WriteLine("Displaying plugin info...");
        Console.WriteLine();

        foreach (var ipChecker in _app.Plugins)
        {
            Console.WriteLine("----------------------------------------");
            Console.WriteLine($"Name: {ipChecker.Metadata.DisplayName}");
            Console.WriteLine($"Description: {ipChecker.Metadata.Description}");
            Console.WriteLine($"Version: {ipChecker.Metadata.Version}");
        }
    }

    private static void PluginWatcher_FolderUpdated(object sender, FileSystemEventArgs e)
    {
        Console.WriteLine();
        Console.WriteLine("====================================");
        Console.WriteLine("Folder changed. Reloading plugins...");
        Console.WriteLine();
        
        _app.LoadPlugins();

        PrintPluginInfo();
    }
}

After these changes I started the application with 2 plugins in the target folder and added a 3rd one while it’s running and got this output:

It also works the same way for deleted plugins but not for updates because the assemblies are locked by .NET. Adding new plugins at runtime is painless but removing and updating would require more attention as the plugin might be running at the time.

Resources

design, development, design patterns, csharp comments edit

A few days ago I published a post discussing Factory Method pattern. This article is about the other factory design pattern: Abstract Factory.

Use case: Switching between configuration sources easily

Imagine in a C# application you accessed ConfigurationManager.AppSettings whenever you needed a value from the configuration. This would essentially be hardcoding the configuration source and it would be hard to change if you needed to switch to another configuration source (database, web service, etc). A nicer way would be to “outsource” the creation of configuration source to another class.

What is Abstract Factory?

Here’s the official definition from GoF:

Provide an interface for creating families of related or dependent objects without specifying their concrete classes.

Implementation

The application first composes the main class (ArticleFeedGenerator) with the services it will use and starts the process.

static void Main(string[] args)
{
    IConfigurationFactory configFactory = new AppConfigConfigurationFactory();

    IApiSettings apiSettings = configFactory.GetApiSettings();
    IFeedSettings feedSettings = configFactory.GetFeedSettings();
    IFeedServiceSettings feedServiceSettings = configFactory.GetFeedServiceSettings();
    IS3PublisherSettings s3PublishSettings = configFactory.GetS3PublisherSettings();
    IOfflineClientSettings offlineClientSettings = configFactory.GetOfflineClientSettings();

    var rareburgClient = new OfflineRareburgClient(offlineClientSettings);
    var rareburgArticleFeedService = new RareburgArticleFeedService(feedServiceSettings);
    var publishService = new S3PublishService(s3PublishSettings, feedSettings);
    var feedGenerator = new ArticleFeedGenerator(rareburgClient, rareburgArticleFeedService, publishService, feedSettings);

    feedGenerator.Run();
}

This version uses AppConfigConfigurationFactory to get the values from the App.config. When I need to switch to DynamoDB which I used in this example all I have to do is replace one line of code in the application:

var configFactory = new DynamoDBConfigurationFactory();

With this change alone we are essentially replacing a whole family of related classes.

On the factory floor

The abstract factory and the concrete factories implement it are shown below:

Concrete configuration factories create the classes that deal with specific configuration values (concrete products). For instance AppConfigConfigurationFactory looks like this (simplified for brevity):

public class AppConfigConfigurationFactory : IConfigurationFactory
{
    public IApiSettings GetApiSettings()
    {
        return new AppConfigApiSettings();
    }

    public IFeedServiceSettings GetFeedServiceSettings()
    {
        return new AppConfigFeedServiceSettings();
    }
}

Similarly, DynamoDBConfigurationFactory is responsible for creating concrete classes that access DynamoDB values:

public class DynamoDBConfigurationFactory : IConfigurationFactory
{
    protected Table _configTable;
    
    public DynamoDBConfigurationFactory()
    {
        AmazonDynamoDBClient dynmamoClient = new AmazonDynamoDBClient("accessKey", "secretKey", RegionEndpoint.EUWest1);
        _configTable = Table.LoadTable(dynmamoClient, "tableName");
    }
    
    public IApiSettings GetApiSettings()
    {
        return new DynamoDBApiSettings(_configTable);
    }

    public IFeedServiceSettings GetFeedServiceSettings()
    {
        return new DynamoDBFeedServiceSettings(_configTable);
    }
}

Notice all the “concrete products” implement the same “abstract product” interface and hence they are interchangable. With the product classes in the picture the diagram now looks like this:

Finally let’s have a look at the concrete objects that carry out the actual job. For example the IApiSettings exposes 2 string properties:

public interface IApiSettings
{
    string ApiKey { get; }
    string ApiEndPoint { get; }
}

If we want to read these values from App.config it’s very straightforward:

public class AppConfigApiSettings : IApiSettings
{
    public string ApiKey
    {
        get { return ConfigurationManager.AppSettings["Rareburg.ApiKey"]; }
    }

    public string ApiEndPoint
    {
        get { return ConfigurationManager.AppSettings["Rareburg.ApiEndPoint"]; }
    }
}

The DynamoDB version is fairly more complex but it makes no difference from the consumer’s point of view. Here GetValue is a method in the base class that returns the value from the encapsulated Amazon.DynamoDBv2.DocumentModel.Table object.

public class DynamoDBApiSettings : DynamoDBSettingsBase, IApiSettings
{
    public DynamoDBApiSettings(Table configTable)
        : base (configTable)
    {
    }

    public string ApiKey
    {
        get { return GetValue("Rareburg.ApiKey"); }
    }

    public string ApiEndPoint
    {
        get { return GetValue("Rareburg.ApiEndPoint"); }
    }
}

The concrete factory is responsible for creating the concrete classes it uses. So the client is completely oblivious to the classes such as DynamoDBApiSettings or AppConfigApiSettings. This means we can add a whole new set of configuration classes (i.e. a web service) and all we have to change in the client code will be one line where we instantiate the concrete factory.

This approach also allows us to be more flexible with the concerete class implementations. For example DynamoDB config class family requires a Table object in their constructors. To avoid code repetition I derived them all from a base class and moved the table to the base but the that doesn’t change anything in the client code.

Resources

design, development, design patterns, csharp comments edit

I recently developed a toy project called Rareburg article feed generator. It basically gets the articles from Rareburg.com (a marketplace for collectors) and creates an RSS feed. One challenge I had was picking the feed formatter class (RSS vs Atom). I utilised Factory Method design pattern to solve that and this post discusses the usage of the pattern over that use case.

Use case: Creating RSS feed formatter

I wanted my feed generator to support both RSS and Atom feeds based on the value specified in the configuration. In order to pass the XML validations they needed to modified a bit and I wanted these details hidden from the client code.

What is Factory Method?

Here’s the official definition from GoF:

Define an interface for creating an object, but let subclasses decide which class to instantiate. Factory Method lets a class defer instantiation to subclasses

Factory Method pattern is one of the creational patterns which make it easy to separate object creation from the actual system. The actual business logic doesn’t need to care about these object creation details.

Implementation

To see the implementation at a glance let’s have a look at the class diagram:

The base abstract class is called ArticleFeedGenerator (it corresponds to Creator in the GoF book). Here’s a shortened version:

public abstract class ArticleFeedGenerator
{
    private SyndicationFeedFormatter _feedFormatter;

    // Factory method
    public abstract SyndicationFeedFormatter CreateFeedFormatter();

    public void Run()
    {
        var allArticles = _feedDataClient.GetAllArticles();
        _feed = _feedService.GetFeed(allArticles);

        _feedFormatter = CreateFeedFormatter();

        _publishService.Publish(_feedFormatter);
    }
}

ArticleFeedGenerator does all the work except creating a conccrete implementation of SyndicationFeedFormatter. It delegates it to the derived class who provide the implementation for CreateFeedFormatter abstract method.

And here comes the concrete implementations (which correspond to ConcreteCreator)

public class AtomFeedGenerator : ArticleFeedGenerator
{
    public AtomFeedGenerator(
        IFeedDataClient feedDataClient,
        IFeedService feedService,
        IPublishService publishService,
        IFeedSettings feedSettings)
        : base (feedDataClient, feedService, publishService, feedSettings)
    {
    }

    public override SyndicationFeedFormatter CreateFeedFormatter()
    {
        return new Atom10FeedFormatter(_feed);
    }
}

In order to pass RSS validations RSS formatter needed more “love” than the Atom version above:

public class RssFeedGenerator : ArticleFeedGenerator
{
    public RssFeedGenerator(
        IFeedDataClient feedDataClient,
        IFeedService feedService,
        IPublishService publishService,
        IFeedSettings feedSettings)
        : base (feedDataClient, feedService, publishService, feedSettings)
    {
    }
    
    public override SyndicationFeedFormatter CreateFeedFormatter()
    {
        var formatter = new Rss20FeedFormatter(_feed);
        formatter.SerializeExtensionsAsAtom = false;
        XNamespace atom = "http://www.w3.org/2005/Atom";
        _feed.AttributeExtensions.Add(new XmlQualifiedName("atom", XNamespace.Xmlns.NamespaceName), atom.NamespaceName);
        _feed.ElementExtensions.Add(new XElement(atom + "link", new XAttribute("href", _feedSettings.FeedUrl), new XAttribute("rel", "self"), new XAttribute("type", "application/rss+xml")));
        return formatter;
    }
}

Concerete creators create concrete products but the factory method returns the abstract product which ArticleFeedGenerator (abstract creator) works with.

To avoid hard-coding the class name, a helper method called CreateFeedGenerator is added. The client code calls this method to get either a AtomFeedGenerator or a RssFeedGenerator based on configuration value.

class Program
{
    static void Main(string[] args)
    {
        var feedSettings = new AppConfigFeedSettings();
        ArticleFeedGenerator feedGenerator = CreateFeedGenerator(feedSettings);
        feedGenerator.Run();
    }
    
    private static ArticleFeedGenerator CreateFeedGenerator(IFeedSettings feedSettings)
    {
        string feedFormat = feedSettings.FeedFormat;
        switch (feedFormat.ToLower())
        {
            case "atom": return new AtomFeedGenerator(new RareburgClient(), new RareburgArticleFeedService(), new S3PublishService(), feedSettings);
            case "rss": return new RssFeedGenerator(new RareburgClient(), new RareburgArticleFeedService(), new S3PublishService(), feedSettings);
            default: throw new ArgumentException("Unknown feed format");
        }
    }
}

Resources

design, development, design patterns comments edit

Often times we need to change some algorithm with another without changing the client code that’s consuming it. In this post I want to show a use case I came across and utilised Strategy pattern.

What is it?

Here’s the official definition of the pattern from GoF book:

Define a family of algorithms, encapsulate each one, and make them interchangeable. Strategy lets the algorithm vary independently from clients that use it.

Use case: Get external IP address

In an application I was working on I needed to get the external IP address of the computer the application is running on. There are various ways to achieve that. This looked like a good opprtunity to use the Strategy pattern as I wanted to be able switch between the different methods easily.

Implementation

The interface is quite simple:

public interface IIpCheckStrategy
{
    string GetExternalIp();
}

Some services return their data in JSON format, some have extra text in it. But encapsulating the algorithms in their own classes this way, the clint code doesn’t have to worry about parsing these various return values. It’s handled in each class. If one service changes it’s output and breaks the implementation I can recover just by changing the code instantiates the class.

The concrete implementations of the interface are below. They implement the IIpCheckStrategy and are responsible for getting the data and return parsed IP address as string.

AWS IP Checker:

public class AwsIPCheckStrategy : IIpCheckStrategy
{
    public string GetExternalIp()
    {
        using (var client = new HttpClient())
        {
            client.BaseAddress = new Uri("http://checkip.amazonaws.com/");
            string result = client.GetStringAsync("").Result;
            return result.TrimEnd('\n');
        }
    }
}

DynDns IP Checker:

public class DynDnsIPCheckStrategy : IIpCheckStrategy
{
    public string GetExternalIp()
    {
        using (var client = new HttpClient())
        {
            client.BaseAddress = new Uri("http://checkip.dyndns.org/");
            HttpResponseMessage response = client.GetAsync("").Result;
            return HelperMethods.ExtractIPAddress(response.Content.ReadAsStringAsync().Result);
        }
    }
}

Custom IP Checker:

public class CustomIpCheckStrategy : IIpCheckStrategy
{
    public string GetExternalIp()
    {
        using (var client = new HttpClient())
        {
            client.BaseAddress = new Uri("http://check-ip.herokuapp.com/");
            client.DefaultRequestHeaders.Accept.Clear();
            client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));

            HttpResponseMessage response = client.GetAsync("").Result;
            string json = response.Content.ReadAsStringAsync().Result;
            dynamic ip = Newtonsoft.Json.JsonConvert.DeserializeObject(json);
            string result = ip.ipAddress;
            return result;
        }
    }
}

Choosing the algorithm

The consumer of the algorithm can pick any class that implements IIpCheckStrategy and switch between them. For example:

class StrategyClient1
{
    public void Execute()
    {
        IIpCheckStrategy ipChecker;

        ipChecker = new DynDnsIPCheckStrategy();
        Console.WriteLine(ipChecker.GetExternalIp());

        ipChecker = new AwsIPCheckStrategy();
        Console.WriteLine(ipChecker.GetExternalIp());

        ipChecker = new CustomIpCheckStrategy();
        Console.WriteLine(ipChecker.GetExternalIp());

        Console.ReadKey();
    }
}

Also the class name to be used can be stored in the configuration in some cases so that it can be changed at runtime without recompiling the application. For instance:

class StrategyClient2
{
    public void Execute()
    {
        string ipcheckerTypeName = ConfigurationManager.AppSettings["IPChecker"];

        IIpCheckStrategy ipchecker = Assembly.GetExecutingAssembly().CreateInstance(ipcheckerTypeName) as IIpCheckStrategy;

        Console.WriteLine(ipchecker.GetExternalIp());
    }
}

and the appSettings in the configuration would look like this:

<appSettings>
	<add key="IPChecker" value="Strategy.AwsIPCheckStrategy"/>
</appSettings>

Resources

development, aws, lambda comments edit

Just two days ago I was jumping through a lot of hoops (maintain state on S3, subscribe to public SNS topics) just to schedule a Lambda function, as explained here. If I knew all these problems were to be solved in a day, I would just wait! In re-Invent 2015 event AWS announced scheduled event support for Lambda functions.

After tweaking around a little bit with the sample I set my schedule like this which will run first day of every month at 10:00:

cron(0 10 1 * ? *)

That’s all there is to it for a monthly schedule. Reliable and no need for my state management workarounds. So the final code becomes as simple as sending an emali when invoked:

Resources

development, aws, lambda, s3, sns comments edit

UPDATE: Yesterday (October 8th, 2015) Amazon announced official support for scheduled events so I updated my function to use this feature. For the most up-to-date version of this project please visit the updated version

I have a great accountant but he has one flaw: I have to ask for the invoice every month! While waiting for him to automate the process, I decided to automate what I can on my end. There are many ways to skin a cat, as the saying goes, the way I picked for this task was developing an AWS Lambda funciton and trigger it by subscribing to a public SNS topic.

Step 1: Prepare a function to send emails

Developing a simple node.js function that sends E-mails was simple. First I needed the install two modules:

npm install nodemailer
npm install nodemailer-smtp-transport

And the function is straightforward:

var transporter = nodemailer.createTransport(smtpTransport({
    host: 'email-smtp.eu-west-1.amazonaws.com',
    port: 587,
    auth: {
        user: '{ACCESS KEY}',
        pass: '{SECRET KEY}'
    }
}));

var text = 'Hi, Invoice! Thanks!';

var mailOptions = {
    from: 'from@me.net',
    to: 'to@someone.net',
    bcc: 'me2@me.com',
    subject: 'Invoice',
    text: text 
};

transporter.sendMail(mailOptions, function(error, info){
      if(error){
		console.log(error);
      }else{
		console.log('Message sent');
      }
  });

The challenge was deployment as the script had some dependencies. If you choose Edit Inline and just paste the script you would get an error like this

"errorMessage": "Cannot find module 'nodemailer'",

But it’s very easy to deploy a full package with dependencies. Just zip everything in the folder (wihtout the folder itself) and upload the zip file. The downside of this method is you can no longer edit the code inline. So even just for fixing a trivial typo you need to re-zip and re-upload.

Step 2: Schedule the process

One simple method to schedule the process is to invoke the method using Powershell and schedule a task to run the script:

Invoke-LMFunction -FunctionName automatedEmails -AccessKey accessKey -SecretKey secretKey -Region eu-west-1

But I don’t want a dependency on any machine (local or EC2 instance). Otherwise I could write a few lines of code in C# to do the same job anyway. The idea of using Lambda is to avoid maintenance and let everything run on infrastructure that’s maintained by AWS.

Unreliable Town Clock

Unfortunately AWS doesn’t provide an easy method to schedule Lambda function invocations. For the sake of simplicity I decided to use Unreliable Town Clock (UTC) which is essentially a public SNS topic that sends “chime” messages every 15 minutes.

Since all I need is one email, I don’t care if it skips a beat or two as long as it chimes at least once throughout the day.

State management

Of course to avoid bombarding my accountant with emails I have to maintain a state so that I would only send one email per month. But Lambda functions must be stateless. Some alternatives are using AWS S3 or DynamoDB. Since all I need is one simple integer value I decided to store in a text file on S3. So first I download the log file and check the last sent email month:

function downloadLog(next) {
	s3.getObject({
			Bucket: bucketName,
			Key: fileName
		},
		next);

function checkDate(response, next) {
	var currentDay = parseInt(event.Records[0].Sns.Message.day);
	currentMonth = parseInt(event.Records[0].Sns.Message.month);
	var lastMailMonth = parseInt(response.Body.toString());
	if (isNaN(lastMailMonth)) {
		lastMailMonth = currentMonth - 1;
	}
	if ((currentDay == targetDayOfMonth) && (currentMonth > lastMailMonth)) {
		next();
	}
}

Putting it together

So putting it all together the final code is:

Let’s see if it’s going to help me get my invoices automatically!

Conclusion

  • A better approach would be to check emails for the invoice and send only if it wasn’t received already. Also a copule of reminders after the initial email would be nice. But as my new resolution is to progress in small, incremental steps I’ll call it version 1.0 and leave the remaining tasks for a later version.

  • My main goal was to achieve this task without having to worry about the infrastructure. I still don’t but that’s only because a nice guy (namely Eric Hammond) decided to setup a public service for the rest of us.

  • During my research I came across a few references saying that the same task can be done using AWS Simple Workflow (SWF). I haven’t used this service before. Looked complicated and felt like there is a steep learning curve to go through. In Version 2 I should look into SWF which would…

    • allow me to handle a complex workflow
    • make dependency to public SNS topic redundant
    • handle state properly

Resources

csharp, development, WPF comments edit

I like scanning all my documents and keep a digital version as well as the dead-tree version just in case. But storing the documents as individual image files is too messy. There are various solutions to merge image files into a single PDF but I don’t want to upload my sensitive documents to unknown parties so I rolled my own: Image2PDF

Implementation

I decided to go with a WPF application rathen than a web-based solution because it would be unnecessary to upload a bunch of image files to cloud and download the PDF back. It’s a lot of network traffic for something that can be handled locally much faster.

Image2PDF a simple WPF application developed in C#. I’ve used iTextSharp library to create the PDFs. It’s a great open-source library with lots of capabilities.

I’ve also started using SyncFusion WPF control library recently. The entire control suite is free of charge (which is always a great price!). It has a Wizard control which I decided to go with. I think using a wizard and is cleaner UI instead of dumping every control on one window.

Usage

As you might expect you provide the list of input images, re-order them if needed, choose the path for the output folder and go!

Step 1: Select input

Page 1: Specify input image files

Step 2: Select output

Page 2: Specify output PDF path

Step 3: Go!

Page 3: Run!

Simples!

Conclusion

In the future I’m planning to add various PDF operarions and enhance the functionality but since shipping is a feature I’ll limit the scope for now. After all, this was the main feature I needed anyway.

Resources

csharp, neo4j, cypher, json, development comments edit

I have a simple JSON file that contained a bunch of users with their followers which looked like this:

  {
    "user-id": 2,
    "username": "user_2",
    "avatar": "URL",
    "following": [
      "user_10",
      "user_6",
      "user_1"
    ],
    "followers": [
      "user_10"
    ]
  }

It felt like a good exercise would be to import that data into a graph database as it wasn’t something I had done before.

As my go-to language is C# and I had some experience with Cypher before, my initial instinct was to develop a tool to generate Cypher statements from the JSON.

First I create the nodes:

private void GenerateNodesCypher()
{
    string filename = @"..\..\output\nodes.cql";
    var output = new StringBuilder();

    foreach (var user in _userList)
    {
        string s = $"CREATE ({user.username}:User {{ userid: {user.userid} , username: '{user.username}', avatar: '{user.avatar}'  }} ); ";
        output.AppendLine(s);
    }

    File.WriteAllText(filename, output.ToString());
}

and then the relationships:

private void GenerateRelationshipsCypher()
{
    string filename = @"..\..\output\relationships.cql";
    var output = new StringBuilder();
    int n = 0;
    foreach (var user in _userList)
    {
        foreach (var following in user.following)
        {
            string s = $"MATCH (a), (b) WHERE a.username = '{user.username}' AND b.username = '{following}' CREATE (a)-[:FOLLOWING]->(b); ";
            output.AppendLine(s);
            n++;
        }
    }

    File.WriteAllText(filename, output.ToString());
}

So I ended up with two cql files that looked like this

CREATE (user_1:User { userid: 1 , username: 'user_1', avatar: 'URL' }); 
CREATE (user_2:User { userid: 2 , username: 'user_2', avatar: 'URL' }); 
CREATE (user_3:User { userid: 3 , username: 'user_3', avatar: 'URL' }); 

and

MATCH (a), (b) WHERE a.username = 'user_1' AND b.username = 'user_26' CREATE (a)-[:FOLLOWING]->(b); 
MATCH (a), (b) WHERE a.username = 'user_1' AND b.username = 'user_53' CREATE (a)-[:FOLLOWING]->(b); 
MATCH (a), (b) WHERE a.username = 'user_2' AND b.username = 'user_6' CREATE (a)-[:FOLLOWING]->(b); 

and I used neo4j-shell to execute the files and import the data. But it was no bed of roses. I’ll list the problems I faced along the way and how I got around them so that this experience might be helpful for other people as well:

Issues along the way and lessons learned

Running multiple statements on Neo4J browser

First I tried to run the create statements using the Neo4J browser which turned out to be problematic because it cannot run multiple statements that end with semi-colons. So I removed the semi-colons but then it started giving me this error

WITH is required between CREATE and MATCH

I found a workaround for that on SO. So the following works:

MATCH (a), (b) WHERE a.username = 'user_1' AND b.username = 'user_14' CREATE (a)-[:FOLLOWING]->(b); 
WITH 1 as dummy
MATCH (a), (b) WHERE a.username = 'user_1' AND b.username = 'user_22' CREATE (a)-[:FOLLOWING]->(b); 

Now the problem was, if the data was a bit dirty, for instance if user_14 didn’t exist, it stopped executing the rest and no other relationships were created). I had a few nodes like that so this method didn’t work for me after all.

Starting the shell was not as easy as I’d imagined

I installed Neo4J using the default settings and to start the shell just navigated to that directory and ran the batch file. Got an error instead of my shell:

C:\Program Files\Neo4j Community\bin>Neo4jShell.bat

The system cannot find the path specified.

Error: Could not find or load main class org.neo4j.shell.StartClient

Apparently the way to run the shell is using the Neo4J server application and clicking Options -> Command Prompt

Neo4J command prompt

This launches the Neo4J Command Prompt then the rest is easy:

neo4jshell -file nodes.cql

neo4jshell -file relationships.cql

“Exiting with unterminated multi-line input”

Final issue was the statements in a sql file must be terminated with a semi-colon in order for the shell to execute them. Otherwise it gives the above warning and quits

So after all this hassle my data is in the Neo4J database ready to be queried to death!:

Next I’ll investigate converting the data into CSV and achieve the same results by using LOAD CSV command and look into visualization of this data.

Resources

csharp, development, asp.net, mvc comments edit

In ASP.NET, TempData is one of the mechanisms used to pass data from controller to the view. In this post I’ll dive into its source code to investigate its behaviour.

What is TempData

TempData is an instance of TempDataDictionary that is used to pass data from the controller to the view.

Lifespan

The lifespan of TempData is rather unusual: It lives for one request only. In order to achieve this it maintains two HashSets to manage keys as well as the data dictionary:

private Dictionary<string, object> _data;
private HashSet<string> _initialKeys = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
private HashSet<string> _retainedKeys = new HashSet<string>(StringComparer.OrdinalIgnoreCase);

When we read some data using an indexer or TryGetValue method it removes that key from _initalKeys collection.

public bool TryGetValue(string key, out object value)
{
    _initialKeys.Remove(key);
    return _data.TryGetValue(key, out value);
}

The actual dictionary that holds the data is intact at this point. That’s why we can read same data consecutively without any issues. It only removes the key from _initialKeys collection, basically marking it to be deleted when the data is persisted.

Peek and keep

If we want the values in TempData last longer we can use Peek and Keep methods. What Peek does is return the value without removing it from the _initialKeys:

public object Peek(string key)
{
    object value;
    _data.TryGetValue(key, out value);
    return value;
}

Alternatively, we can call Keep method. Similarly it doesn’t manipulate the data directly but just marks the key to be persisted by adding it to the _retainedKeys collection.

public void Keep(string key)
{
    _retainedKeys.Add(key);
}

Parameterless overload of Keep method add all keys in the _data dictionary to _retainedKeys.

Which keys to persist

So as seen above when we get values and call Peek/Keep methods, operations are carried out on _initialKeys and _retainedKeys collections and nothing happens to the actual data. These operations take effect when the _data is actually saved:

public void Save(ControllerContext controllerContext, ITempDataProvider tempDataProvider)
{
    _data.RemoveFromDictionary((KeyValuePair<string, object> entry, TempDataDictionary tempData) =>
        {
            string key = entry.Key;
            return !tempData._initialKeys.Contains(key) 
                && !tempData._retainedKeys.Contains(key);
        }, this);

    tempDataProvider.SaveTempData(controllerContext, _data);
}

Before the data is passed on to the provider it’s pruned. The keys that don’t exist in the _retainedKeys (the keys we explicitly told to keep) and _initialKeys (the keys that have not been touched so far or accessed through Peek method) collections are removed.

Providers

By default, TempData uses session variables to persist data from one request to the next. Serializing and deserializing data is carried out via an object implementing ITempDataProvider. By default SessionStateTempDataProvider class is used to provide this functionality. It occurs in the CreateTempDataProvider method in Controller.cs class:

protected virtual ITempDataProvider CreateTempDataProvider()
{
    return Resolver.GetService<ITempDataProvider>() ?? new SessionStateTempDataProvider();
}

This also means we can replace the provider with our own custom class. For demonstration purposes I wrote my own provider which uses a simple text file to persist TempData:

public class TextFileTempDataProvider : ITempDataProvider
{
    internal readonly string FileName = Path.Combine(HttpContext.Current.Server.MapPath(@"~/App_Data"), @"TempData.txt");

    public IDictionary<string, object> LoadTempData(ControllerContext controllerContext)
    {
        if (File.Exists(FileName))
        {
            string json = File.ReadAllText(FileName);
            return Newtonsoft.Json.JsonConvert.DeserializeObject<Dictionary<string, object>>(json);
        }

        return new Dictionary<string, object>(StringComparer.OrdinalIgnoreCase);
    }
    
    public void SaveTempData(ControllerContext controllerContext, IDictionary<string, object> values)
    {
        string json = Newtonsoft.Json.JsonConvert.SerializeObject(values);
        File.WriteAllText(FileName, json);
    }
}

In order to use this class it needs to be assigned to TempDataProvider in the controller’s constructor

public FirstController()
{
    TempDataProvider = new TextFileTempDataProvider();
}

Of course it’s not a bright idea to use disk for such operations, this is just for demonstration purposes and makes it easier to observe the behaviour.

Conclusion

Often times I’ve found knowledge about the internals of a construct useful. Applications and frameworks are getting more complex each day, adding more layers and hiding the complexity from the consumers. It’s great because we can focus on the actual business logic and application we are building but when we get stuck it takes quite a while to figure out what’s going on. Having in-depth knowledge on the internals can save a lot if time.

Resources

tfs, team foundation server, ci, continuous integration, alm, application lifecycle management comments edit

Setting up continuous integration environment with TFS is quite easy and free. In this post I’ll go over the details of setting up a CI environment. The tools I will use are:

  • Visual Studio 2015 Community Edition
  • TFS 2015 Express
  • AWS EC2 instance (To install TFS)
  • AWS SES (To send notification mails)

Step 01: Download and install TFS Express 2015

I always prefer the offline installer so download the whole ISO just in case (it’s 891MB so it might be a good time for a coffee break!)

Installation is standard next-next-finish so nothing particular about it. Accepting all defaults yields a working TFS instance.

Step 02: Connect to TFS

From the Team menu select Manage Connections. This will open Team Explorer on which we can choose the TFS instance.

It can connect to TFS Online or Github. For this example I will use the hosted TFS we have just installed. We first need to add it to TFS server list.

If you are using EC2 like I did don’t forget to allow inbound traffic on port 8080 beforehand.

Note that we don’t have a Team Project yet, what we have connected is the Project Collection. A collection is an additional abstraction layer used to group related projects. Using Default Collection generally works out just fine for me.

Step 03: Configure workspace

Since a local copy needs to be stored locally we have to map it to the project

Just pick a local empty folder. Leaving “$/” means the entire collection will be mapped to this folder.

Click Map & Get and you will have a blank workspace.

Step 04: Create team project

Now we can create a team project on TFS. click on Home -> Projects and My Teams then New Team Project.

Just following the wizard selects Scrum methodology and Team Foundation Version Control as the source control system. Starting with TFS 2013, Git is also supported now.

After several minutes later, our team project is ready.

Step 05: Clone and start developing

I chose Git as the version control system. If you use TFSVC the terminology you’ll see is a bit different but since the main focus of this post is establishing continuous integration it doesn’t matter much as long you as you can check-in the source code.

So now that we have a blank canvas let’s start painting! I added a class library with a single class like the one below:

public class ComplexMathClass
{
    public int Add(int x, int y)
    {
        return x + y;
    }

    public double Divide(int x, int y)
    {
        return x / y;
    }
}

and 3 unit test projects (NUnit, XUnit and MSTest).

NUnit tests:

public class ComplexMathClassTests
{
    [TestCase(1, 2, ExpectedResult = 3)]
    [TestCase(0, 5, ExpectedResult = 5)]
    [TestCase(-1, 1, ExpectedResult = 0)]
    public int Add_TwoIntegers_ShouldCalculateCorrectly(int x, int y)
    {
        var cmc = new ComplexMathClass();
        return cmc.Add(x, y);
    }

    [Test]
    // [ExpectedException(typeof(DivideByZeroException))]
    public void Divide_DenominatorZero_ShouldThrowDivideByZeroException()
    {
        var cmc = new ComplexMathClass();
        double result = cmc.Divide(5, 0);
    }
}

XUnit tests:

public class ComplexMathClassTests
{
    [Theory]
    [InlineData(1, 2, 3)]
    [InlineData(0, 5, 5)]
    [InlineData(-1, 1, 0)]
    public void Add_TwoIntegers_ShouldCalculateCorrectly(int x, int y, int expectedResult)
    {
        var cmc = new ComplexMathClass();
        int actualResult = cmc.Add(x, y);
        Assert.Equal<int>(expectedResult, actualResult);
    }

    [Fact]
    public void Divide_DenominatorZero_ShouldThrowDivideByZeroException()
    {
        var cmc = new ComplexMathClass();
        cmc.Divide(5, 0);
    //  Assert.Throws<DivideByZeroException>(() => cmc.Divide(5, 0));
    }
}

MSTest tests:

[TestClass]
public class ComplexMathClassTests
{
    [TestMethod]
    // [ExpectedException(typeof(DivideByZeroException))]
    public void Divide_DenominatorZero_ShouldThrowDivideByZeroException()
    {
        var cmc = new ComplexMathClass();
        cmc.Divide(5, 0);
    }
}

Unfortunately MSTest still doesn’t support parametrized tests which is a shame IMHO. That’s why I was never a big fan of it but added to this project for the sake of completeness.

I commented out the lines that expect exception in all tests to fail the tests. So now the setup is complete: We have a working project with a failing test. Our goal is to get alert notifications about the failing test whenever we check in our code. Let’s proceed to the next steps to accomplish this.

Step 06: Building on the server

As of TFS 2015 they renamed the Build Configuration to XAML Build Configuration for some reason and moved it under Additional Tools and Components node but everything else seems to be the same.

Default configuration installs one build controller and one agent for the Default Collection so for this project we don’t have to do add or change anything.

Each build controller is dedicated to a team project collection. Controller performs lightweight tasks such as determining the name and reporting the status of the build. Agents are the heavy-lifters and carry out processor-intensive work of the build process. Each agent is controlled by a single controller.

In order to build our project on the build server we need a build definition first. We can create one by selecting Build -> New build Definition

  • One important setting is the trigger: By default the build is not triggered automatically which means soon enough it will wither and die all forgotten! To automate the process we have to select Continuous Integration option.

  • In order to automatically trigger the build, the branch that is to be built must be added to monitored branches list in the Source Settings tab.

  • Last required setting is failing the build when a test fails. It’s a bit buried so you have to go 3.1.1 Test Source and set “Fail build on test failure” to true.

Step 07: Notifications

At this point, our build definition is triggered automatically but we don’t get any notifications if the build fails (due to a compilation error for example). TFS comes with an application called Build Notifications. It pops up an alert but it requires to be installed on the developer machine so I don’t like this solution.

A better approach is enabling E-Mail notifications. In order to do that first we need to set up Email Alert Settings for TFS. In the Team Foundation Server Express Administration Console select Application Tier and scroll down to the “Email Alert Settings” section. enter the SMTP credentials and server info here.

You can also send a test email to verify your settings.

Second and final stage is to enable the project-based alerts. In Visual Studio Team Explorer window select Home -> Settings -> Project Alerts. This will pop up a browser and redirect to alert management page. Here select “Advanced Alert Management Page”. In the advanced settings page there are a few pre-set notification conditions and the build failure is at the top of the list!

I intentionally broke the build and checked in my code and in a minute, as expected I received the following email:

With that we have automated the notifications. We already set the build to fail upon test failure in the previous step. Final step is to run the tests on the build server to complete the circuit.

Step 08: Run tests on build server

Running tests on the server is very simple. For NUnit all we have to do is install the NUnitTestAdapter package:

Install-Package NUnitTestAdapter

After I committed and pushed my code with the failing test I got the notification in a minute:

Uncommented the ExpectedException line and the build succeeded.

For xUnit the solution is similar, just install the xUnit runner NuGet package and checkin the code

Install-Package xunit.runner.visualstudio

For MSTest it works out of the box so you don’t have to install anything.

In the previous version of TFS I had to install Visual Studio on the build server as advised in this MSDN article. Seems like in TFS 2015 you don’t have to do that. The only benefit of using MSTest (as far as I can see at least) is that it’s baked in so you don’t have to install extra packages. You create a Unit Test project and all your tests are immediately visible in the test explorer and automatically run on the server.

Wrapping up

Continuous Integration is a crucial part of the development process. Team Foundation Server is a powerful tool with lots of features. In this post I tried to cover basics from installation to setting up a basic CI environment. I’ll try to cover more features in the future but my first priority will be new AWS services CodeDeploy, CodeCommit and CodePipeline. As I’m trying to migrate everything to AWS having all projects hosted and managed on AWS would make more sense in my case.

Resources

aws, route 53, csharp, development comments edit

If you don’t have a static IP and you need to access your home network for some reason you may find Dynamic DNS services useful. I was using No-IP until its limitations became offputting. Since I love DIY stuff and AWS I thought I could build the exact same service on my own using Route 53.

Enter DynDns53

It’s a simple application. What it does is basically:

  • Get current external IP
  • For each subdomain in the configuration, call AWS API and get domain info
  • Check if the recorded IP is different than the current one
  • If so, call AWS API to update it with the new one, if not do nothing.

In a different post on setting up Calibre on a Raspberry Pi I developed a script that did the same thing:

import boto.route53
import urllib2
currentIP = urllib2.urlopen("http://checkip.amazonaws.com/").read()

conn = boto.connect_route53()
zone = conn.get_zone("volki.info.")
change_set = boto.route53.record.ResourceRecordSets(conn, '{HOSTED_ZONE_ID}')

for rrset in conn.get_all_rrsets(zone.id):
    if rrset.name == 'calibre.volki.info.':
        u = change_set.add_change("UPSERT", rrset.name, rrset.type, ttl=60)
        rrset.resource_records[0] = currentIP
        u.add_value(rrset.resource_records[0])
        results = change_set.commit()

The difference of DynDns53 is that it can run as a background service and it has a cute logo! The C# equivalent of above code is like this:

public void Update()
{
    var config = _configHandler.GetConfig();
    string currentExternalIp = _ipChecker.GetExternalIp();

    foreach (var domain in config.DomainList)
    {
        string subdomain = domain.DomainName;
        string zoneId = domain.ZoneId;

        var listResourceRecordSetsResponse = _amazonClient.ListResourceRecordSets(new ListResourceRecordSetsRequest() { HostedZoneId = zoneId });
        var resourceRecordSet = listResourceRecordSetsResponse.ResourceRecordSets.First(recordset => recordset.Name == subdomain);
        var resourceRecord = resourceRecordSet.ResourceRecords.First();

        if (resourceRecord.Value != currentExternalIp)
        {
            resourceRecord.Value = currentExternalIp;

            _amazonClient.ChangeResourceRecordSets(new ChangeResourceRecordSetsRequest()
            {
                HostedZoneId = zoneId,
                ChangeBatch = new ChangeBatch()
                {
                    Changes = new List<Change>() 
                    { 
                        new Change(ChangeAction.UPSERT, resourceRecordSet)
                    }
                }
            });
        }
    }
}

That’s it. The rest is just user interface, a Windows service and some basic uni tests.

Usage

The project has a WPF user interface and a Windows service. Both are using the same library (DynDns53.Core) so the functionality is the same. Using the user interface is pretty straightforward.

Just add your AWS keys that have access privileges to your domains. You can set it to run at system start so you don’t have to start it manually.

I built the Windows service using Topshelf. To install it, build the application and on an elevated command prompt just run

DynDns53.Service.exe install

Similarly to uninstall:

DynDns53.Service.exe uninstall

What’s missing

  • Landing page: I’ll update the project details on the site generated by GitHub Pages.
  • Mac/Linux support: As the worlds are colliding I’d like to discover what I can do with the new .NET Core runtime. Since this is a simple project I think it’s a good place to start

Resources