Matthew Jones's Picture

Matthew Jones

I'm an ASP.NET and Microsoft-stack developer who loves teaching and programming.

Creating a Post Archive with the Ghost API and jQuery

I've long been missing an important feature in Ghost, my blog publishing platform: there's no inherent feature to create a post archive, or a list of all my posts in one place. I've gotten several requests for this feature, so I finally decided to just sit down and develop it using the Ghost Public API and a tiny bit of jQuery.

What follows is how I built my post archive the first time around. I have since replaced it with a Ghost-generated structure using the #get helper to solve some caching issues, but I like the look of the following system better. Please note that this post was written using Version 0.11.4 of Ghost and so it may change as Ghost changes.

Using jQuery and Ghost

Setting Up the Ghost Page Template

The first problem I had was that this page (the archive page) was not going to be either a post or a static page using my theme's default template. It was going to be a page, but it needed its own template, one that was very stripped-down.

Ghost supports using custom page templates by having a .hbs file that is prefixed with "page-". So, in my Ghost theme's root folder, I now have a file called page-all-posts.hbs, which looks like the following:

{{!< default}}

{{! This is a page template. A page outputs content just like any other post, and has all the same
    attributes by default, but you can also customise it to behave differently if you prefer. }}
{{#post}}
    <header class="post-header">
        <h1 class="post-title">{{title}}</h1>
    </header>
    <section class="post-content">
        {{content}}    
        <div id="postLoading">
            <h3>Loading post archive...</h3>
        </div>
        <div id="postList"></div>
    </section>
{{/post}}

The template is pretty darn simple. It has the "normal" Ghost stuff, like the {{#post}} tag, as well as the page title and content. The difference is what happens at the end: the div "postList" is where we will populate the list of all posts.

However, just having the template doesn't help us; I also need to create a static page with the same url as the name of the page template file. In my Ghost admin window, I created a new post, marked it as a static page, and gave it the URL "all-posts".

With that template and page set up, we can now write the jQuery to get us the actual posts.

Querying for All Posts

The first thing we have to do in order to display all posts is to get all the posts. I've already blogged about something like this when I implemented the 5 random posts sidebar, and so this solution will be very similar.

Ghost exposes a public API that can be used to query posts, users, and tags. In this case, I only want posts and I specifically want all my posts, so my query is very easy:

$(document).ready(function () {
    $.get(
        ghost.url.api('posts', {limit: 'all'})
    ).done(onSuccess);
});

The onSuccess method is really just a pass-through to another method, called showArchive:

function onSuccess(data) {  
    showArchive(data.posts);
    .... //Here's the code that does the 5 random posts sidebar
}

The real tricks begin when we implement the showArchive method.

Displaying the Posts

When we query for posts using the Ghost API, the posts object which is returned looks like this (I have simplified this schema):

"posts":[{
    "id":1,
    "title":"Welcome to Ghost",
    "slug":"welcome-to-ghost",
    "markdown":"...",
    "html":"...",
    "status":"published",
    "created_at":"2014-11-17T19:02:27.147Z",
    "created_by":1,
    "updated_at":"2014-11-17T19:02:27.147Z",
    "updated_by":1,
    "published_at":"2014-11-17T19:02:27.173Z",
    "published_by":1,
    "author":1,
    "url":"/welcome-to-ghost/"
  }, {
    "id":2,
    "uuid":"ac0a0374-a43c-15c4-391b-128d6bbba7c5",
    "title":"Lorem Ipsum Dolor",
    "slug":"lorem-ipsum-dolor",
    "markdown":"...",
    "html":"...",
    "status":"published",
    "created_at":"2014-11-18T19:02:27.147Z",
    "created_by":1,
    "updated_at":"2014-11-18T19:02:27.147Z",
    "updated_by":1,
    "published_at":"2014-11-18T19:02:27.173Z",
    "published_by":1,
    "author":1,
    "url":"/lorem-ipsum-dolor/"
  }],

From this, I can use the published_at property to get the month and year each post was published.

NOTE: The query we executed earlier already loads the posts in most-recent-first order, so we don't need to do any further ordering.

So, here' the outline of what we need the showArchive function to do:

  1. For each post, get the month and year that post was published.
  2. Each time the month or year changes, output a new subheader for that month and year.
  3. For each post, output a link to that post.

Here's the complete function:

function showArchive(posts) {  
    var monthNames = ["January", "February", "March", "April", "May", "June",
      "July", "August", "September", "October", "November", "December"
    ];
    var currentMonth = -1;
    var currentYear = -1;
    if(window.location.pathname == "/your-archive-page-url/"){ //Only display on this page
        $(posts).each(function(index,value){ //For each post 
            var datePublished = new Date(value.published_at); //Convert the string to a JS Date
            var postMonth = datePublished.getMonth();  //Get the month (as an integer)
            var postYear = datePublished.getFullYear(); //Get the 4-digit year

            if(postMonth != currentMonth || postYear != currentYear)
            { //If the current post's month and year are not the current stored month and year, set them
                currentMonth = postMonth;
                currentYear = postYear;
                //Then show a month/year header
                $("#postList").append("<br><span><strong>" + monthNames[currentMonth] + " " + currentYear + "</strong></span><br>");
            }
            //For every post, display a link.
            $("#postList").append("<span><a href='" + value.url +"'>" + value.title + "</a></span><br>");
        });
    }
}

I fully realize that this is not the best HTML (or Javascript, really), but it suits my purposes for now. Here's a screenshot of what this looks like on my blog:

Woohoo! I've got a working solution to show all my posts! Which would be great...except this isn't what I'm actually using now.

Using Ghost Only

The problem is that I use CloudFlare on this blog, and when doing so CloudFlare caches this page before the script has a chance to run. This results in an empty page, which is obviously not what I want. So, instead, I ended up going with a native Ghost solution, which looks something like this:

<div id="postList">  
    {{#get "posts" limit="all"}}
        {{#foreach posts}}
            {{#if featured}}
                 <span>
                     <span class='fa fa-star'></span>  
                     {{date published_at format="MMM DD, YYYY"}}:&nbsp;<a href="{{url}}">{{title}} </a>
                 </span>
                 <br>
            {{else}}
                <span>
                    {{date published_at format="MMM DD, YYYY"}}:  <a href="{{url}}">{{title}}</a>
                </span>
                <br>
                    {{/if}}
                {{/foreach}}
            {{/get}}
        </div>

Here's a little breakdown of what this does:

  • The {{get}} helper gets me all my posts.
  • Within the {{get}} context, the {{foreach}} helper loops through each post.
  • Within the {{foreach}} context, the {{if}} helper enables me to check if a particular post is featured, and if so, output a star icon.
  • The {{date}} helper allows me to get the post's publishing date and format it.
  • Finally, the {{url}} helper and the {{title}} helper output the post's URL and title respectively.

Here's how the native Ghost solution looks:

The major downside to the native Ghost solution is that I no longer have the month and year section headers, something I would rather like to have. But, the page is no longer cached by CloudFlare, so it works. Further, this solution is much cleaner than the jQuery solution.

At any rate, now I've got a working page that lists all my blog posts! Check it out!

Summary

Please let me know if you found this post useful, and share other interesting tips about Ghost in the comments! Check out the Ghost documentation to learn about the other helpers they have available.

Happy Coding!

Mapping DataTables and DataRows to Objects in C# and .NET

My group regularly uses DataSet, DataTable, and DataRow objects in many of our apps.

(What? Don't look at me like that. These apps are old.)

Anyway, we're simultaneously trying to implement good C# and object-oriented programming principles while maintaining these old apps, so we often end up having to map data from a data set to a C# object. We did this enough times that I and a coworker (we'll call her Marlena) decided to sit down and just make up a new mapping system for use with these DataTable and DataRow objects.

As always with my code-based posts, there's a GitHub project with a full working example app, so check that out too!

One Jump Ahead

So here's a basic problem with mapping from DataSet, DataTable, and DataRow objects: we don't know at compile time what columns and tables exist in the set, so mapping solutions like AutoMapper won't work for this scenario. Our mapping system will have to assume what columns exist. But, in order to make it more reusable, we will make the mapping system return default values for any values which it does not locate.

There's also another, more complex problem: the databases we are acquiring our data from use many different column names to represent the same data. 20 years of different maintainers and little in the way of cohesive naming standards will do that do a database. So, if we needed a person's first name, the different databases might use:

  • first_name
  • firstName
  • fname
  • name_first

This, as might be imagined, makes mapping anything rather difficult. So our system will also need to be able to map from many different column names.

Finally, this system wouldn't be worth much if it couldn't handle collections of objects as well as single objects, so we'll need to allow for that as well.

So, in short, our system needs to:

  1. Map from DataTable and DataRow to objects.
  2. Map from multiple different column names.
  3. Handle mapping to a collection of objects as well as a single object.

We'll need several pieces to accomplish this. But before we can even start building the mapping system, we must first acquire some sample data.

Mine, Mine, Mine

We're going to create some DataSet objects that we can test our system against. In the real world, you would use an actual database, but here (for simplicity's sake) we're just going to manually create some DataSet objects. Here's a sample class which will create two DataSet objects, Priests and Ranchers, each of which use different column names for the same data:

public static class DataSetGenerator  
{
    public static DataSet Priests()
    {
        DataTable priestsDataTable = new DataTable();
        priestsDataTable.Columns.Add(new DataColumn()
        {
            ColumnName = "first_name",
            DataType = typeof(string)
        });
        priestsDataTable.Columns.Add(new DataColumn()
        {
            ColumnName = "last_name",
            DataType = typeof(string)
        });
        priestsDataTable.Columns.Add(new DataColumn()
        {
            ColumnName = "dob",
            DataType = typeof(DateTime)
        });
        priestsDataTable.Columns.Add(new DataColumn()
        {
            ColumnName = "job_title",
            DataType = typeof(string)
        });
        priestsDataTable.Columns.Add(new DataColumn()
        {
            ColumnName = "taken_name",
            DataType = typeof(string)
        });
        priestsDataTable.Columns.Add(new DataColumn()
        {
            ColumnName = "is_american",
            DataType = typeof(string)
        });

        priestsDataTable.Rows.Add(new object[] { "Lenny", "Belardo", new DateTime(1971, 3, 24), "Pontiff", "Pius XIII", "yes" });
        priestsDataTable.Rows.Add(new object[] { "Angelo", "Voiello", new DateTime(1952, 11, 18), "Cardinal Secretary of State", "", "no" });
        priestsDataTable.Rows.Add(new object[] { "Michael", "Spencer", new DateTime(1942, 5, 12), "Archbishop of New York", "", "yes" });
        priestsDataTable.Rows.Add(new object[] { "Sofia", "(Unknown)", new DateTime(1974, 7, 2), "Director of Marketing", "", "no" });
        priestsDataTable.Rows.Add(new object[] { "Bernardo", "Gutierrez", new DateTime(1966, 9, 16), "Master of Ceremonies", "", "no" });

        DataSet priestsDataSet = new DataSet();
        priestsDataSet.Tables.Add(priestsDataTable);

        return priestsDataSet;
    }

    public static DataSet Ranchers()
    {
        DataTable ranchersTable = new DataTable();
        ranchersTable.Columns.Add(new DataColumn()
        {
            ColumnName = "firstName",
            DataType = typeof(string)
        });
        ranchersTable.Columns.Add(new DataColumn()
        {
            ColumnName = "lastName",
            DataType = typeof(string)
        });
        ranchersTable.Columns.Add(new DataColumn()
        {
            ColumnName = "dateOfBirth",
            DataType = typeof(DateTime)
        });
        ranchersTable.Columns.Add(new DataColumn()
        {
            ColumnName = "jobTitle",
            DataType = typeof(string)
        });
        ranchersTable.Columns.Add(new DataColumn()
        {
            ColumnName = "nickName",
            DataType = typeof(string)
        });
        ranchersTable.Columns.Add(new DataColumn()
        {
            ColumnName = "isAmerican",
            DataType = typeof(string)
        });

        ranchersTable.Rows.Add(new object[] { "Colt", "Bennett", new DateTime(1987, 1, 15), "Ranchhand", "", "y" });
        ranchersTable.Rows.Add(new object[] { "Jameson", "Bennett", new DateTime(1984, 10, 10), "Ranchhand", "Rooster", "y" });
        ranchersTable.Rows.Add(new object[] { "Beau", "Bennett", new DateTime(1944, 8, 9), "Rancher", "", "y" });
        ranchersTable.Rows.Add(new object[] { "Margaret", "Bennett", new DateTime(1974, 7, 2), "Bar Owner", "Maggie", "y" });
        ranchersTable.Rows.Add(new object[] { "Abigail", "Phillips", new DateTime(1987, 4, 24), "Teacher", "Abby", "y" });

        DataSet ranchersDataSet = new DataSet();
        ranchersDataSet.Tables.Add(ranchersTable);

        return ranchersDataSet;
    }
}

We'll test our system against this sample data.

Something There

Now we can build our actual mapping solution. First off, we need a way to decide what column names map to object properties. It was Marlena's idea to keep those things together, and so we came up with a class called DataNamesAttribute that looks like this:

[AttributeUsage(AttributeTargets.Property)]
public class DataNamesAttribute : Attribute  
{
    protected List<string> _valueNames { get; set; }

    public List<string> ValueNames
    {
        get
        {
            return _valueNames;
        }
        set
        {
            _valueNames = value;
        }
    }

    public DataNamesAttribute()
    {
        _valueNames = new List<string>();
    }

    public DataNamesAttribute(params string[] valueNames)
    {
        _valueNames = valueNames.ToList();
    }
}

This attribute can then be used (in fact, can only be used, due to the AttributeUsage(AttributeTargets.Property) declaration) on properties of other classes. Let's say we're going to map to a Person class. We would use DataNamesAttribute like so:

public class Person  
{
    [DataNames("first_name", "firstName")]
    public string FirstName { get; set; }

    [DataNames("last_name", "lastName")]
    public string LastName { get; set; }

    [DataNames("dob", "dateOfBirth")]
    public DateTime DateOfBirth { get; set; }

    [DataNames("job_title", "jobTitle")]
    public string JobTitle { get; set; }

    [DataNames("taken_name", "nickName")]
    public string TakenName { get; set; }

    [DataNames("is_american", "isAmerican")]
    public bool IsAmerican { get; set; }
}

Now that we know where the data needs to end up, let's start mapping out the mapper (heh).

Reflection

Our mapper class will be a generic class so that we can map from DataTable or DataRow objects to any kind of object. We'll need two methods to get different kinds of data:

public class DataNamesMapper<TEntity> where TEntity : class, new()  
{
    public TEntity Map(DataRow row) { ... }
    public IEnumerable<TEntity> Map(DataTable table) { ... }
}

Let's start with the Map(DataRow row) method. We need to do three things:

  1. Figure out what columns exist in this row.
  2. Determine if the TEntity we are mapping to has any properties with the same name as any of the columns (aka the Data Names) AND
  3. Map the value from the DataRow to the TEntity.

Here's how we do this, using just a bit of reflection:

public TEntity Map(DataRow row)  
{
    //Step 1 - Get the Column Names
    var columnNames = row.Table.Columns
                               .Cast<DataColumn>()
                               .Select(x => x.ColumnName)
                               .ToList();

    //Step 2 - Get the Property Data Names
    var properties = (typeof(TEntity)).GetProperties()
                                      .Where(x => x.GetCustomAttributes(typeof(DataNamesAttribute), true).Any())
                                      .ToList();

    //Step 3 - Map the data
    TEntity entity = new TEntity();
    foreach (var prop in properties)
    {
        PropertyMapHelper.Map(typeof(TEntity), row, prop, entity);
    }

    return entity;
}

Of course, we also need to handle the other method, the one where we can get a collection of TEntity:

public IEnumerable<TEntity> Map(DataTable table)  
{
    //Step 1 - Get the Column Names
    var columnNames = table.Columns.Cast<DataColumn>().Select(x => x.ColumnName).ToList();

    //Step 2 - Get the Property Data Names
    var properties = (typeof(TEntity)).GetProperties()
                                        .Where(x => x.GetCustomAttributes(typeof(DataNamesAttribute), true).Any())
                                        .ToList();

    //Step 3 - Map the data
    List<TEntity> entities = new List<TEntity>();
    foreach (DataRow row in table.Rows)
    {
        TEntity entity = new TEntity();
        foreach (var prop in properties)
        {
            PropertyMapHelper.Map(typeof(TEntity), row, prop, entity);
        }
        entities.Add(entity);
    }

    return entities;
}

You might be wondering just what the heck the PropertyMapHelper class is. If you are, you might also be about to regret it.

Dig a Little Deeper

The PropertyMapHelper, as suggested by the name, maps values to different primitive types (int, string, DateTime, etc.). Here's that Map() method we saw earlier:

public static void Map(Type type, DataRow row, PropertyInfo prop, object entity)  
{
    List<string> columnNames = AttributeHelper.GetDataNames(type, prop.Name);

    foreach (var columnName in columnNames)
    {
        if (!String.IsNullOrWhiteSpace(columnName) && row.Table.Columns.Contains(columnName))
        {
            var propertyValue = row[columnName];
            if (propertyValue != DBNull.Value)
            {
                ParsePrimitive(prop, entity, row[columnName]);
                break;
            }
        }
    }
}

There are two pieces in this method that we haven't defined yet: the AttributeHelper class and the ParsePrimitive() method. AttributeHelper is a rather simple class that merely gets the list of column names from the DataNamesAttribute:

public static List<string> GetDataNames(Type type, string propertyName)  
{
    var property = type.GetProperty(propertyName).GetCustomAttributes(false).Where(x => x.GetType().Name == "DataNamesAttribute").FirstOrDefault();
    if (property != null)
    {
        return ((DataNamesAttribute)property).ValueNames;
    }
    return new List<string>();
}

The other we need to define in ParsePrimitive(), which as its name suggests will parse the values into primitive types. Essentially what this class does is assign a value to a passed-in property reference (represented by the PropertyInfo class). I'm not going to post the full code on this post (you can see it over on GitHub), so here's a snippet of what this method does:

private static void ParsePrimitive(PropertyInfo prop, object entity, object value)  
{
    if (prop.PropertyType == typeof(string))
    {
        prop.SetValue(entity, value.ToString().Trim(), null);
    }
    else if (prop.PropertyType == typeof(int) || prop.PropertyType == typeof(int?))
    {
        if (value == null)
        {
            prop.SetValue(entity, null, null);
        }
        else
        {
            prop.SetValue(entity, int.Parse(value.ToString()), null);
        }
    }
    ...
}

That's the bottom of the rabbit hole, as it were. Now, we can use the DataSet objects we created earlier and our mapping system to see if we can map this data correctly.

Two Worlds

Here's a quick program that can test our new mapping system:

class Program  
{
    static void Main(string[] args)
    {
        var priestsDataSet = DataSetGenerator.Priests();
        DataNamesMapper<Person> mapper = new DataNamesMapper<Person>();
        List<Person> persons = mapper.Map(priestsDataSet.Tables[0]).ToList();

        var ranchersDataSet = DataSetGenerator.Ranchers();
        persons.AddRange(mapper.Map(ranchersDataSet.Tables[0]));

        foreach (var person in persons)
        {
            Console.WriteLine("First Name: " + person.FirstName + ", Last Name: " + person.LastName
                                + ", Date of Birth: " + person.DateOfBirth.ToShortDateString()
                                + ", Job Title: " + person.JobTitle + ", Nickname: " + person.TakenName
                                + ", Is American: " + person.IsAmerican);
        }

        Console.ReadLine();
    }
}

When we run this app (which you can do too), we will get the following output:

Which is exactly what we want!

(I mean, really, did you expect me to blog about something that didn't work?)

Go the Distance

It concerns me that this system is overly complicated, and I'd happily take suggestions on how to make it more straightforward. While I do like how all we need to do is place the DataNamesAttribute on the correct properties and then call an instance of DataNamesMapper<T>, I feel like the whole thing could be easier somehow. Believe it or not, this version is actually simpler than the one we're actually using in our internal apps.

Also, check out the sample project over on GitHub, fork it, test it, whatever. If it helped you out, or if you can improve it, let me know in the comments!

Finally, extra special bonus points will go to anyone who can figure out a) what the hell those odd section titles are about and b) where I got the sample data from.

Happy Coding!

"I Don't Trust Anything That We Didn't Build"

The problems started small, as they often do. But as we've seen many times before, lots of small problems in quick succession tend to make one big problem.

In this case, the problem got big fast. It started off easy enough: read the big report, find the bug, fix it, the usual. Our bug-tracking team located the source of the issue right away, and my team set about trying to work out the fix. We found the data source that was causing the issue, and it happened to be a web service owned by another team. We couldn't see into it, we could only see the inputs and outputs; it was essentially a black box to us.

Here's where the bad part starts. Due to a lack of efficient communication on all sides, impending deadlines, frustrated coders and general hesitancy to deal with this particular, ahem, time-intensive project, the actual bug fix took about four days to nail down. Yes, four days; we were just as annoyed as you are, and probably more so.

To make a long story short, the project we were attempting to deal with was:

  • old,
  • slow,
  • in desperate need of a rewrite,
  • using a data source which we had no visibility into (the aforementioned service),
  • not written by anyone currently on the team,
  • still our responsibility to fix AND
  • needed to be fixed right friggin now.

You should read that list and cringe a little for each bullet point. I know I did.

All of those problems put together (plus the fact that it took us four days to figure it out) prompted my manager, normally a well-reasoned, thoughtful individual, to say during our bug post-mortem:

"I'm starting to really loathe this project. It's getting to the point where I don't trust anything that we didn't build."

I have to say, it's hard to blame him for wondering if we shouldn't be using things that were not invented here.

It's incredibly easy for a software development team, even an experienced one like mine, to fall into the comfortable trap of believing that everybody else's code is terrible and their own is awesome. We developers often forget (or, quite possibly, want to forget) that most of the time the bug is in our code, no matter how much we wish that it wasn't.

Do this enough and the untamed wild of other people's code starts to look like an intimidating place. It's safer, easier, to believe that your code is correct and everyone else's is wrong, because that means you don't have to spend time trying to understand what the other person was thinking. Or, just as often, spending time figuring out how you are wrong, something nobody enjoys doing.

I've written before that I believe code should exist for a reason. The difficulty in working with other people's code is that not only are you trying to understand what the code does, you're trying to comprehend the reason why it does that. That's a difficult thing to accomplish in the best of times (efficient communication being a feat that usually fails, except by accident), and when you're approaching a deadline and trying to have a meaningful conversation with the original developer who has his own deadlines and responsibilities to deal with, it can be nigh impossible.

Let me be perfectly honest: there are times I completely understand my manager's frustration. It would be SO much easier if the only code I had to deal with was my own, because then the only stupid person in the equation is me and I can fix that. Dealing with other stupid people is infinitely more frustrating than dealing with your own stupidity.

To be clear, I am not calling my coworkers stupid; they are in fact quite the opposite. But it's tempting to fall back to lazy thinking and believe they are stupid merely because they were dealing with requirements and scenarios that I didn't have time to thoroughly understand. That temptation, to believe that things are stupid because I don't understand them, is something I find myself fighting against on a daily basis. It's an innate human quality, and not unique to programmers or other technical people.

Here is a basic fact of life: people, on the whole, are not stupid. Programmers do not write code for no reason, as the best code is no code at all and if we could have our way there would be no code, ever. But because code needs a reason to exist, it almost certainly had a set of requirements, or scenarios, or something which shaped its current form. Even if those requirements were merely thoughts in the original developer's head, they existed. It is not the fault of that developer that some idiot who saunters up to a laptop and is trying to break her code doesn't understand what said code is meant to do.

But it's easy to think that, isn't it? It's easy, it's simple, it's lazy. When we don't have time or energy to think, really think, the lazy thoughts are what we are left with. Given that programming is an almost-entirely-mental task, accepting the lazy thoughts as fact could even be seen as a reprieve from needing to think critically all day, every day.

Resist the lazy thoughts. Resist the idea that your fellow programmers are stupid, or wrong, or only doing a half-done job. Resist Not Invented Here syndrome. Resist the idea that because someone didn't understand you, they're dumb. Resist all these little thoughts that end up with a conclusion of "those other people are stupid," and instead try to answer "what were they trying to accomplish?" There's nothing wrong with digging a little deeper for a better understanding.

That's what I say to you: resist the lazy thoughts, and dig a little deeper. You will eventually have to trust something you didn't build. If you keep digging, you'll find what you are looking for.

Post image is Digging a hole for the digester from Wikimedia Commons, used under license.

How Do You Fix An Impossible Bug?

Within the span of an hour, it had all gone to hell.

The first deployment went rather smoothly. It was a fix to an existing web service, and went out with no problems, or so we thought. Within ten minutes of the deployment, the users started complaining of a minor bug, one that was seemingly omnipresent but didn't really stop them from doing meaningful work. The team which had sent out the deployment immediately set to work figuring out what was going on.

Unrelated to that deployment and forty minutes later, my team was launching a major change to a web site that was consuming that other team's web service. When our change went out, the bug that the users had been complaining about from the web service deployment vanished, replaced by a major bug that caused them to be unable to do any work at all. Naturally we were a little concerned.

The major bug caused an error message that included this statement:

"Unknown server tag 'asp:ScriptManager'."

Now usually when I see an error message like that, I think the project is missing a reference. But no, the references the project needed did, in fact, exist. The second thing I think is that it was using the incorrect version of .NET (in this case, 2.0). Wrong again, it's using the proper version. So now I'm a bit stumped; I pull one of my developers off his project to work on this, and he and I go to our manager and the web service team to try to hash this out.

It took the five of us about an hour to work out where exactly the problems were. As so often happens with major problems like this, it wasn't one problem but several, all intersecting at one time. They snowballed like so:

  • The previous day, my group attempted to roll out the major change to the web site. The roll out didn't work, and another manager (who had previously owned the project, and unbeknownst to me) immediately copied the application to a different location on the same server. He figured this would solve the problem, as it had before with a different app; it didn't.
  • Before the web service change went out, the users had already been notified to use the new location. Consequently they started complaining about the major error.
  • When the web service change was deployed, a different set of users within the same group complained about the minor bug, as word had not reached them to use the new location.
  • When our web site change went out (to the original location), the users of that site noticed and now complained of a "new" bug at the old location, despite the fact that it was the same bug as the bug at the new location.
  • All of this taken together meaning that the fact that our web site and their web service were related was a coincidence. The fact that the two deployments went out so near to each other had nothing to do with what the actual problem was. Coincidences like this are the worst possible thing that can happen when trying to find a bug.

Got all that?

Ultimately we worked out the problem. Well, really, we stumbled onto it. How we got there was such blind luck that I'm not convinced we actually solved the problem so much as lucked into a solution.

A bit of googling for the error message revealed this StackOverflow answer which states that the reason we get the above error is that a piece of the web.config is missing. Here's what it should be.

<pages>  
    ...
    <controls>
        <add tagPrefix="asp" namespace="System.Web.UI" assembly="System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
    </controls>
</pages>  

We had previously confirmed that in our application configuration that line did, in fact, exist. Obviously this could not be the problem. (I apparently forgot about what happened the last time I assumed something was true.) Later, when we started getting more desperate to find the source problem, I had our server team give me a copy of the app running on the production servers. This is what I found:

<pages>  
    ...
    <controls>
        <!--<add tagPrefix="asp" namespace="System.Web.UI" assembly="System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>-->
    </controls>
</pages>  

Yes, you read that right. For some damn reason, the offending line of configuration code had been commented out.

And we had no idea how this happened.

We tried examining the config transforms; there was nothing that would result in that line being commented out. We tried looking at the server logs, the build logs, the source control history, anything that might give us a scrap of information as to how the hell this happened. We found nothing.

As you might imagine, this was a little frightening. This line got commented out, and we couldn't reproduce how. How can you fix a bug that should have never occurred in the first place? When we deployed the corrected configuration file, it worked, of course. But in the meantime we had wasted nearly an entire day looking for something that should have been impossible.

But was it impossible, or did we miss something? I'm inclined to believe the latter. One of the things I stress to my team when they come to me with bug reports is the important question of what changed? What changed between when the system worked and when it didn't? Was it business rules, data sources, the build process? If we can determine what changed, the time needed to pinpoint the actual bug shrinks dramatically. In this case, either we simply didn't know what had changed (the most likely scenario) or nothing had changed (the far scarier scenario). Either way, something was off and we couldn't determine what it was.

What was even more worrisome was that there had been a minor bug reported before the major bug showed up, the one that was annoying but not work-stopping. That minor bug was not reproducible now, so it's entirely possible it's still lurking out there, waiting for some unsuspecting user to click the wrong button or enter the wrong date and then blow up the whole system. We don't think it will be that serious, but we also don't really know that it won't. We're stuck in bug-induced purgatory.

That's a terrible feeling. Something went wrong, and you can't figure out how. You know what the solution is, but you don't know why.

I suppose I should just be happy we figured out the problem. I am, sort of. And yet I am forced to conclude that there exists a bug which caused a critical part of our application configuration to be commented out for no discernible reason. A part of me still wonders: how can I find and fix that bug, a bug that should have never existed in the first place?

And what if it happens again?

What about you, dear readers? What are some bugs you found that you couldn't source? How did you fix these "impossible" bugs? Share in the comments!

Happy Coding Debugging!

Post image is 2008 broken computer monitor from Wikimedia, used under license.