Continuations-Based Web Applications in Common Lisp With Weblocks

Tuesday, November 20, 2007

Introduction

Whenever I have to write convoluted code to express a simple idea, I instinctively try to avoid doing work. The corollary of this is that every time I reach for my inbox during coding, there's a pretty good chance I need an abstraction I don't have.

I suspect this is true for most programmers, not just myself. If this is so, GMail owes most of its load to web application programmers. Writing web applications sucks for a number of reasons (most of which, I think, are fixed by Seaside framework, it's only peril being that it forces people out of Emacs). Top three biggest problems that I encountered are managing data, managing UI state, and managing control flow.

Web frameworks made great strides in solving half of the first problem. ORM layers provide a reasonable (though not ideal) solution by automatically mapping data to the database. To date no framework that I know of other than Weblocks does the other half of the work and automatically maps the data model to portable, accessible, standards compliant UI1.

A few frameworks do a reasonable job managing UI state. The best example of a widely adopted framework to do this is ASP.NET. ViewState and Page_Load are terrible ideas, and because delegates aren't true closures writing a good ASP.NET component quickly takes you on a path to warm, inviting arms of heavy narcotics, but at least they set out in the right direction (pun intended). UCW and Seaside do this well. Weblocks does this. Pretty much any framework based on components instead of MVC and template engines does this well.

Aside from Weblocks, three frameworks I know of do a good job dealing with control flow - Seaside, UCW, and PLT. Proper control flow management is difficult to implement - it requires continuations, which most languages don't support. Because frameworks with good control flow management are so rare, most people don't understand what they're missing. In this article I'll describe the problem and how it is solved in Weblocks using continuations. A live Weblocks demo that showcases a practical implementation of these concepts is available here.

The Problem

Control flow management in web applications rears its ugly head in two situations: modal interaction and sequencing.

Modal interaction is anything that involves feedback from the user before the application can move on. Consider action confirmation: user tries to delete something and the application needs to present him with a confirmation dialog. It's possible to simply show a JavaScript popup, but what if the client has JavaScript turned off? The proper solution of returning another page (or code via AJAX) is surprisingly complicated. The server needs to store the current operation somewhere and redirect to the confirmation page. If the user doesn't confirm the action, the confirmation page needs to redirect back to the original page. But what happens if the user confirms? In a less powerful language we have the option of faking closures with a myriad of design patterns, or of hard coding the necessary operation in the confirmation page itself. In a more powerful language we can use closures, but then we're forced to write programs in a continuation passing style. That wouldn't be so bad in this situation, but it becomes less and less practical when sequencing is involved.

Sequencing is a series of nested modal interactions, potentially with branches. Presenting a series of license agreements, going through a shopping cart checkout process, paying your credit card in a series of steps, and approving someone's timesheet are all examples of sequencing. This is workflow - the part of the application that gets things done. Because sequencing can be thought of as a series of modal interactions, all the problems outlined in the previous paragraph apply to every step of the sequence as well. Without closures doing this and still maintaining sanity is not possible. With closures this is more comfortable but not ideal - unless you have three widescreen monitors the indentation will quickly make long workflows very difficult to maintain.

An elegant solution for these problems involves using first class continuations if they are available, or transforming code to CPS style. Because Common Lisp doesn't provide continuations out of the box, I wrote cl-cont - a library that transforms selected pieces of code to CPS. Let's see how this works in Weblocks in practice.

Handling User Actions

Before we get to continuations, let's first discuss how Weblocks deals with user actions. When the user clicks on a link or a button, what happens? Weblocks makes this really easy - you simply render functions into appropriate HTML. For example, a link can be presented to a user like this:

(render-link (lambda (&rest args)
               (declare (ignore args))
               (do-something))
	     "Some Link")

The lambda function is stored in a hash table with a unique string as its key. The link with text "Some Link" and the key in the reference is presented to the user. If the user clicks on the link, the key is looked up in the hash table, and the function is retrieved and called. Because we're using a link, in this case we can safely ignore args. If we were rendering a form instead, we'd use keyword arguments to access parameters passed with the request that interest us.

The implementation of render-link makes the link asynchronous by default - the callback is invoked via AJAX. If the client has JavaScript turned off, a regular request is used instead. AJAX can be turned off by passing :ajaxp nil to render-link.

User Actions And Continuations

Now let's consider modal interaction. Suppose "Some Link" takes the user to content that requires authentication. Normally we'd have to go through a two phase redirect ritual, and right about now we'd be opening GMail. With proper flow control we can just express our intentions in an elegant and straight forward manner:

(render-link (lambda/cc (&rest args)
               (declare (ignore args))
	       (if (do-page (make-instance 'my-login-widget))
	           (show-protected-content))
                   (show-error))
	     "Some Link")

We call lambda/cc instead of lambda in order to turn on CPS transformation for our action. We then call do-page with the login widget as the argument. This is where continuations shine. At this point the continuation is saved and the user is taken to a new page and shown a login box. When the login widget decides if the user can be authenticated it must call answer with the result - a function that will restore the saved continuation. At this point do-page will return as if it were a simple function call, and generation of the original page continues. Our handler for "Some Link" can then choose whether to show protected content or an error.

A lot could be written about lambda/cc and do-page, but that would be an article about implementing a CPS transformation in Common Lisp, and using delimited continuations to implement interesting things. This is somewhat complicated. But once the functionality is in place, using it is as simple as the code snippet above.

Weblocks provides a number of primitives built on top of continuations. One such primitive is do-choice - a function that presents a modal dialog with a number of choices the user can pick from. On top of do-choice there is do-confirmation2 and do-information - functions that let the user confirm an action and review information, respectively.

Before we move on to sequencing and workflows, let's talk a little about user interface state management and widget-based approach to web applications.

UI State Management And Widgets

Technology is the limiting factor of ideas. Web 1.0 was powered by ASP, C++, J2EE, and occasionally, when after six months of trying people realized they still can't reliably instantiate a stateful session bean, Perl. It shouldn't come as a surprise that most of the failing Web 1.0 applications were glorified shopping carts - it took about two million dollars to build an online store.

Web 2.0 is powered by MVC. We've transcended shopping carts. MVC lets you build really simple applications really fast. The idea is that simple applications is what people want. I suspect that what people really want, Black Swans aside, is simple UI to really complex processes that take up their time and money - something that cannot easily be done with MVC, the way it is normally used.

As the applications get more complex, managing state of user interface components becomes more and more difficult. At some point one runs into a complexity asymptote that cannot be crossed without jumping to a different abstraction. Components (or widgets) provide an abstraction that allows crossing this invisible line.

Perhaps the best way to illustrate the componetized philosophy is by example. Weblocks provides a number of widgets to perform common operations. Dataform widget allows the programmer to present the data to the user, let the user click "Modify" to see a form and modify the data, let the user submit or cancel the changes, and return to the data view. All in one line of code:

(make-instance 'dataform :data some-employee)

Another example is an automatically sortable, pagable grid of items. In Weblocks I can create one like this:

(make-instance 'datagrid :data
               (list some-employee-1
                     some-employee-2
		     some-employee-3)
	       :data-class 'employee)

The biggest advantage of widgets is that they maintain their own state. I can place a dozen datagrid widgets on a page and let the user sort them in different ways and go to different pages. There is no central place that must keep track of this state - each widget instance is responsible for itself.

Widgets are composable. For example, a Weblocks gridedit widget uses datagrid and dataform to allow the user to add items to the grid and drill down on existing ones. This paradigm can be extended to the entire application. A composite is a widget that simply holds a list of other widgets (possibly including other composites), to be rendered in order. A page is simply a composite widget with other widgets laid out inside.

This approach allows for very complex (from the programmer's point of view) user interfaces that can be easily implemented and reused. In component based frameworks I can take an entire application and embed it into one of my widgets without affecting any part of my application. Code reuse doesn't get any better than that!

Another benefit is free AJAX support (the framework simply sends updated widget HTML to the client) which automatically degrades if JavaScript is turned off. But I digress.

Sequential Control Flow

Let's consider a simple workflow scenario - displaying three license agreements in a row. At each step the user has an option to agree or decline - if the user agrees we move on to the next agreement. If he declines, we show an error page. Normally each step would contain logic to redirect to the next step. In a simple scenario this is uncomfortable because the workflow isn't described in a centralized place. In a more complex scenario with branches, maintaining the workflow becomes a nighmare.

Let's see how this can be done in Weblocks:

(with-flow (composite-widgets (root-composite))
  (if (and (yield agreement-1)
           (yield agreement-2)
           (yield agreement-3))
    (show-protected-content)
    (show-error))

The macro with-flow expands to the boilerplate code necessary to make the transformation possible. The first argument to the macro is the location in the widget tree where the agreement widgets should be placed. In our case the agreements will take up the entire page - the root of the tree. From there we write regular code, using yield to present each step. As with do-page before, each agreement widget is expected to call answer to return control. In this case each widget should return a boolean value. If the user ever chooses not to abide by the agreement, the and expression short circuits and we show an error. If the user abides by all three agreements, we show the protected content.

Note that at any given time we can have multiple flows in different parts of the application (or page). Each flow is independent from the others, and is saved and restarted as many times as necessary.

What's Next?

While continuations are initially difficult to understand, using them when they're wrapped in a higher level of abstraction is easy. There is no reason to suffer through artificial redirects and adhoc state machines that simulate control flow. Grab Seaside, PLT, UCW, or <shameless plug>Weblocks</shameless plug> and get started!

Comments?

If you have any questions, comments, or suggestions, please drop a note at coffeemug@gmail.com. I'll be glad to hear your feedback.

1I know about scaffolding. It's not an abstraction, it's a toy.

2You can see how do-confirmation works by attempting to delete an item in the live demo.