Updated: SketchFlow introduces ink to the design process

If Twitter is any gauge, a forthcoming feature of Expression Blend, called SketchFlow is one of the highlights of this week’s Mix09 conference.

In brief, SketchFlow is a paper-and-ink-like model applied to application design within Expression Blend–a tool used by WPF and Silverlight developers. The idea? Sketch out application concepts and sequences of your ideas in ink first before you commit them to the more straight-jacket designed developer apps. By staying in sketch mode, the theory goes, you’ll have more flexibility to try out different and hopefully better ideas. SketchFlow is not just about drawing states of an application though. You can animate the handcreated components too. Yes, the sketches become manipulatable objects. And when you have the design the way you want it, the goal is that it shouldn’t be too difficult for Blend to generate a pixel clean and aligned version of your design.

Like a lot of people, when I saw Microsoft demo SketchFlow during the Mix09 keynote (starts about 99 minutes into the presentation), I immediately jumped for the Blend 3 Preview download. Unfortunately, SketchFlow is not supported in it yet. We’ll have to wait for a later build.

In the meantime, we can study the Mix09 videos (more are to be posted later according to Arturo Toledo PM of Blend and Silverlight). I don’t quite know how SketchFlow works, but supposedly you’ll be able to draw you dialogs or whatever, sequence them, and then transition them into a final version of your app.

You’ll even be able to generate documentation from your SketchFlow sequence. Handwritten annotations are also supported. And playback is supported on Windows as well as the Mac.

As a Tablet enthusiast, the part that interests me the most right now is the drawing of the sketches. Supposedly you can do this from within Blend. So look closely at what Microsoft was showing at Mix. These sketches don’t appear to be using the classic Tablet PC ink. That’s quite notable.

However, I think we have a couple different “sketch” styles here going on here. The details aren’t clear though. Via Twitter people are saying that you can create sketches in Blend, which would be terrific, but I wonder if they are meaning a subset can be created in Blend and for others you have to create bitmaps in some 3rd party app. Not sure.

Anyway, I’ve been thinking for awhile that the definition of what ink is in Windows needs to be revisited. It’s not just a smooth stroke. It needs to have variability and the appearance of pencil on paper, or ink on paper, or oil on canvas and so on. WPF and Silverlight are beginning to take us in this direction in terms of viewing ink as a vector with some style applied to it. It appears as if the Blend team has gone this direction and is making the sketches look “realistic.” If so, this beats hands down any “Tablet” ink.

If this is the case, I’d like to see the Blend style ink migrate into WPF and Silverlight as inking styles. Maybe one day they can then be blended into the Windows platform, but hey, if they’re available to all WPF or Silverlight apps, I’d be quite satisfied.

Now another thing that appears to be going on here, is that within the Blend demo you can see the person creating standard controls with a handdrawn rendering style, but in actuallity they’re drawing the control just as they normally would with the mouse. The rendering is different; not the input approach. The person isn’t drawing the control freehand.

I hope that’s not what people are referring to when they say Blend 3 can be used to create these sketches. This would just give the viewer the impression that the screen is handdrawn, but it’s not. Now maybe that’s sufficient, but I don’t think so. People would quickly catch on and think of the wiggly-styled controls just like any regular control. I hope there’s more to it than drawing perturbed vectors with the mouse. I wonder.

Reading a post by Christian Schormann on SketchFlow this sentence seems to lead me to believe that there’s “sketching” and then there’s “sketching” when it comes to SketchFlow:

“SketchFlow is informal and quick, enabling you to sketch out plentiful ideas for dynamic interactions in a cost effective manner. SketchFlow also supports the evolution of your rough sketches into living and breathing prototypes that can be as real as you need them to be.”

If I read this correctly, you can create “sketches” quickly–I’m guessing using the mouse and keyboard rendering them the wiggly style and handwritten font (Buxton Sketch TTF). These aren’t sketches at all; they’re just rendered as them. And then there’s the “rough sketches” which I’m guessing are scanned paper images or something drawn in Photoshop or Illustrator. Again, I hope this distinction doesn’t live out.

Now where these “rough sketches” are created is a very interesting question to me. I see a huge opportunity here with even OneNote or Journal or even an add-in to Visual Studio (like I’ve worked on in the past) that supports some form of inking. Of course, AutoDesk’s Sketchbook would be the ultimate here with it’s “realistic” drawing instruments, but we’ll have to see. In fact, come to think of it, maybe an ArtRage style set of screen renderings would fit in here. Oil painted or crayon rendered screens anyone? 🙂

To me, Blend 3’s SketchFlow is a perfect place for the Tablet PC. As someone mentioned to me on Twitter, SketchFlow screams Tablet. I’m very excited to see how this plays out.

Update: A few more screenshots are on my keynote post from the other day.

Update #2: About the notion of drawing freehand sketches with the pen versus the mouse. I can appreciate the issue of freehand drawing and the need to clean it up (such as snapping coordinates and algning multiple objects in a column or row) as well as the need for recognition. However, that’s where the state of the art is and that’s where the magic is. The Tablet PC group has created a reasonable first attempt at shape recognition in their SDK and possibly this could be used. I see lots of context here so the recognizer would minimally need to be adjusted. However, for standard dialogs, I can see handdrawn creations being recognized in the background whether they are replaced with recognized symbols or not. And I think there’s this whole other question as to how much recognition you need to do anyway. There’s lots of room for creativity here. A programming model and access to ink (if there is any) are two requests I’d make.

Update #3: The Mix09 session on SketchFlow (Sketch Flow: From Concept to Production) is now online. Christian Schormann, Director of Program Management for Expression, gives the presentation.

Well, it looks like my original take on Blend 3 was right. Blend 3 does not support realistic looking sketching. It allows you to draw vectors with the pen tool just like Blend always has. For those of you familiar with Blend, these are not quick and dirty sketching tools like let’s say in AudoDesk’s Sketchbook. And they aren’t handwriting friendly pen strokes either unless you write very large.

So really, SketchFlow is not really that Tablet friendly. Mouse and keyboard still are king. That’s not a bad thing, but it’s worth pointing out.

In a nutshell, here’s a screenshot “handdrawn” in SketchFlow’s Blend 3:

I wonder if you’re thinking what I’m thinking: This looks like something I’d draw. Not very inspiring, but doable.

As a step up, Blend 3 also provides predefined handdrawn looking styles for all common controls. That way you can keep your screens looking less finished, which will help promote thinking in terms of the design itself than the appearance of the controls. These are called Wiggly styles:

Now here’s a SketchFlow screenshot more like what Microsoft showed in the keynotes:

Looks better doesn’t it? It looks more like what a designer would come up with. Unfortunately this is something that was not drawn in Blend. In fact, it wasn’t drawn with any tools. It was drawn on paper and scanned in.

Uhm, this is quite unfortunate.

I think Blend is missing a fundamental opportunity here: Sketching itself needs to be part of the electronic process. Either better tools need to be added to Blend or the Expression team needs a better 3rd party story. With Tablet PCs especially, but of course with any Wacom digitizing tablet, there should be a reasonable story for creating sketches.

Without this easy flow, there’s going to be more work unless most of the time is spent on transition editing. Maybe that’s the case. Also, there’s the issue of managing all of these assets during the design process. Imagine all the permutations.

Minimally, here’s what I’d like to see. For Blend 3, I’d like to see two tie ins for sketching, since my guess is there’s no feasible way to add realistic sketching to Blend 3. First, AutoDesk’s Sketchbook needs support. You should be able to copy and image from Sketchbook into Blend without creating any intermediate image file. Blend should paste the image copy from Sketchbook and create a local Blend asset for you. The copy/paste is a little extra work for the user, but not terrible. Currently, since Blend doesn’t support copy/paste like this a user has to save to an image file–probably with their other working sketches, then copy the file over to Blend’s project infrastructure.

Alternatively, there needs to be good OneNote and/or Journal support (I’ll leave MS Paint out of the picture for now, because I doubt most people would use it for designing.) I can see people using OneNote to sketch out their ideas and then after they’ve settled of a few of them copy/paste or import the content into Blend–again without having to copy/paste to an image file via the snipping tool or any direct means of copying. Note also that OneNote doesn’t have a file model like Word or Sketchbook or most other apps. You just draw and write, you don’t think about saving to a file. All of that is done for you in the background. So to use ink or an image out of OneNote requires creating an external image file (via the snipping tool possibly) and then importing the image into Blend. That’s not optimal, nor is it the natural way to work. As I see it, minimally Microsoft should make OneNote the sketch king and Blend the flow king.

I’d take this issue of copy/paste even further. Just being able to take a screenshot of an existing window let’s say and paste it into a page makes sense to me. Blend ought to be able to do this.

Beyond all this, what I’d like to see is Blend become a better Tablet PC tool. May I suggest to the Expression team that one of the things they do is switch to Tablet PCs for their notebooks. And by all means if you’re giving a presentation, think about using a Tablet.

Interaction with a mouse and keyboard still needs to be very good, but pen support needs to be better than it is. Small targets are already an issue in Blend. Now the sketching tools or at least sketching process needs to be revisited.

I applaud the Expression team for trying to improve the design process and like Bill Buxton said in his keynote, this is a step in the right direction.