Overview and Goals


Everyone I know who's tried programming has had a similar experience -- at the start, it's fun and intersting! You're given an elegant language with simple structures that are all coherently composed, and it's very easy to do simple tasks like printing text, doing math, inputting data, manipulating data, etc.

Then you try to write a more ambitious program, and you discover that seemingly simple operations become messy and ugly. That beautiful and coherent language turns into a complex hierarchy of classes and functions with unintuitive names, behaviour, and side-effects.

As Jonathan Edwards points out:

The inelegance becomes especially noticeable when dealing with graphics in any way -- text is just not the right way to express pictures. Proof: A picture = 1000 words. What a waste! Look at how much work it takes to draw squares in PyX: Drawing Squares in PyX. And this is considered to be a nice drawing API! It's ridiculous! Why can't you just sketch out your squares with the mouse and let the system convert your strokes into those Python path commands (with some kind of cleanup optimization)? (See also: Improving Programming; The Limits of Text)

Now, all of these problems can't be solved without a very powerful AI framework for creating programs and dynamic user interfaces to editing those programs. However, one thing that would make programming more enjoyable is real-time feedback.

I really like IPython because it allows me to manipulate and play around with Python modules and data until I come up with what I want. The problem is that after I've finished playing around, I have to remember all the steps I took to get to the solution. I'd like to have an environment that stores every transformation and operation I do on every data item so that I can evolve my algorithms through experimentation, then extract them for incorporation into full programs.

Another important feature to a programming sketchpad is being able to visualize all the data in the interpreter's environment. I was quite impressed by tmPython -- it's got all the basics, but it lacks the ability to introspect your environment. The ideal solution would contain features from tmPython, Matlab, Subtext, and Scratch, plus the novel addition of a pen-based input that allows you to use draw unique symbols to identify your algorithms and objects, as well as allow you to input algorithms by drawing structures and animating them.


Cognitive ergodynamics

The programming environment should offer multiple representations of a single algorithm. Code should be the last-resort, and only used for creating the algorithm. There are many more effective ways of conveying the meaning of an algorithm, for example:

These would all contribute to the user's ability to understand, and remember the program; weeks after creating an algorithm, a picture of it could still be in your mind. With code, you're lucky if you recognize it days after creating it. The program will feel more concrete and familiar to the programmer with better representations; code feels very abstract, and requires continual effort to maintain familiarity with it.

Easily comprehensible by children

I believe that a good way to measure how successful this project is is how quickly a child can learn the system. Children's minds are the most adaptable and the least educated, so if something is confusing about the system or based too strongly on previous programming paradigms, then they should be able to instantly spot it.

Able to deal with ambiguity

Keep all possibilities in a superposition until the ambiguity must be resolved.

Machine learning and introspection

The environment should be able to watch the programmer developing their program and experimenting with code, and learn patterns. The programmer could then name these learned patterns and create "macros" out of them. Whenever the "macro" runs into ambiguity or uncertainty, it could prompt the programmer for advice.

More on this under EmergentComplexity.

Semantic Typing

Perceptual Control Theory is a good model of how our minds work -- perception is as important as action.

The Sketchpad should be able to understand the kinds of data it's dealing with (from their context, perhaps).

This may manifest itself as fuzzy static typing -- data types can be ambiguous and automatically convertible, but when the ambiguity must be resolved, then the user will be presented with the options to resolve.


This should be at least as powerful as your standard language.

Minimimal memorization

The user should be able to see all the objects and data available, and should be presented with likely ways to use each object, given the current context.

Speech recognition

A very efficient method for interacting with humans is to let them verbally voice commands while being given visual feedback. This allows them to "hold things in their hands" while simultaneously issuing commands.

(see: Speech vs. Vision)

Unlimited revisions/branching

All transformations that the programmer does in the system should be remembered. This would allow for unlimited undo, or experimenting with different ways of transforming data and algorithms.

In addition, this would greatly aid the pattern recognition.

Basic Functionality

The basics have already been implemented in tmPython:



The basic version should be able to do the following:

UI Enhancements


Related Projects


HCI/Algorithm_Sketchpad (last edited 2011-11-28 19:18:34 by Chris)