The Clarity Code Processor (Demo 1994)

Author: MIchael L. Van De Vanter


I developed the Code Processor as the "front end" technology for a prototype programming language + environment developed by the Clarity project at Sun Microsystems Laboratories (1992-95). A key architectural feature of the environment was program storage and modification as persistent, semantically attributed Abstract Syntax Trees (ASTs). Based on disappointing prior experience with AST-based code editors, I chose to make the fundamental representation of the editor something in between traditional text editors (where it is impossible to know much about the code with any reliability) and AST-based editors (where the user models had proven unworkable).

The "editable representation" was a lexical token stream, updated with every keystroke. There was no explicit representation of white space in that stream, and text presentation (including the visual gap between tokens) was driven by fine-grained code presentation styles developed by human factors research. The virtue of the token stream was that it is an exact representation that can be updated perfectly per-keystroke, when combined with detailed behavior designed to create the impression of a plain text editor. From the goal to create a natural text-editing experience came the name assigned to this technology, the "code processor" by analogy to the (then) rapidly growing adoption of "word processors".

In the video, note in particular the code typography (driven mostly by the lexical structure, though modified slightly when the AST gets updated) and the incremental feedback made possible by the editor's lexical data structure.

Code Processor prototype demonstration, Video Demonstration, Sun Microsystems Laboratories, Inc., 1994.