Live Programming by Example:
Using Direct Manipulation for
Live Program Synthesis

1   Introduction

Most interactive UI applications today are based on an event-based programming model in which user interactions are represented as events. The main event loop then dispatches these events to registered event handlers that change both the internal application state as well as the visible user interface. However, imperative updates to a stateful user interface can results in data dependencies that are difficult to keep consistent. Therefore, patterns such as Reactive Programming [1] and Model-View-Controller [11] have been adopted that separate the handle code for updating the application state from the render code for generating the output. Figure 1 illustrates how this programming model processes user interactions.

While common user interactions only affect the application state and output, live programming environments like Smalltalk [6] and TouchDevelop [2] also allow changes to the code of a running application in order to provide earlier feedback to the programmer for both debugging and development. Large code updates may require a restart or manual data migration but smaller code updates can often be applied automatically while retaining the current application state. In contrast to regular user interactions performed through the application’s user interface, these live code updates are performed in the source code editor of the development environment (see Figure 2).

In this paper, we introduce a new kind of user/developer interaction that extends live programming with UI interactions. Instead of supplying a change to the source code, the current user interface can modified through direct manipulation [18] to change the running code. The changed user interface serves as example for the intended output of rendering the current state. Depending on the domain and programming language, there are various different programming-by-example techniques [12, 7] to ‘repair’ or ‘synthesize’ a new program whose rendered output partially or fully conforms to the provided output example. To better describe the general design space of these systems, as illustrated in Figure 3, we also include a minimal formalization.

Additionally, we present a concrete prototype based on a live programming environment for JavaScript for applications that separate rendering from event handling [2]. The environment allows string constants in the render code to be changed directly in the HTML output based on a dynamic string origin analysis [21]. This programming environment serves as a precursor for future systems that allow synthesis of more complex live code modifications by direct manipulation.

To summarize, the contributions of this paper are

• a new kind of user interaction that combines live programming of stateful event-based applications with programming by example based on direct manipulation of the user interface,
• a formal definition of such systems, and
• a JavaScript implementation for direct manipulation of string constants in the render code of running programs.

2   Related Work

The term live programming is used in different contexts and can denote the act of programming as part of a live art or music performance or the feature of programming environment to continuously evaluate expressions and displaying results alongside code. However, this research focusses on live programming as the ability to change the code of a running stateful application without restarting it, also known as hot-swapping or dynamic software update [9], while providing immediate and continuous feedback about these code changes. Such live programming systems were described and motivated by Hancock [8] and further explored by systems like SuperGlue, which uses dynamic inheritance and explicit FRP signals [14], and Elm, which demonstrates live programming and time traveling with first-order FRP [4].

The solution described in this paper is based on earlier work on live programming for event-based systems [17] and generally follows the live programming technique for TouchDevelop which requires UI rendering and stateful computation to be separated and prohibits function values (closures) in the application state to allow the state and the code to be updated independently [2]. More recent work outlined the possible design space between live programming systems that resume computation (with a possibly inconsistent state) and systems that record and replay execution [15], as well as introducing managed time as concept for supporting both live programming and time travel [16].

Programming-by-example allows users to author programs by providing examples instead of writing source code. Prior work on programming-by-example and programming-by-demonstration ranges from domain-specific macro systems, visual programming languages and inference of string processing rules [12, 7] to depth-limited generate-and-test approaches for general-purpose programming languages. Most noteworthy, CodeHint [5] synthesizes short Java code snippets at runtime based on user-provided queries. In order to synthesize larger code snippets, SMT solver-aided approaches may be a promising alternative [19].

Direct Manipulation [18] of the graphical user interface is a well-known form of user interaction with popular applications in Smalltalk [6], Morphic [13] and others. However, direct manipulation usually only affects the current state of visible objects. Recent work on Prodirect manipulation [3] shows how direct manipulation of SVG vector graphics can be used to automatically modify the SVG rendering code. The same idea has also been applied to the manipulation of string constants in PHP web applications [21]. These two projects are most closely related to this paper but do not support live code updates of stateful applications without restarting the execution.

3   Live Programming for Event-based Languages

Live programming enables changes to the source code of running programs without restarting its execution. The inherent entanglement of state and code poses a challenge for live programming but approaches like functional reactive programming or a separation of rendering and event handling enable programmers to perform many live updates without manual data migration or restarting the execution.

3.1   Example JavaScript Application

To illustrate the challenges of live programming for general imperative applications, we consider a simple interactive application that replaces keywords in a text according to a fixed rule^{1 1}There are several browser extensions to replace keywords on webpages, some of which are inspired by https://xkcd.com/1031/. .

Figure 4 shows a ‘traditional’ way of implementing this application with imperative updates to register and modify event handlers (inp.onkeyup), mutate global state (str) and update the graphical user interface (out.innerHTML).

In the context of live programming, changing "leopard" to "butterfly" in Figure 4 should ideally also update the visible output without modifying the application state. However, this would involve registering the modified function as new event handler (line 9) and updating the output (line 11 and 12) without also mutating the state (line 10). Due to the structure of the code, the only choice is to update the event handler and display an inconsistent output until the next keyup event, or to restart the application and force the user to re-enter the string.

3.2   Separating Rendering from Event Handling

In the example shown in Figure 4, any change in the application state involves corresponding updates in the output to ensure consistency. Programming models such as Reactive Programming [1] and Model-View-Controller [10] solve this issue by separating the rendering code from other stateful computation. Figure 5 shows an implementation of the same application but the output will be generated by a pure render function, which cannot modify the application state, and events will be handled without directly accessing the DOM output. Instead, any state modification automatically triggers a re-rendering to keep the output continuously consistent.

3.3   Live Code Updates

Separating rendering from event handling has the additional advantage that both can be updated independently. Following the example from Section 3.1, changing "leopard" to "butterfly" in Figure 5 can now automatically re-render the output with the new rendering code. Since render is pure, this process does not affect the application state.

In contrast to changes to the rendering code, changes to the event handling will not immediately be visible in the output. Instead, the updated code will be used to handle all subsequent events^{2 2}Alternatively, it is possible to restart the execution and replay past events. .

In addition to live programming, separating rendering from event handling also has advantages for back-in-time debugging. By either recording events or snapshotting the application state between events, it is possible to travel back in time and visualize past execution states using the pure rendering mechanism. Combining back-in-time debugging with live programming enables the programmer to navigate both execution as well as the version history simultaneously.

3.4   Limitations

Changes to the program code may involve added, modified or removed function definitions. Unfortunately, function values/closures in the application state cannot always be automatically transformed to match the new code, therefore closures are currently not allowed in the application state.

Additionally, an application that resumed execution with an updated event handler may behave different than an application that has been restarted. Depending on the development or debugging context, this may or may not be desirable.

Finally, changes to the initialization code of the program or the data type of the application state will likely result in an incompatibility and require either manual data migration or a restart.

4   Live Programming by Example

Live programming systems as described in the previous section enable program updates at runtime for changes made to the program source code. By separating rendering from event handling, the application output can be updated in reaction to code changes to provide visual feedback to the developer. Extending on this idea, the output of the program can also be used by the developer to make changes to the program — in particular in cases where direct manipulation is more convenient or intuitive than edits to the source code. This idea of live programming by example is also illustrated in Figure 3.

4.1   Formal Definition

Live programming by example is applicable to a wide range of applications and programming languages. To clarify the semantics and system requirements for the approach outlined in this paper, we model the system configuration and its transitions on a higher level of abstraction.

Figure 6 shows a formal definition of a system that supports live programming by example. The input events q, user interface representation o, values v, expressions e and evaluation semantics e ↓ v all depend on the concrete implementation and are mostly left unspecified. However, the system has to be event-based in the sense that input from the environment is supplied as sequence of events q. Additionally, the output of the application is not generated imperatively (e.g. via printf statements); instead, it is summarized as a single value o (e.g. the DOM of a JavaScript application or the framebuffer of an OpenGL application).

Following the programming model described in Section 3.2, the program code is partitioned into event handling h and rendering r such that h is a function that mutates state s in response to events while r is a pure function that generates output o.

There are three different high-level interactions with the system. Regular input events are processed by evaluating the event handler and rendering the potentially modified state (e-event). Updates to the program code h or r are performed without changing the state such that the output is re-rendered with the new rendering code r′ and subsequent events are handled with h′ (e-swap). Given output o, the developer can initiate live programming by example by changing the output o into a desired output example o′ that is then used for synthesis (e-example). The judgement (r,s,o) ▽ r′ infers a modified render function r′ from r such that, ideally, the output o′′ generated by the synthesized render function r′ exactly matches the user example o′ = o′′. However, it may be advantageous to use a heuristic that prioritizes smaller changes to the render function r and tolerates minor differences between new output and the example o′≈ o′′.

Apart from language details of the implementation and the concrete UI representation of the output, this definition highlights the design space for live programming by example. Both the program synthesis technique (▽) as well as the user intent of supplied examples (≈) enable a wide range of different approaches ranging from the simple manipulation of string literals to sophisticated direct manipulation interactions and end-user programming.

5   Editing String Literals in JavaScript Programs by Direct UI Manipulation

Based on prior work on live programming environments for event-based languages [17], we implement a first prototype that supports live programming by example for editing string literals in JavaScript application through UI manipulation. Its source code^{3 3}Source code at http://github.com/levjj/rde/ and a live demo^4
4Online live demo at http://levjj.github.io/rde/ are both publicly available.

5.1   Example Interaction

Given an example application for replacing keywords in a text according to a static replacement rule (see Section 3.1), this rule in the source code can be modified through UI manipulation. Our approach depends on a separation of rendering and event handling (see Section 3.2), so that changing the string constant "leopard" to "butterfly" immediately updates the output accordingly. This change can be done by directly editing the source code, but it can also be performed by editing the output such that the generating string literal in the source code will be changed to match the intended output example (see Figure 7).

Interactions with the application’s user interface are usually handled by the application itself. In order to support live programming by example, the programming environment provides a way to halt execution 1 and enable a special interaction mode for the application’s user interface. Alternatively, it may be possible to reserve certain controls for this purpose, e.g. reserving a special meta key for direct manipulation or using halo controls [13].

Different forms of direct manipulation may be available (e.g. resizing or reordering via drag and drop). In this example, the text of the static label (<p>) becomes editable by the user/programmer 2.

Parts of the label text "leopard" originate from a string literal in the program source code. Therefore, the string literal is highlighted in the editor 3 and changes in the user interface will also be applied to the source code. Text parts that do not originate from source code literals cannot be modified. Any change to the source code causes the output to be re-rendered, so if the word “keyboard” would appear more than once in the input text, all of its occurrences would be replaced according to the new render function thereby ensuring consistency between code and output.

5.2   String-origin Analysis

In order to support the interaction outlined in the previous section, the environment needs a mapping of strings in the output to their generating string literals in the source code. This mapping can be obtained by instrumenting the execution such that string values are tagged with provenance information. This form of dynamic analysis is closely related to dynamic taint analysis for information flow security.

As a first step, all string literals/constants in the source code are assigned a unique identifier by replacing them with constructor calls, yielding tagged string values with origin information.

Additionally, built-in unary and binary operations which cannot be instrumented are replaced with calls to custom functions implementing these operators.

Tagged string objects include a custom implementation of concatenation, substring extraction, replacement and other common string operators (e.g. str.toLowerCase()), such that origin information is retained but otherwise behave like regular string values. As a result, different parts of a string can originate from different string literals. The origin information therefore includes all subparts of a string alongside the identifier of the generating string literal and offset.

JavaScript code that is not part of the program, especially built-in/native code, is not subject to source code rewriting. To ensure correct behavior for applications passing tagged strings to built-in functions, the tagged string values are wrapped in a proxy [20] that automatically converts tagged strings to primitive strings when no instrumentation is possible or necessary (e.g. for parsing strings as integers). Additionally, input events from the DOM passed to event handlers are wrapped in a proxy membrane that transparently converts primitive strings to tagged strings. While this approach preserves program behavior, it is possible for a string to ‘lose’ its origin information due to built-in JavaScript functions.

5.3   Programming Environment Integration

The kinds of UI interactions supported for live programming depend on the domain and concrete representation of the output. In the context of the live programming environment for JavaScript, the output is a tree of HTML/DOM elements and attributes. With string origin tracking, plain text content, attributes and element names can potentially contain origin information.

The programming environment shown in Figure 7 supports two ways of manipulating the output for the purpose of live program synthesis. The developer can either edit the raw HTML code or manipulate the graphical user interface. The HTML representation has the advantage that all parts of the output including element names and attribute keys can easily be modified textually. Manipulation of the actual UI is limited to the plain text content of visible HTML elements but is highly intuitive and immediate.

Given a modified DOM tree, the program synthesis generally follows the following informal algorithm:

1. Determine the previous output based on the current rendering code and application state.
2. Compute the difference between the provided example and previous output. (The modified DOM tree either has new characters inserted, existing characters removed or both^{5 5}Changes to the tree structure of the DOM output are not currently supported and remain future work. .)
3. Check origin information of the modified characters. Modifications to string parts that do not have origin information cannot be handled and will be suppressed.
4. Determine source code location of the generating string literals using string origin information and a mapping from string literals to AST nodes.
5. Use the source code location and computed offsets to insert or delete characters in the program source code.
6. Recompile code and obtain a new rendering method.
7. Render output using the modified rendering code and the current program state.

6   Discussion and Conclusions

Based on both live programming and programming by example, we introduce live programming by example as a way to change the code of a running program by direct manipulation of its user interface. We also describe a concrete live programming environment for JavaScript that allows changes to string literals in the source code by editing text in HTML/DOM output.

The approach presented in this paper requires applications to separate rendering from event handling. Thereby, any updates to the rendering code can be synthesized and applied immediately. Updates to the event handling code will only affect subsequent events and therefore live programming by example is not directly applicable to the event handling code. A possible solution is to replay past events instead of resuming execution with the existing state. However, it is not always clear how many events have to be replayed as replaying all events may not be practical or desirable for long-running applications and replaying just the last event may not suffice.

Changing string literals in the program by manipulating text in the output is a very simple implementation of live programming by example and thereby avoids ambiguities that are common in program synthesis applications^{6 6}The string synthesis technique is still ambiguous as inserted characters between two different generating string literals can be inserted either at the end of the first string literal or the start of the next one. We currently resolve this ambiguity by always prioritizing the first string literal. . However, more complex live programming by example solutions have to address potential ambiguities with heuristics or manual user intervention.

Finally, the program environment presented in this paper mainly serves as a precursor for future systems that support more sophisticated program synthesis guided by more flexible forms of direct manipulation. Moreover, the approach still needs to be evaluated for larger applications and development tasks — potentially as part of a user study.