New programming tools can turn sketches and handwriting into code

Researchers at Cornell University have created an interface that allows users to write and sketch by hand in computer code — a challenge to traditional coding, which typically relies on typing
.

This pen-based interface, called Notate, allows users of computational digital notebooks to open drawing canvases and handwrite diagrams
within traditional lines of digital computer code.

Supported by deep learning models, the interface connects the handwriting and text programming contexts: symbols in a handwriting graph can reference text code and vice versa
.
For example, Notate can recognize handwritten programming symbols, such as "n," and then link them with their printed equivalents
.

Ian Arawjo, the paper's first author and a PhD student in the field of information science, said: "Such systems are very good for data science, especially when sketching and diagramming, and they can interoperate
with text code.
" "Our research shows that the current programming infrastructure is actually holding us back
.
People are ready for this type of functionality, but interface developers for entering code need to be aware of this and support both graphical and graphical interfaces
in their code.
”

Arawjo also said that at a time when tablet use is becoming more widespread, the work shows a new way
forward by introducing AI-driven handwriting coding.

Tapan Parikh, associate professor of information science and co-author of the paper, said: "Tools like Notate are important because they open up new avenues for us to think about what programming is and how different tools and representation practices can change that perspective
.
"