tags: - instrumenter - programming - graphs

Visual Coding

Also known as "node based programming" and a bunch of other terms; an interface to build software by visually assembling together nodes in a graph (I refer here to the mathematical notion of a graph, one made up of nodes and edges).

It is a cool way to represent software. It inherently makes visual the connections and relationships between different parts of a program. In the course of working with software in a typical fashion where we are writing code, actually you will realize that the connectivity of the program we're building is mostly made through repetitions of variable names, through the implicit transfer of control flow from e.g. one line of code to another, or hopping around from an if condition to the else clause, and so on.

There are plenty of good examples for this, nonexhaustively I could cite:

and many others.

There are many drawbacks inherent to the typical graph based coding approaches that we see. Sometimes we can't even get simple things like loops, if you have just a few if/elses and a loop or two your graph has gotten to a point where it's pretty difficult to reason about, while it remains very simple to grasp to any programmer who has learned to read code.

While it is true in principle that visually representing how a program works (its internal connectivity if you will) can be valuable and this does become abstracted away in a significant way in textual code, in practice, a complete visual representation of real programs ends up being much harder to work with. When we try to push it up a bit to make the visual elements slightly more abstract, representing "bigger units of computation", what happens is these become little black boxes that you have to open up to look under the hood at the "real code" that's driving them, and now you're flipping back and forth opening up little boxes. That's certainly a workflow from hell.

Furthermore, when we look at good software made with modern languages, their power and productivity comes from features that we leverage to achieve higher order patterns like abstraction, encapsulation and decoupling to help us manipulate data. There's perhaps nothing preventing a visual first programming solution from enabling such things, but it's clear from countless existing attempts that visual programming systems have a number of barriers that simply make it harder to practically achieve these things. In a visual system you may represent something like a function by encapsulating a particular flow of nodes into a larger node that you could work with on its own, but to effectively actually develop with that, we're going to need to seamlessly hop inside and outside, and now we have a node system where each node could itself contain a whole graph inside of it, with this recursion going arbitrarily deep. Conceptually none of this should be a showstopper, and actually conceptually there is a good bit of elegance to it. It is just that in practice we are so far from seeing anything resembling this.

A large part of the problem with trying to make code entirely visual is that it imports a great deal of added complexity in managing the fundamental change in structure. Instead of a text file which we've worked out lots of practical means of manipulating, we're dealing instead with an often proprietary directed graph system. You're starting from scratch (look, a pun) in terms of software version control, for starters.

All that being said, shaders are a good example of a niche in which node based coding makes good sense. The programming environment is more constrained, and the output is very visual, so a system that relieves you of syntax-related fiddling and guarantees an instant feedback cycle with automated compiling and testing can be highly attractive. It would be a bit more cumbersome to build functions in shaders, but many shaders don't really have the kind of complexity where you really need to have functions. So, in this niche, we're able to get most of the benefits and avoid most of the drawbacks of node based programming.

What's next?

I believe there is something of value in the visual coding approach. There are many things of value there. It has to be the case because it is a well established (and some may argue very much superior) workflow in many kinds of creative software.

Where I want to take this idea forward is to explore the space of augmenting arbitrary software with visual representations of what that software does. The intrinsic benefit and value of the visual coding paradigm is intuitive visualization of control flow. The other benefits of such interfaces, such as preventing you from worrying about type checking, compiling, syntax, I could actually hand wave them all out as being fringe and offering little value and irrelevant. This is a bias of mine, being a 20 year coding veteran, of course.

I find in my experience writing software across many industries that it's true that we're often blocked from making progress on our work when we lack a sufficient understanding of both how the software is intended to work, and how it's actually working, which likely has diverged.

One simple perspective would be that we wouldn't have that class of problems if we built our software with visual coding technology. That is clearly a non-starter, if your software isn't just a shader, because the amount of complexity you could conceivably achieve with a pure visual programming system is probably a good few decades behind the features and capabilities available in "real" programming languages.

No, the approach I have been exploring is to apply instrumentation on modern languages to pick up where they left off on language features like introspection and reflection, add them in where necessary, and building and optimize an engine from which to visualize the execution of the software and the data it is operating on, across time. Conceptually you might think of it as some kind of debugger/tracer.

To summarize, node based coding is a nice concept, but most if not all implementations go about it in a way that requires dispensing with all the tooling that traditional textual programming languages have accumulated. To do so is pure folly and dooms all such efforts! At least for general purpose programming.

One of my projects (probably the one I'm the most excited about, ever) is definitely about making software visual, and I have been planning and building it out for over a year already now, but it wasn't until now that I actually really got to comparing my vision for it to existing visual coding platforms:

  1. I will NOT convert code from text into nodes in a graph. I will keep its existing structure at all costs so that it makes zero impact on how you work with it with existing tools.
  2. I will support a large number of existing popular programming languages instead of creating a new programming language.
  3. I will find a way to visually surface graph-like structures from the parse output of existing languages. Abstract syntax tree structure and variable identifier relationships highlighted in practical and usable ways. This is an area I find lacking in the current state of the art of text editors.
  4. I will find a way to generate, consume, and visually represent code execution flow in relation to the above natural representations of software so we can visually inspect execution snaking across the tapestry of our code.