1y ago

A different kind of programming workflow paradigm

Over 10 years ago, I had this sort of a prediction that, with the massive adoption of a dynamic language like javascript on both client/server sides and test-driven development gaining a lot of ground, the future of programming would be dynamic and "feedback-driven". As in, you would immediately see the results of your code as you type, based on the tests you created. To naively simplify, imagine a split screen of your code editor and a console view showing relevant watch expressions from the code you're typing.

Instead what happened was the industry's focus shifted to type safety and smart compilers, and I followed along. I'm just not smart enough to question where the whole industry was heading. And my speck of imagination on how coding would have looked like in the future wasn't completely thought out. It was just that, a speck of imagination that occurred to me as I was debugging something tedious.

Now, most of the programming language world, seem to be focusing on smarter compilers. But is there some language or platform, that focus instead on a different kind of programming paradigm (not sort of OOP, FP paradigm, may be call it the programming workflow paradigm?). May be it comes with a really strong debugger tooling that's constantly giving you feedback on what your code is actually doing. Think REPL on steroids. I can imagine there would be challenges with parsing/evaluating incomplete code syntax and functions. So I guess, the whole compiler/translator side has to be thought out from the ground up as well.

Disclaimer: There's a good chance I simply don't know what I'm talking about because I'm no language designer or even close to understanding how programming languages and it's ecosystems are created. Just sharing some thoughts I had as a junior dev back in the day.

32 comments

The main issue I have with this is the exclusive focus on happy path. The point of unit tests and especially mutation testing is to validate that every code path has a defined test and hence defined behavior. I wouldn't want to give that up, myself. I'm a huge believer in type safety and immutable objects and other things that prevent architectural misuse.
- Just to share a perspective from erlang/elixir: pattern matching to filter for only happy-path inputs and the principle of “letting it fail” (when the inputs don’t match the expected shape) works really well in some paradigms (in this case, the actor model + OTP, erlang’s 9 9s of uptime, etc). In that kind of architecture you can really only care about the happy path, because the rest is malformed and can be thrown away without issue.
- I'd imagine there would be no need to give up type safety, unit testing and all that though. I'm thinking more about language and tool creators' focus and efforts going mostly into compiler and type safety.
  
  I've seen live updates for things like Go, NodeJS, Vue, Flutter, Angular, and you can recompile a Java class during debugging in Java (with limitations as certain things like static variables and methods need a full restart). More support and integration is certainly better but it doesn't feel like it's in a bad place to me.

This is what Smalltalk is all about, and it has been like that since it’s origin: you basically program in the debugger, you program running, you change something, you proceed the debugger, etc. That’s why technics like TDD, refactoring, and others were developed in Smalltalk and just later translated to other languages (and always lacking, since no one reproduces the live programming experience 100%). As the time passed, attention has moved to other languages and most people not ignores what it was to program like that. But there are still some implementations around: I work with Pharo (https://pharo.org), and I can to say is all what you ask for in this post :)
- Oops… most people now ignores… I meant :)
  
  You can edit comments
- Oh, I completely forgot about smalltalk. Better look into again.
- It's always fun when someone comes up with a new idea of how to write code and it is something smalltalk did in the 80s.

Compose and flutter both have "realtime" previews? And I believe react has something similar? They aren't truely realtime, because usually there is a compile/execute step happening, but they can be fairly good for rapid feedback?
https://docs.flutter.dev/tools/hot-reload
https://blaszcz.uk/live-reload-with-compose-web/
For unit tests, there is https://infinitest.github.io/, which automatically updates your tests as you code. Never used it though.
- I am familiar with hot reloads. What I had in mind was something more fine-grained, not just the UI. A simple example would be that I declared a function signature. Then I write a test. As I start to implement the function, there would be constant feedback visible based on the inputs to the functions from test I wrote. If I declare a variable 'x' by adding function params 'let x = y + z', the feedback view would show a watch expression of x based on the test's input. If I changed it to 'let x = y * z', the watch expression would immediately change. I would be constantly seeing the result of my actions. May be this is asking for too much with the current technology we have. I don't know.
  
  Okay, sounds more like infinitest, but sadly that is Java only.
  I've never really played with it, but jupyter might also be closish to your vision?
  https://jupyter.org/
  Let's you do python stuff and see results of your code near realtime I think?
- infinitest looks interesting though. Will check it out, though I'm not much of a Java dev.

I'm not sure that what developers really, really need is faster programming cycles. Most teams could benefit more by controlling the process - from idea to deployed. How much technical debt is incurred because users/customers can't prioritize features or give accurate requirements, there's way too much WIP, features are huge, releases are huge and infrequent and the feedback cycles are far too long.
So yeah, as programmers it's always cool to look at ways to program faster, but what's the point in programming stuff nobody needs faster? Or programming the wrong things faster?
I'd be willing to be that if you asked any team, "What are the biggest impediments to delivering value to your users faster?", the answer would be that you can't cut code fast enough.

IMO Clojure really shines with a REPL-driven development, where you have a REPL to interact with a live running application. You can have a very productive workflow where you conduct little code experiments and incrementally build up functionality. Typically, in this workflow, you're also sending code (as S-expression forms) to the REPL from your editor to inspect or test the output (some tools have inline display of the output like the Calva Clojure extension for VS Code ). Any tests you come up with in the REPL can usually be turned into unit tests fairly easily (e.g. copy/paste into a tests file) and those tests can be run automatically on source change in the background as well or from the REPL itself. There are also tools on built on top of the REPL that can be used for live inspection of data structures, data visualization/exploration, creating watches on variables, etc. Although the Clojure REPL is best used when developing Clojure applications, I've also found it useful for getting familiar with Java apps such as a complex Spring Framework web app that I had to work on.
For Python, I typically use Jupyter/IPython to hash out and test the functionality I need and I find provides a much superior experience to just using the regular $ python REPL. With PyCharm, the Jupyter integration is really nice and makes the experience a lot more liquid than having to flip between a browser tab and your editor. It's not quite as nice as a REPL integrated with your live running app, but I personally find it more productive than running a debugger or just spamming print statements and modifying and rerunning my Python script 100X.
I really hope that more languages start to better support for more exploratory, experimental development workflows since sometimes I just want to play around to get familiar with using a library or tool in a sandbox. I think there's a lot of room to improve developer productivity through blending and integrating together REPLs, notebook-like environments, hot code reloading and other tools/techniques to enable rapid prototyping and faster feedback for developers. Tests and type checking are very useful when you've figured out what you need and need to have sanity checks, but can be a major hindrance in that "figuring it out" exploratory stage.
- The REPL experience in Clojure is powerful. Calva includes a functional debugger as well and I have to say it all leads to a frictionless experience.
- It's the same with elixir and it's interactive REPL! I really love working with it.

I can imagine there would be challenges with parsing/evaluating incomplete code syntax and functions. So I guess, the whole compiler/translator side has to be thought out from the ground up as well.
What I always found kind of strange in this regard, is that many of the scripting languages (Python, JS etc.), which are good for this workflow, decided to be extremely loose on the syntax front.
Compilers love semicolons and scope braces, because whenever you encounter one, you know what you just parsed is supposed compile. And if it doesn't compile, you can easily narrow down that location and still reason about the rest.
In a REPL, you could potentially tell the user that you don't yet know what velocity is, because that doesn't compile, but the result of the remaining calculations is going to be 6.28 * velocity.

Huh... I've used Guard and TDD to do this in Ruby, works great. Yes you're testing the happy path, but it is easy to define negative tests as well.

Thank you for raising the question. I think it's an important one to think about. I constantly hear about good things about the REPL experience of LISP family languages. You can set up a code fragment (the test in your example) to run constantly in the background as you edit. Then you can jump to the REPL anytime and interact with the state.
I myself am more on the ML-family side of FP, where you'd encode the expected behavior with an expressive type system and work with the type checker (the smart compiler) to implement that behavior.
One important thing to note is that the type checking process is also a fast feedback loop. The difference is that it's often on the abstract level and you're more concerned about the expected behavior instead of the actual behavior.
It's harder to write, but the advantage is that you'll have more confidence once it type checks.
Of course, the two styles are not mutual exclusive, just that the tooling ecosystem will often reflect the culture of that language family. And it's easier to add a simple watch make task, but harder to go the other way around.

What you're saying reminds me a lot of a talk I listened to some time ago, https://www.youtube.com/watch?v=8Ab3ArE8W3s. If you can spare 45 minutes, it's a really good listen. Particularly the debugging section should be interesting for you, but I'm not sure I'd recommend skipping ahead.
The talk is more on a conceptual level but there are some really good concrete examples near the end. Maybe listening can give you a good starting point on what to look for.
- Watching it now. So far he'd been describing exactly what I had in mind. Thanks for this!
  
  Glad that you liked it :)
- Here is an alternative Piped link(s): https://piped.video/watch?v=8Ab3ArE8W3s
  Piped is a privacy-respecting open-source alternative frontend to YouTube.
  I'm open-source, check me out at GitHub.

At one point I was working in python and made a 30 line script to run tests automatically on file change.
Then if it a test starts failing then I would play one sound once.
Then when fixed I played another sound.
The feedback was both lightning fast and didn't require leaving the code.
You can still have it as long as your tests run fast.

I think of this as interactive development, or repl-driven development. You can work this way today in Clojure (frontend, backend, and lately even for scripting via babashka), and with lisps in general - the syntax lends itself to sending expressions to the repl and returning values to your editor.
It’s really the best way (my favorite, at least) to program that i’ve found for exactly the reasons you mentioned - it’s excellent for debugging and ensuring the behavior of small functions with minimal overhead.
Types are frustrating because they lock things up and they don’t guarantee behavior, which is really all a program cares about. I feel similarly about unit tests… it’s extra code locking up your behaviors, so make sure they’re what you actually want! A general problem with types is that you have to commit to some shape early, which can lead to premature design and basically some arbitrary DSL when you just needed a couple functions/transformations. Feels like the problem of OO at times.
On the other side, the trouble (beyond people generally not wanting to read/learn lisps, which is unfortunate) is that repl-driven dev requires that you take care of your tools, which means there’s a tough learning curve and then some maintenance cost for whatever editor you want to use.
At a career-level scale, in my opinion, the investment is well worth it, but it’s a tough thing to figure out early in your career. I expect most devs with a couple years of js/python see types and feel like it’s a huge relief, which is real, and maybe types make sense at a certain team size…
I think people should spend time in several different languages and paradigms - it makes the ones you go back to make more sense :D
- I think elixir/erlang is also in the same class of languages as clojure in that sense. A lot of lisp-like languages tend to go into that trend, I guess. I love working in it.
  May be my headspace was a bit too much in systems that benefit from rapid prototyping. Other class of systems might benefit greatly from type safety and unit tests. Even though, I still felt a bit iffy about unit tests and almost ideological spouting points of it. I struggled with unit testing for a few years and now I just use them for automation of bigger picture behaviour testing. Call them integration tests or whatever.

Something like this? https://codepen.io/darrengriffith/pen/KpKxqR
I can edit the code and see the changes without rewhatevering.

32 comments