Implementing The Experimental ST Meta-Syntax

2026-07-13

Jul 14, 2026

Creating a new language no longer implies that you have to write a full-blown compiler, too.

As a follow-on to a previous article on architectural comments, this 80-minute, Twitch-like video shows how easy it is to implement new meta-syntaxes over top of existing programming languages.

Starting with a fresh working directory, the PBP toolset is cloned (mostly for its inclusion of the T2T tools). The Ohm-editor is used to rough-in a grammar and to examine the resultant parse tree in a REPL-like manner. Then, the grammar and rewrite rules are iteratively improved until valid Python (and Javascript) are produced.

We strictly focus only on test.st. This allows the implementation process to proceed quickly and to leverage on the capabilities of existing compilers. If one concedes thattest.st represents an interesting view of a part of an architecture (the Tiny PBP kernel in this case), then this video shows how to map that meta-syntax into valid Python and Javascript while, hopefully, making the Design Intent (DI) of this portion of the kernel more clear to readers.

A highlight of this technique is that it is possible to use T2T and OhmJS to “skip” over code. With this trick, you don’t need to write grammar rules for every millimetre of a language. T2T (Ohm, PEG, recursive descent) parses in a top down manner, whereas LR tools like YACC work in a bottom-up manner. The bottom-up approach needs a scanner that tokenizes everything and the associated grammar must then handle every combination in an excruciatingly detailed manner that takes a lot of development effort. The top-down approach lets you skip over text that you don’t want to deal with. This makes it possible to write little “skins” to wrap around existing languages - you don’t have to parse all of the existing language bits - you can just skip over other-language code.

T2T (text to text transmogrification) works in two stages
1. parse
2. rewrite.

This means that the same grammar (step 1. parse, above) can be used to create different rewriters for multiple target languages. In fact, PBP encourages this multiple back end approach, as seen in this snippet from the PBP kernel directory:

whereas simply using shell scripts - as is done in the example herein - doesn’t encourage the multiple back end approach (it can still be done using scripts, but it doesn’t naturally suggest this kind of approach).

Unlike with REGEX, top-down parsing can be used to write recursive rules, so this trick can be used to drill down into nested languages (which, say, use brace brackets, etc.).

Most modern programming languages use a line-oriented syntax, whereas Lisp uses a recursive syntax. Top-down parsing makes it easier to write recursive grammar rules (as opposed to the eBNF and Kleene-star stuff that we’ve become used to) for handling Lisp. Maybe we can invent more recursive-syntax languages and notations? A notation only needs to map onto hardware CPUs, it does not necessarily need to mimic CPU operational characteristics, like most modern programming languages do today.

[disclaimer: test.st is only an isolated snippet of code and DI. It has not yet been bench-tested, hence, this example might contain errors. The point of this video is only to show a technique for implementing meta-syntaxes.]

Appendix

st meta-syntax github
PBP tools github repository (the PBP tools include T2T)
tinypbp (WIP)

Programming Simplicity

Discussion about this post

Ready for more?