D-Expressions: Lisp Power, Dylan Style

Abstract

📜 Abstract

This paper aims to demonstrate that it is possible for a language with a rich, conventional syntax to provide Lisp-style macro power and simplicity. We describe a macro system and syntax manipulation toolkit designed for the Dylan programming language that meets, and in some areas exceeds, this standard. The debt to Lisp is great, however, since although Dylan has a conventional algebraic syntax, the approach taken to describe and represent that syntax is distinctly Lisp-like in philosophy.

Description

✨ Summary

The paper presents a macro system and syntax-manipulation toolkit for Dylan intended to provide Lisp-like expressive power while retaining conventional algebraic syntax. Its central abstraction is the skeleton syntax tree (SST): a lightweight, shape-oriented representation that lies between an unstructured token stream and a language-specific abstract syntax tree. SSTs allow user-defined constructs to be introduced without modifying the core grammar, while preserving nested structure, source locations, syntactic context, and hygiene information.

Dylan’s rewrite-rule macro system uses concrete-syntax patterns with constrained variables and concrete-syntax templates for source-to-source transformation. The paper describes hygienic expansion, referential transparency across namespaces, local rewrite rules, recursive auxiliary rules, identifier concatenation, template calls, and macro-expansion tracing. It also introduces d-expressions, a class hierarchy and library for representing, parsing, constructing, reading, and writing SST fragments. These facilities are presented as analogues of Lisp destructuring, quasiquotation, and read/write operations, adapted to a language with conventional syntax.

The paper additionally evaluates models for procedural macro execution, including runtime evaluation, compiler plugins, and staged compile-time loading. The authors report that the d-expressions library had been implemented and used in the Dylan compiler, while the plugin model of compile-time evaluation was supported; the more integrated compile-time evaluation model remained proposed rather than fully implemented.

The paper is documented by the Open Dylan project as a 1999 technical report. Its proposed template-call extensions were subsequently incorporated into Open Dylan’s macro-system documentation, which explicitly identifies them as originating in an appendix to this paper. (github.com) The paper is also cited as prior work in later research on compile-time metaprogramming and language extension, including work on DSL implementation and dynamically typed object-oriented languages. (tratt.net) Community documentation further reports that the described machinery was used in the Open Dylan compiler and compiler libraries, including facilities for C foreign-function interfaces and other syntax-oriented extensions; this is evidence of implementation impact within the Dylan ecosystem rather than broad industrial adoption. (lobste.rs)