Full Code of RyanWelly/lisp-in-rs-macros for AI

Repository: RyanWelly/lisp-in-rs-macros
Branch: master
Commit: 04872f1b090f
Files: 7
Total size: 39.0 KB

Directory structure:
gitextract_377ca60x/

├── .gitignore
├── Cargo.toml
├── EXPLANATION.md
├── LICENSE
├── README.md
└── src/
    ├── lib.rs
    └── macros.rs

================================================
FILE CONTENTS
================================================

================================================
FILE: .gitignore
================================================
/target


================================================
FILE: Cargo.toml
================================================
[package]
name = "lisp-in-rs-macros"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]


================================================
FILE: EXPLANATION.md
================================================
## Overview

The internal_lisp! macro emulates a [SECD machine](https://en.wikipedia.org/wiki/SECD_machine). Lisp lists are represented by ($($elem:tt)*) in Rust macros, ie a list of token trees.
Using the SECD machine for a Lisp is a very old idea; Lispkit Lisp, originally written in the 80s, famously used a SECD machine as a compiler target. However, we are not really compiling to target the SECD machine, we're just directly interpreting lisp expressions with one.
The SECD machine has four components: 
- a stack to store intermediate results,
- an environement which stores values bound to names,
- a control stack, the top of which is the expression currently being evaluated and determines the next state of the machine.
- a dump, which saves the current state of the machine before evaluating the body of a closure. It's the "call stack" of the machine.

Rust macro's make it surprisingly easy to write state transitions as macro match arms. 

## Internals

In our SECD machine, we have the stack, the environment, the control, and the dump (which explains the name!).
Intermediate results are stored on the stack, the environment is an associative array of variable bindings, the control is a stack of lisp expressions to evaluate, and the dump holds state when we're evaluating the inner expression of a lambda.

When you write `lisp!(CONS (QUOTE A) (QUOTE (B)))` it gets expanded to  


`internal_lisp!(stack: [] env: {} control: ((ATOM (QUOTE A))) dump: []).  `


First, we evaluate the arguments to functions before we evaluate the actual function. So we rewrite this to:


`internal_lisp!(stack: [] env: {} control: ( (QUOTE A) CONS ap) dump: []).  `

Which means that now the two arguments to the CONS function are at the top of the control stack. The next expression to evaluate is `(QUOTE A)`. Now, QUOTE is a special form; this means that we do something different than our usual method of evaluating the arguments. QUOTE will place its unevalated argument straight onto the stack.

`internal_lisp!(stack: [A] env: {} control: ( ) CONS ap) dump: []).  `

Now, ATOM will evaluate to some kind of primitive function token. In my lisp I call it __ATOM:

`internal_lisp!(stack: [__ATOM A] env: {} control: ( ap) dump: []).  `
Now, we have ap (short for apply) on the top of the control stack. This means that we try and apply the function on top of the stack, to its arguments. In this case, CONS is a primitive so it's quite straightforward. The macro arm for this would look like:

```rust
    //ATOM 
    (stack: [__ATOM ($atom:ident) $($stacks:tt)*] env: $env:tt control: [ap $($controls:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: [ TRUE $($stacks)*] env: $env control: [$($controls)*] dump: $dump)
    };

    (stack: [__ATOM ($not_atom:tt) $($stacks:tt)*] env: $env:tt control: [ap $($controls:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: [ () $($stacks)*] env: $env control: [$($controls)*] dump: $dump)
    };
```
macro_rules! tries to match rules sequentially. So if we apply ATOM to an atom (an ident) then we push TRUE onto the top of the stack, and that rule doesn't match, it will instead match on the next rule (atom applied to a token tree, ie a list) and place the empty list on top of the stack. So our final expansion would look like 

`internal_lisp!(stack: [TRUE] env: {} control: () dump: []).  `

And both the dump and control are empty, so we halt. TRUE is the evaluated output of our lisp code, so we stringify that and return it as our final expanded value. 




For evaluating lisp code that only involves simple primitives like quote, cons, cdr and friends, the control and the stack is all we need. For evaluating lambdas, we add an environment and a dump. Let's walk through how we evaluate `((LAMBDA (X) X) (QUOTE A))`, ie applying the identity function to an argument.




`internal_lisp!(stack: [] env: [] control: ( ((LAMBDA (X) X) (QUOTE A))  ) dump: []).  `




As usual, we first evaluate the arguments before we evaluate the function. I'll skip that step:


`internal_lisp!(stack: [A] env: [] control: ( (LAMBDA (X) X) ap  ) dump: []).  `


So we have the atom A on the stack, and we want to apply the identity function to it. Our first step is to evalute the lambda itself, which results in a closure which stores the code, current environment, and a variable reference:


`internal_lisp!(stack: [ [{X}, X, []] A] env: [] control: ( ap  ) dump: []).  `


Now the top of the stack is a closure. How do we apply the closure to the argument? First we save the current stack, current control, and current env to the dump, and then we replace the control stack with the inner "code" of the closure, replace the env with our closure's env, and bind the variable X to our argument A.



`internal_lisp!(stack: [] env: [X: A] control: ( X ) dump: [((), (), ())]).  `


Now we have a single atom on top of the control; so we look it up in the environment. This evaluates to A, so:


`internal_lisp!(stack: [A] env: [X: A] control: ( ) dump: [((), (), ())]).  `


Now control is empty, so we go up a level and return to the previous code by slurping it up from the dump (this is how our machine supports calling arbitary nested lambdas).


`internal_lisp!(stack: [A] env: [] control: ( ) dump: []).  `



We finish with a single value on the stack, and our control and dumps are empty, so we stop execution. We've just successfully used the identity function!

## Some macro hacks

One lisp function that is slightly tricky to implement is EQ, testing if two lisp objects are equal. In theory, we want something like
```rs
macro_rules! lisp_equality {
    ($x:tt $x:tt) => {TRUE} 
    ($x:tt $y:tt) => {FALSE}
} 
```
where the first arm matches if the two token trees are equal, and the second matches otherwise. But this is not possible in rust macros; you cannot have duplicate bindings. We get around this quite easily by generating a macro in our macro:

```rs
macro_rules! lisp_equality {
    ($x:tt $y:tt) => {
        macro_rules! inner {
            ($x $x) => {TRUE},
            ($x $y) => {FALSE},
        };
        inner!($x $y)
    };
}
```

In the actual interpreter, we use this trick to "branch" on the truth value of a value; the rule in question looks like:

```rs
 // EQ 
    (stack: [__EQ ($val1:tt $val2:tt) $($stacks:tt)*] env: $env:tt control: [ap $($controls:tt)*] dump: $dump:tt) => {{
        macro_rules! __internal_eq {
            ($val1 $val1) => {internal_lisp!(stack: [TRUE $($stacks)*] env: $env control: [$($controls)*] dump: $dump)};
            ($val1 $val2) => {internal_lisp!(stack: [() $($stacks)*] env: $env control: [$($controls)*] dump: $dump)};
        }
        __internal_eq!($val1 $val2)}
    };

```
So if the top two values on the stack are equal, we expand to an `internal_lisp!` call where the top two operands have been consumed and replaced with a TRUE; otherwise we expand to a call where it's been replaced with an empty list (our version of false). Crucially, this internal macro trick works even with many EQ calls; Rust allows us to redefine a macro many times, 
each time we invoke the `__internal_eq!` rust uses the most recently defined macro, which is the one we just defined and we just expanded to.


The other macro hack is not needed, but is probably more "efficient"; looking up values in our enviroment. You could of course do this recursively (lisp style, think association lists) but it's nicer to use rust macros as designed:

```rs
macro_rules! env_example {
    ($arg:ident [$($var_binding:ident : $exp:tt),*]) => {
        {macro_rules! inner_test {
            $(
            ($var_binding) => {$exp};
            )*
            ($not_found:ident) => {error!("couldn't find value in env ")}; 
        }
        inner_test!($arg)}
    }
}
env_example!(test [x: 5, y: 7, test: "hello"]) //evaluates to "hello"
```
We use another trick, generating yet another inner macro. Just like the previous trick with EQ, in the actual implmentation we use this trick to branch.
In practise, this does lead to a explosion in the size of the generated, since the corpses of many generated macros litter the final result.



## Metacircular evaluation and Recursion

Currently, the lisp implemented seems to be missing a key element; recursion. We can get around this by using the Y combinator, and implement a lisp interpreter in our lisp, like the one listed in README.md. But this is brutally inefficient; everytime our lisp creates a closure, it copies the entire environment into that closure. This quickly becomes too much to handle for rustc, as for some reason rustc isn't optimised for declarative macros generating millions of tokens and immediately passing it to a macro.



================================================
FILE: LICENSE
================================================
MIT License

Copyright (c) 2024 Ryan Winter

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

================================================
FILE: README.md
================================================
# `lisp-in-rs-macros`


A simple, lexically scoped Lisp interpreter that operates fully in Rust's declarative macros. The `lisp!` macro expands to the lisp value computed by the code, and then stringifies it. This means that `lisp!(CAR (CONS (QUOTE A) (QUOTE (B))))` expands to the string "A" and that all this computation happens at compile time by rustc expanding macros. 

## Why

It's a lisp interpreter written fully in Rust's macros, I think that's pretty cool. It's also less than 250 lines, which is neat.


## Example
```rust
let output = lisp!(CAR (LIST (QUOTE A) (QUOTE B) (QUOTE C)));
assert_eq!(output, "A"); 

lisp!(PROGN
    (DEFINE message (LAMBDA () (QUOTE "hello there")))
    (DISPLAY (message))
    (DEFINE NOT (LAMBDA (X) (COND (X NIL) (TRUE TRUE))) )
    (DISPLAY (NOT NIL))
); // will print "hello there" and "TRUE"
// "DISPLAY" forms first evaluate their arguments, then expand to a println!("{}", stringify!(evaled_argument))

```

As another fun example, here is a quine:

```rust
lisp!
       ((LAMBDA (s) (LIST s (LIST (QUOTE QUOTE) s)))
       (QUOTE (LAMBDA (s) (LIST s (LIST (QUOTE QUOTE) s)))));
```
This code expands to:
```rust
stringify!(((LAMBDA (s) (LIST s (LIST (QUOTE QUOTE) s)))
       (QUOTE (LAMBDA (s) (LIST s (LIST (QUOTE QUOTE) s))))));
```
In other words, the code evaluates to itself. Isn't that wonderful?





## Recursion

This lisp does not currently support any explicit form of recursion. Luckily, explicit recursion is not needed, all we need is lambda.

You can write a simple function that appends two lists by using self application:


```rust
lisp!(PROGN
(DEFINE append 
    (LAMBDA (self X Y) 
        (COND 
            ((EQ X NIL) Y) 
            (TRUE (CONS (CAR X) (self self (CDR X) Y))) 
        )))
(append append (QUOTE (A B)) (QUOTE (C D)))

)
```
This results in "(A B C D)". The append function does not mention `append` in its body, yet we can call it recursively. Wonderful!


## Notes for use
The lisp! macro only evaluates a single expression; if you want to evaluate multiple expressions, use `(PROGN expr1 expr2 expr3)`. This evaluates all the expressions, and returns the value of the last expression. The DISPLAY form evaluates a single expression, then generates a `println!("{}", stringify!(...))` statement which prints the stringified version of the tokens. The empty list is not self evaluating, you can use `NIL` or `(QUOTE ())` to obtain an empty list value. The empty list is the sole "falsy" object. 
Dotted lists aren't supported, cons assumes its last argument is a list. The define form can be used anywhere and evaluates to the empty list, but does not support recursion. TRUE is the only self evaluating atom (that isn't a function).


## Supported forms
```rust
DEFINE 
QUOTE 
LAMBDA 
LET
PROGN 
CAR 
CDR 
CONS
LIST
EQ
ATOM
APPLY
```

Note: dotted lists are not supported, CONS assumes its latter argument is a list. Define does not handle recursive definitions, it's more like internal definitions in Scheme than a true lispy define.


## Metacircular interpreter

Here is a lisp interpreter written in my lisp:

```rust
lisp!(PROGN
            // Y "combinator" for two arguments
        (DEFINE Y2 
                        (LAMBDA (h)
                            ((LAMBDA (x) (h (LAMBDA (a b) ((x x) a b))))
                                (LAMBDA (x) (h (LAMBDA (a b) ((x x) a b)))))))
        
        (DEFINE CADR (LAMBDA (X) (CAR (CDR X))))
        (DEFINE CAAR (LAMBDA (X) (CAR (CAR X))))
        (DEFINE CADAR (LAMBDA (X) (CAR (CDR (CAR X)))))
        (DEFINE CADDR (LAMBDA (X) (CAR (CDR (CDR X)))))
        (DEFINE CADDAR (LAMBDA (X) (CAR (CDR (CDR (CAR X))))))
        (DEFINE CAADAR (LAMBDA (X) (CAR (CAR (CDR (CAR X))))))

        (DEFINE ASSOC (Y2 (LAMBDA (ASSOC) (LAMBDA (X ENV) 
                        (IF (EQ (CAAR ENV) X) (CADAR ENV) (ASSOC X (CDR ENV)))
                    )))
                )


            
        (DEFINE eval (Y2 (LAMBDA (EVAL) (LAMBDA (E A) 
                (COND
                    ((ATOM E) (ASSOC E A))
                    ((ATOM (CAR E)) 
                        (COND 
                            ((EQ (CAR E) (QUOTE quote)) (CADR E))
                            ((EQ (CAR E) (QUOTE atom)) (ATOM (EVAL (CADR E) A)))
                            ((EQ (CAR E) (QUOTE car)) (CAR (EVAL (CADR E) A)))
                            ((EQ (CAR E) (QUOTE cdr)) (CDR (EVAL (CADR E) A)))
                            ((EQ (CAR E) (QUOTE equal)) (EQ (EVAL (CADR E) A) (EVAL (CADDR E) A)))
                            ((EQ (CAR E) (QUOTE cons)) (CONS (EVAL (CADR E) A) (EVAL (CADDR E) A)))
                            (TRUE (EVAL (CONS (ASSOC (CAR E) A) (CDR E)) A)) 
                        )
                    )
                    ((EQ (CAAR E) (QUOTE lambda)) (EVAL (CADDAR E) (CONS (LIST (CAADAR E) (EVAL (CADR E) A)) A)  )) //Evaluate the inner expression of the lambda, in the environment with the argument bound to the parameter
                
                )
            ))))

        (eval (QUOTE (quote (A))) NIL)
        // (eval (QUOTE (atom (quote A))) NIL )
        // (eval (QUOTE (cdr (cdr (quote (A B))))) NIL)
        // (eval (QUOTE (cons (quote a) (quote (a)))) NIL)
        // (eval (QUOTE ((lambda (x) (quote a)) (quote b))) NIL)
        (eval (QUOTE ((lambda (X) X) (quote a))) NIL)

        );
```
It appears to work, but trying to evaluate `((lambda (X) X) (quote a))` in the interpreter takes more than 30 seconds and generates far more than a million tokens before cargo gets sigkilled. Using the explicit y combinator for recursion isn't particularly efficient here! To fix this, I should add an explicit recursion primitive. If you want a wonderful walktrhough of how to write something like a metacircular evaluator, Paul Graham's "Roots of Lisp" is awesome.



## Technical explanation

Look at EXPLANATION.md. The macro essentially simulates a SECD machine, which is a simple stack-basd abstract machine for evaulating lambda calculus terms. 


## Awesome resources
- Functional Programming: Application and Implementation by Peter Henderson
- Ager, Mads Sig, et al. "A functional correspondence between evaluators and abstract machines." Proceedings of the 5th ACM SIGPLAN international conference on Principles and practice of declaritive programming. 2003.
- The Implementation of Functional Programming Languages by Simon Peyton Jones
- Anything Matt Might has ever written about lisp on his blog (https://matt.might.net)

## TODO

- Add letrec
- Add recursive defines





================================================
FILE: src/lib.rs
================================================
#![recursion_limit = "1000"]
#[macro_use]
mod macros;


================================================
FILE: src/macros.rs
================================================
#[macro_export]
macro_rules! internal_lisp {

    // Evaluate special forms

    // QUOTE special form
    (stack: [$($stack_entries:tt)*] env: $env:tt control: [(QUOTE $x:tt)$($rest:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: [$x $($stack_entries)*] env: $env control: [$($rest)*] dump: $dump)
    };

    // LAMBDA special form - saves closure onto stack with current env
    // When we encounter a lambda, we wrap it up into a closure and push it to the stack.
    // closures store the names of any arguments, the inner expression, and the current env.

    (stack: [$($stacks:tt)*] env: $env:tt control: [(LAMBDA ($($names:ident)*) $T:tt) $($controls:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: [[{$($names)*}, $T, $env] $($stacks)*] env: $env control: [$($controls)*] dump: $dump)
    };


    // Let form
    // (LET (x1 e1) ... (xn en) e) === ((LAMBDA (x1 ... xn ) e ) e1 ... en)
    // simple transformation, nothing weird.

    // since the naive "(LET $(($xn:tt $en:tt))* $e:tt)" leads to local ambiguity, we need to split it up into the two possibilities.

    (stack: $stack:tt env: $env:tt control: [(LET ($(($xn:tt $en:tt))*) $e:ident) $($controls:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: $stack env: $env control: [  ((LAMBDA ($($xn)* ) $e) $($en)*) $($controls)* ] dump: $dump )
    };
    (stack: $stack:tt env: $env:tt control: [(LET ($(($xn:tt $en:tt))*) ($($forms:tt)*)) $($controls:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: $stack env: $env control: [  ((LAMBDA ($($xn)* ) ($($forms)*)) $($en)*) $($controls)* ] dump: $dump )
    };




    // COND special form
    // (COND (p1 e1) .... (pn en)) => p1 :: (IFS e1 (COND (p2 e2) ... (pn en)) )
    // (COND (p1 e)) => p1 :: (IFS e nil)



    (stack: $stack:tt env: $env:tt control: [(COND ($p:tt $e:tt) $(($pn:tt $en:tt))+ )  $($control:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: $stack env: $env control: [$p {__IFS $e (COND $(($pn $en))+  ) } $($control)*] dump: $dump)
    }; 
    (stack: $stack:tt env: $env:tt control: [(COND ($p:tt $e:tt)) $($control:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: $stack env: $env control: [$p {__IFS $e NIL } $($control)*] dump: $dump)
    }; // If we reach the end of the COND without a predicate matching, we return nil

    (stack: [() $($stack:tt)*] env: $env:tt control: [{__IFS $p:tt $e:tt } $($control:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: [$($stack)*] env: $env control: [ $e $($control)*] dump: $dump)
    }; // NIL/false case for the __IFS

    (stack: [$val:tt $($stack:tt)*] env: $env:tt control: [{__IFS $p:tt $e:tt } $($control:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: [$($stack)*] env: $env control: [ $p $($control)*] dump: $dump)
    }; // truthy case for the __IFS - everything but nil/empty list is true


    // If special form - simple desugaring to internal primitive
    // (IF p e1 e2) === (COND (p e1) (TRUE e2))


    (stack: $stack:tt env: $env:tt control: [(IF $p:tt $e1:tt $e2:tt)  $($control:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: $stack env: $env control: [$p {__IFS $e1 $e2 } $($control)*] dump: $dump)
    };







    // Define special form
    // semantics: can use at top level or any level at all, (define name exp) just evalutes exp in the current env, then binds it to `name` in the current env.
    // Returns nil.
    // (DEFINE name expr) => expr :: {__DEFINE name}
    // TODO - enable recursive defines somehow - maybe specialise when there is a closure on tbe stack and {__DEFINE: $name} on the control
    // maybe return a "recursive closure" (fixpoint?)




    (stack: $stack:tt env: $env:tt control: [ (DEFINE $name:ident $exp:tt) $($rest:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: $stack env: $env control: [$exp {__DEFINE:$name} $($rest)*] dump: $dump)
    };


    // TODO - wrap defined lambdas in the z combinator to get them working recursively 

    (stack: [$value:tt $($stack:tt)*] env: [$($key:ident : $val:tt)*] control: [{__DEFINE:$name:ident} $($rest:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: [() $($stack)*] env: [$name: $value $($key : $val)*] control: [$($rest)*] dump: $dump )
    };

    //PROGN special form
    // basically a hack to be able to write multiple lisp expressions
    // (PROGN e1 e2 .. eN) evalutes to the value of eN, but evaluates each expression in order


    (stack: $stack:tt env: $env:tt control: [(PROGN $($forms:tt)*) $($control:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: $stack env: $env control: [(LIST $($forms)*) {__LAST} $($control)* ] dump: $dump)
    };

    (stack: [($last:tt) $($stack:tt)*] env: $env:tt control: [{__LAST} $($control:tt)*] dump: $dump:tt ) => {
        internal_lisp!(stack: [$last $($stack)*] env: $env control: [$($control)*] dump: $dump)
    };
    (stack: [($first:tt $($rest:tt)+) $($stack:tt)* ] env: $env:tt control: [{__LAST} $($control:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: [($($rest)+) $($stack)*] env: $env control: [{__LAST} $($control)*] dump: $dump)
    };

    // (t1 t2) => t2::t1::ap
    // (f a1 a2 a3 ..) expands to essentially (LIST a1 a2 a3 ...) :: f :: ap

    (stack: $stack:tt env: $env:tt control: [($op:tt $($args:tt)*) $($rest:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: $stack env: $env control: [NIL $($args {cons})* $op ap $($rest)* ] dump: $dump)
    };


    //quick dirty hack for fitting arguments into a list; add a new `cons` opcode, similar to the ap, which just cons the top two values on the stack.
    // (f a1 a2 a3) => NIL :: a1 :: {cons} :: a2 :: {cons} :: a3 :: {cons} :: f :: ap
    // this has the massive advantage of making applying functions very straightforward; you don't need to slurp the right number of arguments off the stack.
    
    
    (stack: [ $x:tt () $($stacks:tt)*] env: $env:tt control: [{cons} $($rest:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: [($x) $($stacks)*] env: $env control: [$($rest)*] dump: $dump)
    };
    (stack: [$y:tt ($($vals:tt)*)  $($stacks:tt)*] env: $env:tt control: [{cons} $($rest:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: [($($vals)* $y) $($stacks)* ] env: $env control: [$($rest)*] dump: $dump)
    };

    // Evaluate primitives - top of the stack


    // CAR
    (stack: [__CAR (($car:tt $($cdr:tt)*)) $($stack_entries:tt)*] env: $env:tt control: [ap $($rest:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: [$car $($stack_entries)*] env: $env control: [$($rest)*] dump: $dump)
    };

    // CDR
    (stack: [__CDR (($car:tt)) $($stack_entries:tt)*] env: $env:tt control: [ap $($rest:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: [() $($stack_entries)*] env: $env control: [$($rest)*] dump: $dump)
    };
    (stack: [__CDR (($car:tt $($cdrs:tt)* )) $($stack_entries:tt)*] env: $env:tt control: [ap $($rest:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: [($($cdrs)*) $($stack_entries)*] env: $env control: [$($rest)*] dump: $dump)
    };

    // DISPLAY
    (stack: [__DISPLAY ($val:tt) $($stacks:tt)*] env: $env:tt control: [ap $($controls:tt)*] dump: $dump:tt) => {
        {println!("{}", prettify_lisp!($val));
        internal_lisp!(stack: [$val $($stacks)*] env: $env control: [$($controls)*] dump: $dump)}
    };

    //ATOM 
    (stack: [__ATOM ($atom:ident) $($stacks:tt)*] env: $env:tt control: [ap $($controls:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: [ TRUE $($stacks)*] env: $env control: [$($controls)*] dump: $dump)
    };

    (stack: [__ATOM ($not_atom:tt) $($stacks:tt)*] env: $env:tt control: [ap $($controls:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: [ () $($stacks)*] env: $env control: [$($controls)*] dump: $dump)
    };

    // EQ 
    (stack: [__EQ ($val1:tt $val2:tt) $($stacks:tt)*] env: $env:tt control: [ap $($controls:tt)*] dump: $dump:tt) => {{
        macro_rules! __internal_eq {
            ($val1 $val1) => {internal_lisp!(stack: [TRUE $($stacks)*] env: $env control: [$($controls)*] dump: $dump)};
            ($val1 $val2) => {internal_lisp!(stack: [() $($stacks)*] env: $env control: [$($controls)*] dump: $dump)};
        }
        __internal_eq!($val1 $val2)}
    };

    // CONS 
    (stack: [__CONS ($val:tt ()) $($stack_entries:tt)*] env: $env:tt control: [ap $($rest:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: [ ($val)  $($stack_entries)*] env: $env control: [$($rest)*] dump: $dump)
    };
    (stack: [__CONS ($val:tt ($($list:tt)*)) $($stack_entries:tt)*] env: $env:tt control: [ap $($rest:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: [ ($val $($list)*)  $($stack_entries)*] env: $env control: [$($rest)*] dump: $dump)
    };

    // LIST
    (stack: [__LIST ($($vals:tt)*) $($stack_entries:tt)* ] env: $env:tt control: [ap $($rest:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: [($($vals)*) $($stack_entries)*] env: $env control: [$($rest)*] dump: $dump)
    };

    // APPLY
    (stack: [__APPLY ($f:tt ($($args:tt)*))  $($rest:tt)* ] env: $env:tt control: [ap $($control:tt)*] dump: $dump:tt) => {
        internal_lisp!(stack: [$f ($($args)*) $($rest)*] env: $env control: [ap $($control)*] dump: $dump )
    };



    // Deal with closures

    // tail recursion (?) - if the last entry on the control stack is ap, then we don't save anything to the dump (because otherwise we'd be saving an state to the dump
    // with an empty control, which would immediately get popped which reached anyway)
    // TODO: test
    (stack: [ [{$($vars:ident)*}, $T:tt, [$($key:ident : $value:tt)*]] ($($args_list_entry:tt)*) $($stacks:tt)*]  env: $env:tt control: [ap ] dump: $dump:tt) => {
        internal_lisp!(stack: [] env: [$($vars : $args_list_entry )* $($key: $value)*] control: [$T] dump: $dump)
        
    };


    //errors if the number of vars is not the same as the number of arguments given.
    (stack: [ [{$($vars:ident)*}, $T:tt, [$($key:ident : $value:tt)*]] ($($args_list_entry:tt)*) $($stacks:tt)*]  env: $env:tt control: [ap $($controls:tt)*] dump: [$($dump:tt)*] ) => {
        internal_lisp!(stack: [] env: [$($vars : $args_list_entry )* $($key: $value)*] control: [$T] dump: [  ([$($stacks)*], $env, [$($controls)*]) $($dump)*])
        
    };

    // done processing the current closure, pop stack/env/control off the dump and continue evaluating

    (stack:[$val:tt $($stack_entries:tt)*] env: $env:tt control: [] dump: [ ([$($prev_stack:tt)*], $prev_env:tt, $prev_control:tt) $($rest_of_dump:tt)* ]  ) => {
        internal_lisp!(stack: [$val $($prev_stack)*] env: $prev_env control: $prev_control dump: [$($rest_of_dump)*])
    };


    // Evaluate symbol in environment
    (stack: [$($stack_entries:tt)*] env: [$($key:ident : $val:tt)*] control: [$symb:ident $($rest:tt)*] dump: $dump:tt) => {
        {macro_rules! evaluate_in_env {
            $(
                ($key, $stack:tt, $env: tt, $control:tt) => {internal_lisp!(@fix stack: $val $stack env: $env control: $control dump: $dump)};
            )*
            ($symb, $stack:tt, $env: tt, $control:tt) => {error!(Lisp error: cannot find $symb in env)}
        }
        evaluate_in_env!($symb, [$($stack_entries)*], [$($key : $val )*], [$($rest)*]) } // we do this cursed expansion here so we can pass the entire $($rest)* capture groups along, instead of repeating one by one.
    };
    (@fix stack: $top:tt [$($stack_entries:tt)*] env: [$($key:ident : $val:tt)*] control: [$($rest:tt)*] dump: $dump:tt) => { //needed to avoid some fuckery in the first statement, fixes up the formatting.
        internal_lisp!(stack: [$top $($stack_entries)*] env: [$($key : $val)*] control: [$($rest)*] dump: $dump)
    };


    (stack: [$top:tt $($rest:tt)*] env: [$($envs:tt)*] control: [] dump: []  ) => {
        prettify_lisp!($top)
    }; //Termination
}


// Pretty printing to make the printed representation of closures nicer. 
macro_rules! prettify_lisp {
    ([{$($vars:ident)*}, $T:tt, $env:tt]) => {stringify!([PROC ($($vars)*) $T ])};
    ($($toks:tt)*) => {stringify!($($toks)*)}; //for anything other than a closure, just return 
}

// Helper error handler macro
// either causes a compiler error with the error message, or evalutes program to a string of the error message. 
macro_rules! error {
    ($($toks:tt)+) => {compile_error!(stringify!($($toks)+))}; // program halts rust compilation with the error message
    ($($toks:tt)*) => {println!("{}", stringify!($($toks)*))}; // compilation continues, but lisp executation evaluates to a error string 

}

macro_rules! lisp { //call internal_lisp! with the default env
    ($($toks:tt)*) => {internal_lisp!(stack: []
        env: [CAR: __CAR
        CDR: __CDR
        ATOM: __ATOM
        DISPLAY:__DISPLAY
        EQ: __EQ
        CONS: __CONS
        LIST: __LIST
        APPLY: __APPLY
        NIL: ()
        TRUE: TRUE]
        control: [($($toks)*)]
        dump: [] )};
}


#[cfg(test)]
mod tests {
    #[test]
    fn primitive_tests() {
        assert_eq!(lisp!(ATOM(QUOTE X)), stringify!(TRUE));
        assert_eq!(lisp!(ATOM(QUOTE(X))), stringify!(()));
        assert_eq!(
            lisp!(ATOM
            (CONS 
            (QUOTE X) NIL)),
            stringify!(())
        );
        assert_eq!(lisp!(QUOTE X), "X");
        assert_eq!(lisp!((LAMBDA(x)x)(QUOTE gibbergabber)), "gibbergabber");
        assert_eq!(lisp!(CAR(QUOTE(X))), "X");
        assert_eq!(lisp!(CDR(QUOTE(X))), "()");
        assert_eq!(
            lisp!(CAR(CDR(QUOTE(COMPLEX(QUOTE(A)(B)))))),
            stringify!((QUOTE(A)(B)))
        );

        assert_eq!(lisp!(CDR (QUOTE (A B C))), stringify!((B C)));
        assert_eq!(lisp!(ATOM(QUOTE(QUOTE(A)(B)))), "()");

        assert_eq!(lisp!(DISPLAY(CAR(QUOTE(X)))), "X");

        assert_eq!(lisp!((LAMBDA (Y) (CAR Y) ) (LIST NIL NIL NIL)), stringify!(()));
    }

    #[test]
    fn multi_args() {
        assert_eq!(
            lisp!(EQ 
            (QUOTE A) (QUOTE A)),
            "TRUE"
        );
        assert_eq!(lisp!(EQ (QUOTE A) (QUOTE B)), "()");
        assert_eq!(lisp!(EQ CAR CAR), "TRUE");
        assert_eq!(lisp!(EQ (LAMBDA (X) X) (LAMBDA (X) X)), "TRUE");
        assert_eq!(lisp!(EQ (LAMBDA (Y) Y) (LAMBDA (X) X)), "()");

        assert_eq!(lisp!(CONS (QUOTE A) (QUOTE ())), stringify!((A)));
        assert_eq!(lisp!(CONS (QUOTE A) (QUOTE (B C))), stringify!((A B C)));
        assert_eq!(lisp!(CONS (QUOTE A) (QUOTE (B))), stringify!((A B)));
        assert_eq!(
            lisp!(CONS (QUOTE A) (CONS (QUOTE B) NIL)),
            stringify!((A B))
        );
        assert_eq!(lisp!(PROGN
        (DEFINE test NIL)
        (EQ test NIL)
        (EQ (QUOTE A) (QUOTE A))
        ), "TRUE");
    }

    #[test]
    fn list() {
        assert_eq!(lisp!(LIST (QUOTE A) (QUOTE B)), stringify!((A B)));
        assert_eq!(
            lisp!(LIST (QUOTE A) (QUOTE B)),
            lisp!(CONS (QUOTE A) (CONS (QUOTE B) NIL))
        );
        assert_eq!(
            lisp!(LIST (QUOTE A) (LIST (QUOTE (A B C)))),
            stringify!((A ((A B C))))
        );
        assert_eq!(lisp!(CAR (LIST (QUOTE A) (QUOTE B))), stringify!(A));
        lisp!(LIST (CAR (QUOTE (A B C))));

    }

    #[test]
    fn progn() {
        assert_eq!(lisp!(PROGN (QUOTE A) (QUOTE B)), stringify!(B));
        let test = lisp!(PROGN (DEFINE A (QUOTE B)) (CONS A NIL)); 
        assert_eq!(test, stringify!((B)));
    }

    #[test]
    fn conditionals() {
        assert_eq!(
            lisp!(COND
                ((EQ (QUOTE A ) (QUOTE B))  (QUOTE B)) 
                ((EQ (QUOTE A ) (QUOTE A))  (QUOTE A))),
            "A"
        );
        assert_eq!(
            lisp!(COND(
                (EQ (QUOTE A ) (QUOTE A))   (QUOTE B))),
            "B"
        );
        assert_eq!(
            lisp!(COND ((EQ (QUOTE A) (QUOTE B)) (QUOTE FIRST))
                                ((ATOM (QUOTE A)) TRUE)),
            "TRUE"
        );

        assert_eq!(lisp!(IF (EQ NIL NIL) (QUOTE A) NIL), stringify!(A));


        // let test = lisp!(COND(EQ)); //incorrect forms will usually just fail with some kind of error. 
    }

    #[test]
    fn define_tests() {
        assert_eq!(lisp!(DEFINE A NIL), stringify!(()));
    }
    #[test]
    fn non_terminating() {
        // This lisp term should never terminate.
        // lisp!((LAMBDA (X) (X X))(LAMBDA (Y) (Y Y )));
    }

    #[test]
    fn lombda() {
        assert_eq!(lisp!((LAMBDA (X Y) (CONS X Y)) (QUOTE A) NIL), stringify!((A)));
        
        assert_eq!(lisp!(PROGN 
            (DEFINE list3 (LAMBDA (X Y Z) (LIST X Y Z)))
            (DEFINE constant (QUOTE A))
            (DISPLAY (list3 constant NIL constant))
            (DEFINE print_a (LAMBDA () (DISPLAY (QUOTE A))))
            (print_a)
        ), "A");

        assert_eq!(lisp!(DISPLAY (LET ((x (QUOTE BANANA))) x)), "BANANA");
        assert_eq!(lisp!(LET ((X (QUOTE A)) (Y NIL)) (CONS X Y)), stringify!((A)));

        assert_eq!(lisp!((CAR(LIST (LAMBDA (X) X) (LAMBDA (X) NIL))) (QUOTE swim)), stringify!(swim));
        dbg!(lisp!(LAMBDA () (DISPLAY NIL))); 
    }

    #[test]
    fn apply() {
        assert_eq!(lisp!(APPLY LIST (QUOTE (A B))), stringify!((A B)));
    }


    #[test]
    fn append_example() {

        let test = lisp!(LET ((APPEND (LAMBDA (s X Y) (IF (EQ X NIL) Y (CONS (CAR X) (s s (CDR X) Y)))))) (APPEND APPEND (QUOTE (A B)) (QUOTE (C D))));
        assert_eq!(test, stringify!((A B C D)));

    }
    #[test]
    fn utility_functions() {
        let test = lisp!(PROGN
            (DEFINE IS_NULL (LAMBDA (X) (EQ X NIL)))
            (DEFINE append (LAMBDA (s X Y) 
            (PROGN
                (IF (EQ X NIL)
                Y  
                (CONS (DISPLAY (CAR X)) (s s (CDR X) Y))
                )
            )))
            (append append (QUOTE (ma da) ) (QUOTE (A B)))
        );
        assert_eq!(test, stringify!((ma da A B)));
        assert_eq!(lisp!((LAMBDA (s X Y) 
                (IF (EQ X NIL)
                Y  
                (CONS (DISPLAY (CAR X)) (s s (CDR X) Y))
                )
            ) (LAMBDA (s X Y) 
                (IF (EQ X NIL)
                Y  
                (CONS (DISPLAY (CAR X)) (s s (CDR X) Y))
                )
            ) (QUOTE (A B) ) (QUOTE (h a))), stringify!((A B h a)));
    }
    #[test]
    fn recursion() {
        let test = lisp!(PROGN
            (DEFINE Y
                (LAMBDA (h)
                    ((LAMBDA (x) (h (LAMBDA (a b) ((x x) a b))))
                    (LAMBDA (x) (h (LAMBDA (a b) ((x x) a b)))))))
            (DEFINE APPEND (Y (LAMBDA (append) (LAMBDA (w z) (IF (EQ w NIL) z (CONS (CAR w) (append (CDR w) z)))))))
            (APPEND (QUOTE (A)) (QUOTE (B C)))
            
        ); 
        assert_eq!(test, stringify!((A B C)));

    }
}

#[cfg(test)]
mod metacircular {
    


    #[test]
    fn meta() {
        // // simple lisp interpreter - only supports lambda with one parameter
        // // At the moment, using y combinator for recursion causes an explosion in code size, exceeding far more than a million tokens and leading to cargo getting sigkilled
        // let test: &str = lisp!(PROGN
        //     // Y "combinator" for two arguments
        // (DEFINE Y2 
        //                 (LAMBDA (h)
        //                     ((LAMBDA (x) (h (LAMBDA (a b) ((x x) a b))))
        //                         (LAMBDA (x) (h (LAMBDA (a b) ((x x) a b)))))))
        
        // (DEFINE CADR (LAMBDA (X) (CAR (CDR X))))
        // (DEFINE CAAR (LAMBDA (X) (CAR (CAR X))))
        // (DEFINE CADAR (LAMBDA (X) (CAR (CDR (CAR X)))))
        // (DEFINE CADDR (LAMBDA (X) (CAR (CDR (CDR X)))))
        // (DEFINE CADDAR (LAMBDA (X) (CAR (CDR (CDR (CAR X))))))
        // (DEFINE CAADAR (LAMBDA (X) (CAR (CAR (CDR (CAR X))))))

        // (DEFINE ASSOC (Y2 (LAMBDA (ASSOC) (LAMBDA (X ENV) 
        //                 (IF (EQ (CAAR ENV) X) (CADAR ENV) (ASSOC X (CDR ENV)))
        //             )))
        //         )


        // (DEFINE UNIMPLEMENTED (LAMBDA () (DISPLAY (QUOTE UNIMPLEMENTED))))


            
        // // using lowercase for the interpreted language, just to be clearer.
        // (DEFINE eval (Y2 (LAMBDA (EVAL) (LAMBDA (E A) 
        //         (COND
        //             ((ATOM E) (ASSOC E A))
        //             ((ATOM (CAR E)) 
        //                 (COND 
        //                     ((EQ (CAR E) (QUOTE quote)) (CADR E))
        //                     ((EQ (CAR E) (QUOTE atom)) (ATOM (EVAL (CADR E) A)))
        //                     ((EQ (CAR E) (QUOTE car)) (CAR (EVAL (CADR E) A)))
        //                     ((EQ (CAR E) (QUOTE cdr)) (CDR (EVAL (CADR E) A)))
        //                     ((EQ (CAR E) (QUOTE equal)) (EQ (EVAL (CADR E) A) (EVAL (CADDR E) A)))
        //                     ((EQ (CAR E) (QUOTE cons)) (CONS (EVAL (CADR E) A) (EVAL (CADDR E) A)))
        //                     (TRUE (EVAL (CONS (ASSOC (CAR E) A) (CDR E)) A)) //replace with looking up the value of (CAR E) in the environment
        //                 )
        //             )
        //             ((EQ (CAAR E) (QUOTE lambda)) (EVAL (CADDAR E) (CONS (LIST (CAADAR E) (EVAL (CADR E) A)) A)  )) //Evaluate the inner expression of the lambda, in the environment with the argument bound to the parameter
                
        //         )
        //     ))))

        // // (eval (QUOTE (quote (A))) NIL)
        // // (eval (QUOTE (atom (quote A))) NIL )
        // // (eval (QUOTE (cdr (cdr (quote (A B))))) NIL)
        // // (eval (QUOTE (cons (quote a) (quote (a)))) NIL)
        // // (eval (QUOTE ((lambda (x) (quote a)) (quote b))) NIL)
        // (eval (QUOTE ((lambda (X) X) (quote a))) NIL)

        // );
        // assert_eq!(test, stringify!((a a)));
        


        // let test = lisp!(PROGN
        //     (DEFINE Y2 
        //         (LAMBDA (h)
        //             ((LAMBDA (x) (h (LAMBDA (a b) ((x x) a b))))
        //                 (LAMBDA (x) (h (LAMBDA (a b) ((x x) a b)))))))
        // );
    }

        
    
    #[test]
    fn quine() {
       assert_eq!(
        lisp!
        ((LAMBDA (s) (LIST s (LIST (QUOTE QUOTE) s)))
        (QUOTE (LAMBDA (s) (LIST s (LIST (QUOTE QUOTE) s))))), 
        stringify!
        (((LAMBDA (s) (LIST s (LIST (QUOTE QUOTE) s)))
        (QUOTE (LAMBDA (s)(LIST s (LIST (QUOTE QUOTE) s)))))));
    }


    #[test]
    fn lexical() {
        // https://stackoverflow.com/questions/32344615/program-to-check-if-the-scoping-is-lexical-or-dynamic
        assert_eq!(lisp!(PROGN
            (DEFINE test (LET ((scope (QUOTE lexical))) (LAMBDA () scope)))
            (LET ((scope (QUOTE dynamic))) (test))
        ), "lexical");
    }



}

// Desription of my lisp:

// a simple LISP with lexical scoping, implemented fully in the Rust macro system.
// Avaliable primitives: atom, cons, eq, cons, car, cdr, lambda, let, display, apply, define.
// NIL is a shorthand for the empty list, and the empty list also represents falsity. TRUE and every other value is truthy.
// The define form is a bit weird in my lisp; (define name expr) evaluates expr and then binds it to name in the current env, so there's no restriction to
// only using it at the top level like there is in Scheme. It does not allow recursive definitions, so use the Y combinator or similar fixpoint operators for recursion.