Major rewrite in the works!

It may appear that development on Fexl has stalled lately, but it actually hasn't. For a few weeks now I've been working on a total rewrite of the Fexl interpreter. Although the code and data structures are very similar to the current release b13, there are subtle differences which make the new system far more elegant and powerful. 'Za matter of fact I'm almost embarrassed by the current b13 code.

For one thing, the new code does not call exit anywhere, except in the case of internal logic errors which should never happen unless the code itself is defective (i.e., assertions).

Specifically, it does not call exit even if the user's program uses too much memory or runs too long. That used to be how I enforced those limits: just bug out with an exit. No longer. Now the entire system is unwound naturally all the way back up to the main program, which exits with status 2 after freeing memory in an orderly fashion. This allows me to test all sorts of abnormal conditions, one after another, in a standard test suite, including running out of memory and running too long.

Also, the code no longer needs a "type" structure. Instead, the "value" structure points directly to the reduction routine for its type. Consequently the core evaluation function is looking very fast and tight. The representation of atomic data is radically simplified as well, since now an atom points directly to the routine that frees its data, and reference counting of atomic values is handled naturally in value.c without the developer of a new atomic type needing to worry about it.

The code no longer uses recursion for evaluating the left side of a function. Instead, it uses its own stack consisting of a linked list of value structures. This makes the system very robust when faced with functions with a very deep left side. It doesn't even use recursion to force the evaluation of atomic values, e.g. when you add two numbers. Instead, the '?' operator (the one which is forces eager evaluation) is applied to make that happen.

I am also making it easy to call the Fexl parser from within a Fexl program itself. This is how you create restricted, "sandboxed" environments when using Fexl as a scripting language. The core Fexl interpreter can do anything on your system, so when you put Fexl into the hands of a user for scripting purposes, you want that power to be reined in, only giving the user a handful of things she can do.

It should be fairly easy to call Fexl as a client process from another language such as Perl, Ruby, Python, Java, Lisp, etc. In the case of Perl, I wrote a simple routine called spawn_client which makes it easy to fork a process and grab handles to its stdin, stdout, and stderr (it uses socketpair, fork, and dup for this).

In the new release, I plan to support calls to dlopen and dlsym, which are the standard mechanims for calling routines in shared libraries. This will enable the creation of plugins for Fexl so it's not necessary to link the entire kitchen sink into one executable. For example, someone may want to use OpenSSL's crypto or bignum libraries without having to link it into Fexl.

I also have ideas on a software layer which can make any Fexl function persistent without the programmer having to know or care about it. If that is successful, then you would have no need for "databases" or even key-value stores as such. You could just write a Fexl function which assumes that everything is in memory all the time, even if you're storing the entire Manhattan phone book, and everything would just work. These ideas are fairly concrete and well-worked out I think, though I'm not promising anything yet.

I've been so busy with code and testing that I haven't had a lot of time to post lately. Stay tuned to this site for further developments.

Sample code

# The infinite list of Fibonacci numbers.
\fib == (\x\y item x; \z=(+ x y) fib y z)
\fib = (fib 1 1)

... and more samples here.