The past couple of weeks have been interesting to say the least. Those of you that follow us on Facebook or Discord already know what we have been up to, but for those that are not in the loop – here is a quick overview.
Up until this point we have used DWScript to parse and generate our program model (the abstract symbol table). This model is then processed by the code generator, which turns your program into lean and mean JavaScript.
But, starting with the next update, we part ways with DWScript.
Our new compiler merges some of the core features that makes DWScript so wonderful with our own ideas. Ideas which gives us the pieces that have been missing. Glaringly so.
- Full support for Delphi style generics
- Full RTTI (runtime type information) compatible with Delphi
- Almost twice as fast compilation
- Binary records
- Variant and Interface support for wasm
The result of all that work is that we have reached what is known as ‘compiler singularity’. The stage where the compiler is capable of compiling itself.
As i type we have the QTX compiler running in the browser and node (!), something that was at least 2 years away on our previous timeline.
Binary record types
Pascal records (same as struct in C/C++) under QTX compile to ordinary JavaScript structures (read: raw objects). And for normal web applications that covers most programming need. You can easily serialize them to JSON, easily load them from your server. JSON is the universal glue of all things web.
But when porting code from from native languages that makes heavy use of records, records that are meant to be written verbatim to a stream or memory location, JSON becomes a problem.
Porting code to the web
I recently ported Reticulum, a system for sending data via radio devices (similar to Meshtastic and Meshcore) from C to object pascal. This codebase talks to the mesh radio over serial (USB), and involves a low-level packet protocol similar to TCP/IP datagrams. In order to be compatible with other Reticulum nodes (be they radio devices or online servers) three quite complex protocols had to be implemented, the next depending on the previous. All of them based on record structures, bit-masks and native behavior of fields and values.
Needless to say the extra work to get Reticulum running on QTX would be monumental. There are some 60 data structures involved, and each of them would need a factory function, a reader function, and a writer function. That’s 180 boilerplate functions just to handle data.
The solution to this is to make binary records intrinsic to the compiler. To make it something the compiler knows and have a strategy for.
Packed records
We have recycled the ‘packed’ keyword to signify that a record type should be sculpted as binary. Whenever you use such a record the compiler will create a buffer object uniquely for the instance. Every read or write to the record fields are resolved to the buffer’s own read and write methods matching the field type. All of this happens automatically, you don’t have to think much about it.
type
TMyMessage = packed record
rId: uInt32;
rSize: UInt16;
rX: uint8;
rY: uint8;
rlabel: string[255];
end;
And yes, writing binary records to a stream or buffer comes with this as a natural consequence. Our RTL already has a rich set of classes and methods for working with binary data. But this makes QTX feel “almost native”, even when you target JS.
Generics
This is probably the feature that will have the most impact, as it dramatically changes how we write maintainable code. I have already isolated several aspects of the RTL that will be reduced by as much as 60% as a consequence.
A universal dictionary is now reduced to:
type
TQTXDictionary<K,T> = class( TObject )
private
fLUT: array[K] of T;
public
property Count: int32 read ( fLut.Count );
property Keys: array of K read ( fLut.Keys );
property Items[const AKey: K]: T
read ( fLUT[k] )
write ( fLUT[k] := Value) ;
procedure Clear; virtual;
end;
A dictionary of this type has technically always been a part of QTX through associative arrays. So you could achieve something similar with:
type
TQTXDictionary = array[string] of TMyObject;
The difference is that the array declaration needs full typing, so the key and value must be defined. The generics class will accept any intrinsic datatype as the key, and more or less any type as the value. That’s the beauty of generics, that you isolate behavior unbound by type.
Any of these are valid:
var dict := TQTXDictionary<string, TQTXLocalDisk>.Create;
var dict := TQTXDictionary<int32, TQTXPixmap>.Create;
var dict := TQTXDicionary<THandle, variant>.Create;
RTTI support
RTTI (runtime type information) is a powerful feature that let’s you examine datatype information at runtime. There are a myriad of ways this can be used to improve maintainability of your code, but where it really helps is object persistence and serialization.
The QTX RTL already have a strategy for persistence. In the class TQTXPersistent we introduce three methods: ReadObject, WriteObject and Assign. Since the target of our persistence remains JSON, this strategy has been manual, where you override ReadObject and WriteObject and process the fields in question.
Persistence of components outside the IDE have not played a part in our codebase so far. We don’t want to load design-files for forms or components, and instead we convert our layout’s to code, and compile them with your form constructors.
With RTTI in place however, we can implement standard persistence of published properties for any class that inherits from TQTXPersistent. Almost no class in our RTL use this, so adding the same persistence that Delphi offers is easy.
TQTXPersistent will soon do more or less the same as Delphi or Lazarus: enumerate published properties and serialize accordingly. And with that we no doubt have to add a TFiler “ish” mechanism for handling exotic datatypes (e.g binary properties, like TStream, TManagedMenory etc).
Summary
As you can see we are going full steam ahead! Quartex is growing on a weekly basis and we are quickly catching up with the big boys in terms of features!
With the new compiler we might even look at adding llvm support and produce native binaries, but for now focus in purely on web technology.