Library project type added

This has been a long time coming, but since it requires no more than a day’s work under our new codebase, I figured I could add it as a surprise. QTX now supports a library project type!

What do you mean, libraries?

While the IDE focuses on visual client projects (browser) and non-visual server projects (node|deno), some of our backers have voiced that it would be really useful to have a more 1:1 approach to Javascript.

The IDE now sports a new project type, namely library. We will no doubt expand on that type in future versions

Typically these backers work with JavaScript daily, either maintaining existing websites, or having to write JS for new projects. While we can all agree that JS is indeed powerful in the right hands — it is not very productive. Just from a time perspective (the time it takes you to implement something) Object Pascal is without a doubt more productive.

For them, being able to write code that doesnt pull in the whole might of the QTX runtime library is a productivity boost.

How does a library look like?

If you have ever written a DLL library with Delphi or Freepascal, you already know what a library unit looks like. It has the same structure as a program unit except that it starts with the “library” keyword. Here is a clean cut library example which contains a single function:

Since QTX contains a full compiler you need to make sure you dont apply symbol obfuscation in the project options, or the names of your functions and procedures will be garbled. But when we compile the above we get the following code:

If you omit using the begin/end. initialization block, it is also omitted from the output, leaving just clean functions

In other words, a library emits a javascript file that you can work with in JavaScript projects, and that other JavaScript developers can make use of. And just like a native DLL you need to make sure the functions and procedure names are valid. I suggest you prefix them with a unique combo (just like we use QTX as a class prefix in the RTL) to avoid confusion or namespace poisioning.

Following the rules

A lot has happened in the world of JavaScript these past 10 years, so you might want to post-process the library code a bit before you shart sharing it. I mentioned namespace poisioning in the above paragraph, and that is very important to avoid. Basically that means, that if you simply load in the generated javascript file – you risk overlapping with existing Javascript functions with the same name.

It is also slightly bad taste to define your functions in global scope, you probably want to isolate your code inside an enclosure, like this:

var _myLibrary = ( 
  function setup() {
  // Library code here
})();

What Javascript developers typically do, is that they return an object (also called a namespace object) in the above function, which only exposes the functions you decide as named members. So something like this:

var __mylibrary = (
 function Setup() {
  var _tmp = {};

  function hello() {
    return "hello";
  }

  _tmp["hello"] = hello;
  return _tmp;
 }
)();

That probably looks very complicated, but it’s really not! Let’s break it down and have a closer look.

The ( and ) characters that we wrap around the “function Setup()” basically tucks away the content as an expression. This means that it’s content is not visible in the global space (which is what we want). The “();” at the end means we execute the expression block. This works because the expression block will resolve to a function.
If you look closer at the actual function, you see that we create an empty object at the beginning of Setup(), and that this object is later returned as the result.
Since this is all wrapped as an expression, that object ultimately bubbles out of the function, and further – out of the expression. This is more or less the same as if you returned an object instance from a Delphi function. The only difference here is that we sculpt that object in it’s entirety inside the function.
Once the object bubbles out of the expression, we store it in the _mylibrary variable. This means that we can only call whatever functions was registered in that object, everything else that those functions might depend on — is neatly isolated inside the expression block and not reachable from public scope.

So if we want to reach the “hello” function in the example code above, we have to call it via the _mylibrary object, like this:

console.log( __mylibrary.hello() );

The moment you null out the __mylibrary variable — the entire library vanishes into the void of the garbage collector.

Creating the namespace object from pascal

You can create that this namespace-object as a part of the pascal startup code if you like (simpler). You still need a bit of post processing (wrapping it as an expression), but it does help simplify things. You can do this for example:

Since there are only functions in JavaScript we can safely add a inline-assembly return call at the end without causing problems. In this case we just dump out the object we just populated, exactly like i outlined earlier in JS code.

Using the RTL units does pull in more code, but you also get more functionality than several JS libraries combined

In this case you would just remove the whole “var main = function” part (and curley brackets), since you are already putting the whole shabam inside the Setup() function. As a bonus — with such a namespace object you can use obfuscation. The compiler will automatically map the obfuscated symbols. So “CoolPublicFunction” will still be called that in the returning object — even though it might be called something else entirely inside the expression block.

An alternative way which doesnt involve creating an empty JS object, is to use anonymous classes. Anonymous classes makes it a lot easier when you suddenly want to add some structures to the object, like version info or something more elaborate. Remember that these classes are not pascal classes, they are “in place sculpted JS objects”:

More fancy-pants syntax wise, but with an elegant output

The output of using an anonymous class is:

var main = function() {
  NameSpace = {
  "CoolPublicFunction" : CoolPublicFunction
  };
return NameSpace;
}

All things considered, libraries via QTX represents a significant time saver. Having to add maybe 5-6 lines in post processing – is nothing compared to the amount of time you save by using pascal.

Creating a global object

JavaScript developers have for the most part moved away from putting everything and it’s grandmother in global scope, but between you and me – in 99% of the cases it makes no difference. Stock JS libraries like jQuery registers as a global object, as does more or less all the major libraries.

I am not advocating that you stuff everything in global space (e.g application critical data). But for libraries that are meant to be used by your whole website the criteria is different.

Making the library globally available simply means registering it with the window object, like this:

window["mylib"] = __mylibrary;

Once registered there, you can access it from anywhere as “mylib”. As you probably understand, with the sheer number of libraries out there this can potentially lead to overlaps and conflicts. I cannot stress enough how important it is to prefix your names with a combination that uniquely identifies you particular code.

Cherrypicking from the RTL

The moment you start referencing the RTL in your library code, the complexity of the generated output increses. Use of TObject will bring in VMT typing, sprintf and a lot more. Fundamental units such as qtx.sysutils.pas depends heavily on lookup tables, so immediately you see a bump in the footprint. Both in initialization code and the included methods.

qtx.sysutils.pas pulls in around 1200 lines of code. But keep in mind that this is memory handling, streams, text codecs and a complex datatype encoding that makes QTX so incredibly powerful to begin with. It’s important to pick your battles.

Having said that, it is still mince-meat compared to what other compilers are spitting out. The QTX RTL is optimized for speed and some of the functions in the RTL are actually libraries in their own right in the JS world. The memory buffer management, streams and datatype conversion (moving between untyped memory and typed datatypes) that we take for granted in QTX, simply do not exist in the JavaScript reality. This is why QTX can do stuff out of the box that would take JS developers days and weeks to accomplish.

As always is the case with object pascal, the RTL usually adds some initial size to your output — but once added, the rest of the code that uses the RTL is typically very small and fast. This is philosophical debate more than it is technical. Object Pascal is a language of practicality. We favor safety and completeness more than we do codesize. I agree that the code elimination process could be better and more agressive, but it ultimately boils down to inter-dependencies in the RTL (of functions using existing functions, thus both have to be included in the output).

Either way, you can cherry pick all you want from the RTL, and compared to what vanilla JS developers have to work with – you can ramp up functionality quickly and efficiently.

In the future we might put more work into doing some of the registrations for you, like wrapping the code as an expression, using attributes to mark functions for export and so on — but right now, low level is what people have asked for. So that is what we provide.

Putting some final touches

We are nearing the end of this developer cycle, meaning that the last deep-dive is coming to an end, and thus we turn back from the low-level coding and return to the UI tickets and final push.

Would you believe that the entire IDE uses between 50 and 70 megabytes of memory? It’s absurdly lightweight in this day and age, and very very responsive now that we have done some hardcore optimization.

There have been a few hiccups along the way, but nothing we cant handle.

Unit structure and AST rebuild

The IDE has a new and very fast parser in place. This parser does not replace DWScript, but is meant to compliment it when doing complex tasks. Our parser is much faster than DWScript since it technically doesnt parse beyond the implementation keyword (so more or less only half the unit, if that).

What our parser excells at is to quickly find out where a piece of text is located. It can be something as simple as the “unit” keyword and accompanying name, to more elaborate tasks like finding a class definition – and further extracting information about a specific property.

The most immediate use is when dragging and dropping widgets onto a form. Here we must quickly check if the unit where the widget is defined is already in the form’s uses list. If it’s not, then it needs to be inserted. Having to literally compile the whole unit during a drag and drop operation is counter productive. So these are some of the tasks where a fast, ad-hoc parser can do miracles.

Those that tested the last update probably noticed that it was faster already, but after these latest changes — it’s virtually instantanious.

Unit structure

What I am doing right now is to make sure the unit structure is rebuilt only when there are changes, and that only the changed nodes are processed. This is not as easy as you might think since so far the unit-structure has been updated whenever we rebuild the model (which the IDE does quite a few places, which is why the new parser will come in handy).

Previously we rebuilt the model “on demand”, so whenever we needed the AST object we basically issued a rebuild on the spot. This works, but as the IDE grows in complexity it is simply not sustainable to have models being spawned left and right. As such I have isolated model rebuild in a separate class, which listens for changes to the current editor (if it’s a Pascal tab) and will schedule a rebuild after a change is finished. So it will know when you start typing, but will wait until you stop typing to issue a rebuild.

The Unit structure manager (called a provider in our system) likewise listens to the same change notifications, and will enter “wait state” when it notices a change. Because it knows the model will be ready as soon as you stop typing.

Ditching Facebook

Facebook have been acting strange lately (again) where it keep classifying articles and content that i cross post as “spam”. How exactly I can spam my own groups with coding links and articles I wrote is beyond me, but I think it’s time we move away from Facebooks.

As such I have invited the members of our Facebook group (the backers of the project) over to Discord, and from now on any news and updates will be first posted to the forum — and then cross posted to Discord. I dont see any other way to deal with this because Facebook is quite frankly unreliable. It’s the same thing that happens again and again, and since I run several groups closely knit by topic — my posting behavior will always be different from whatever they trained their Ai model on.

But it’s not a huge loss. We always did plan to leave Facebook. I had hoped we could use it to finish version 1.0 before we jumped over to our own forums completely, but — might as well do it now and get it over with.

Other additions

The TQTXPixmap class has been getting some much needed love. I have added gradient methods to it, with support for the 5 most common gradient types (including radial).

I likewise added Draw(), DrawStretched() and DrawRotated() methods to it, making it very simple to copy raw pixel data between TQTXPixmap instances. I am using scanline copying as much as possible, so as far as 2D graphics goes this class is now very, very powerful.

I also fixed a problem with the RGBA alpha blending code, and optimized more or less the entire library. This is quite important because this class also works under node.js — which by default has no graphics capacity what so ever. Eventually we can add JPG / PNG and GIF codecs to it, making it 100% runtime independent. I quite fancy compiling a C/C++ font renderer to asm.js too, so we can do live decoding and rendering of TTF font files — but right now this is not a priority.

UI tasks and back-end node.js server

As we now turn the boat and return to land so to speak, we have a relatively small list of UI improvements that needs to be done (including that annoying bug when you rename and/or delete files in the active project folder) — and then the main system is ready.

Before we launch there is just one thing to add, and that is to implement our license server. This will be coded in QTX ofcourse and will handle product registration and trial versions. This is quite crucial but it needs to be in place.

Oh, and I want to add one more project type to the IDE, namely “library”. This will basically be a project type that adds nothing – meaning that it will not inject any startup code. What you write is what you get (more lightweight compilation to JS) which makes it more suitable for doing library code, or code modules that you wish to use from JS. This can be quite handy when you do JS tasks regularly, but would like the benefit of using object pascal and the RTL instead.

So yes, we are making good headway!

The mysterious non-working keypress delegate

This was an interesting case. Kevin, who is a Ninja bughunter and all-round super coder, reported back that he could not for the life of him get the Kepress delegate to work!

He had tried the obvious, like creating the delegate class manually from scratch, forcing it to trigger, giving the delegate various parents to see if that help – but to no avail. Nothing happened. Quite frustrating to say the least when you are creating examples and being a technical writer.

Adding delegates to a widget is simplicity. Just selected a widget, switch to the delegates tab by the inspector, and click Add ..

While working with HTML or Quartex Widgets you have probably noticed that some of them have focus abilities. Like a text edit box, memo field or button. These widgets not only capture the focus, they visually reflect that back to the viewer. Mostly in the form of a changed border or highlighted edit-cursor.

So what makes these widgets so special? And why do they get first dibs at ordinary events?

To focus or not to focus, that is the question

Despite the radical morphing into a multimedia powerhouse, the modern browser remains at heart but a humble document viewer. And in the world of text content, the browser cannot treat all entities equal. Control elements such as buttons, scrollbars, lists, comboboxes. memo fields and the likes are usually “real” widgets. Meaning that they are actual elements created by the underlying operating system (WinAPI, MacOS, Linux to mention the most common).

But, what separates them from pure browser controls, is that they have properties that is rarely accessed.

One of these properties is: tabIndex

As a Delphi, Freepascal or C/C++ developer this property should be no stranger. to you. All visual UI elements in a real, native application has a tabIndex: Int32 property in one form or another. Usually in concert with a tabStop: bool property. So when a user tabs through a form design, the tabIndex says which widget should receive the focus – while tabStop: bool tells the OS if it should skip over a tab item and more to the next.

Why is this even needed you say?

Well, imagine you are making a shopping trolley for your amazing webshop. The customer has ordered 10 amazing products that you display in a common top-down grid. The user tabs through each row just to make sure he ordered the correct amount.

The values you set for tabIndex, determines where in that tabbing-order your widget get’s focus. So in a list of 10 items, if you set your tabIndex to 3, then your widget will get’s focus when the customer has pressed tab 3 times.

There really is nothing more to say about tab order.

QTX Focus and delegates

If we return to Kevin’s case, he had done everything correctly.

He created a KeyPress delegate on the main form, doubled clicked on the delegate in the inspector. This automatically creates the event handler for the delegate. And he went on to implement his code. Except that when he tested it, nothing happened.

Well, it took me a couple of minutes to recognize what this was. It’s been a while since I have been dealing with delegates, so i was baffled myself a while there.

The first thing I did was to move the delegate to the Application.Body object. This is the document itself, and the first element to receive any delegate. All events / delegates bubles upwards in HTML5, so interestingly enough, the elements at the bottom are notified first.

The event fired as it should there, which meant it had to be something else. And that’s when i remembered the tabIndex rule.

In short:

If you want a DIV or any other “non operating system” element to trigger and respond to common events — make sure you set the tabIndex to something higher than -1 (default).

New property added

Sadly, this property was not exposed in our RTL. I probably thought that it was more something that widget creators would need. I was clearly wrong, so I have added it to TQTXWidget, and it will be available in our next update.

In the meantime you can bypass it with a spot of:

// First method
self.Handle.tabIndex := 0;

// Second method
self.Handle["tabIndex"] := 0;

I suggest the first method since it’s more elegant and avoids a lookup in the handle members list. Just remember the prefixing with lowerase and it’s all good.

When writing games or full screen media

If you are writing full-screen games, or games where the keyboard should be operational regardless of what has focus on screen – I strongly urge you to create your delegates manually and attach them to TApplication->Body.

As you can see from the picture right, both TApplication and TQTXBody expose more or less all the common delegate helper methods – so adding a keyboard delegate that is global, is a piece of cake.

If, however, you are writing widgets that are supported to play nicely with other widgets on a form, then never attach to global objects like this. It will only cause problems and confusion for others.

But if your application will be fullscreen, you have full control over the content, then attaching to the document directly is perfectly valid.

New unit structure and mini parser

To speed up the amount of time the IDE uses to display the unit structure, we have spent some time implementing a special parser. This parser is diffent from the one already present in the compiler core – in that it does not go through the ordinary compiler steps (tokenizing, parsing, building an AST, structure verification). It is designed to chew trough the syntax according to the language rules, extract symbol names and their positions – and finish the moment it reaches the “implementation” or “end.” termination symbols.

This is also a very hands-on test for the parser framework I wrote about earlier, testing to see how well our framework works with such a daunting task. One thing is having a parser that handles miniscule tasks like an ini-file or dfm text files; a full on language parser is something else entirely.

Performance

Since this is not a full parser in the traditional sense, although it could relatively easily be turned into one, the cost of parsing is surprisingly low. Normally you can draw some statistical vectors between speed and resource consumption, where the latter is usually a penalty of caching and pre-calculation. In our parser there is none of this. It does exactly what it says, and it’s every bit as tricky to implement as you imagine.

The current parser test application. Here parsing qtx.dom.widget.pas from our RTL.

When you write a parser for a language like Object Pascal, especially a living, vibrant dialect such as QTX with it’s many syntax modernizations and enhancements – you really find out just how many combinations and variations the language truly has.

This one for example:

function jsObject: variant; external 'Object' property;

Just like Delphi or Freepascal can reference DLL functions by declaring them as external, so can QTX. The difference is that under QTX we are not mapping DLL library symbols to a local symbol so we can use it; we are telling the compiler what it should replace our mapping with. Whenever we use JsObject() in our code, the compiler will eventually just replace that with “Object”. This is a big difference.

So all we are doing here is telling the compiler that there is an external symbol called “object” that we want mapped as a local function.

The wildcard is that extra keyword at the end, namely “property”.

What does the property keyword do? Well, when the property keyword is found after the external qualifier, then the method will be codegen’ed without parenthesis. If it has parameters, brackets will be used instead of parenthesis.

Another road rarely traveled, is when you have external objects which (just as an example) only allow a write-only property. Take this for example:

type
   JExternal = class external
      function SetRW(value : String); external 'setRW';
      property FieldR : Integer read external 'hello';
      property FieldW : Integer write external 'foo';
      property FieldRW : String read external 'world' write SetRW;
   end;

Here we have a class that is marked as external, but the JS names for properties doesnt really suit us, so we use our own names in the pascal code. This is perfectly legal as long as we mark the properties as external, and provide a valid name for them.

But notice that last property, FieldRW, and how we suddenly have an external keyword right in the middle of the definition. Basically what this syntax is saying, is that whenever you read from this property — you will just read the JS property called “world”. But you can write to it all you want.

The number of times you will be using these declarations will be slim. And honestly, most people would define these wrapper classes differently. The above will be more useful if you have a JS object that has a “world” property, and then a SetWorld() method. Since these go together in the pascal world you can use the esoteric syntax above to unify things.

My point with all of this was not to scare you with esoteric, edge case syntax — but rather to underline that my parser code have to deal with all of this. I still need to add that somewhat odd use of external, but I think I have implemented the rest of it.

This is why it’s so important to test with real-life units, like those from our RTL that is already being used. Because if it can handle the RTL, then it’s ready for inclusion in the IDE.

Why a separate parser to begin with?

DWS is awesome, but building an AST means doing a full compile of the unit you are editing. If this is a form-unit, it will pull in all the units in the uses-clause (recursively), which however brief is a consuming task. In a perfect world we would just stick to the AST for everything, but reality has proven that it’s sometimes good to have a scalpel and not try to chainsaw our way through everything.

Things like Rename Unit, Add unit to uses, Remove unit from uses, Check if unit is in the uses, check if a class is in the unit, check if a member is defined in a class — stuff that you probably dont think about that the IDE does. We also want to expand things, like rename classes etc too later.

This was yesterday before i optimized things, so it was 16ms slower then 🙂

Once this is in place I will circle back and put our background compiler in a thread, now that a lot of the housekeeping jobs can be delegated to our mini-parser. And then it’s time to deal with the UI tickets and license back-end (yes we need to have the license node.js server on our website before we launch, but just the license stuff, not a full shop!) – and then (unless something else turns up) we are done!

New JQuery UI package

One of our more experienced backers, Ed Van Der Mark, has updated his JQuery-UI package for Quartex Pascal. His latest package installs both the widgets into the component palette, and also registers project templates – so that you can create new project types tailored for JQuery UI directly from the “New project” dialog.

The JQuery-UI package registers both widgets in the palette and a new project type

Here Ed’s original post:

I have made an update for the JQueryUI widgets. It’s now possible to add the widgets in the Widgets Palette of the Quartex Pascal IDE. The widgets are dependent of a Javascript framework and theme files.

Install steps

Download the JQueryUI.zip file by using the download-link
Upzip the JQueryUI.zip file in the ‘projects’ folder
Within this project folder you’ll see a ‘install’ folder
Execute ‘install.cmd’, this command copies a package file and a template file with a template folder

Restart the IDE

After (re)starting the Quartex Pascal IDE you’ll see the new widgets of JQueryUI. After starting a new project you can choose a new template JQueryUI, this template contains the needed dependencies for this framework.

It’s my first try of adding a package to the Quartex Pascal IDE. In the near future I hope to expand the widgets with useful dialogs.

If you are ready with testing the widgets and want to get rid of the extra JQueryUI widgets, please delete the ‘jquery.widgets.pkg’ file in the ‘packages’ directory (only possible after leaving the Quartex Pascal IDE, because of a file lock).

A parsing framework for Delphi, FPC and QTX

Text parsing is something most people rarely think about. We mostly work with file formats that already have existing parsers – be it JSON, XML or Inifiles. But there are edge cases where knowing how to properly parse a custom format or string can make all the difference.

In this article I will cover the essential architecture of the Quartex parser, the default parser for the Quartex RTL and our native framework. This framework is the codebase we wrote to implement the IDE for Quartex Pascal, and it contains a wealth of useful units. The Quartex native framework will be available to backers and customers shortly after release of Quartex Pascal.

Some background

The most difficult part of implementing a new development toolchain and RTL, is that you literally have to implement everything from scratch. In our case we compile object pascal to Javascript, which means that very little of the infrastructure we know from Delphi and Freepascal exists. At best you can write a thin wrapper that use features from the JavaScript runtime (if applicable), but when it comes to more complex functionality – you have no choice but to write it from scratch.

Since Quartex Pascal targets node.js / deno, being able to work with ini-files is useful for things like server configuration. Json is great and i could have used that, but I wanted to make the RTL as compatible as possible (or at least offer similar features to Delphi and Freepascal), so I ended up taking ‘the hard way’.

Concepts and approaches

The first two iterations of my parser were a bit rough around the edges. They worked fine, but I always ended up exhausting their design and getting frustrated, so eventually I ended up writing a new one. It was only when i reached the third revision that I finally had a mold that was flexible enough to cover all the use cases I had encountered (with includes the LDEF language parser). It seems so simple and obvious when I explain it below, but a fair bit of work went into this (and yes I realize that the model i arrived at is not new).

What I ended up with was a parsing model consisting of 4 elements that are largely decoupled:

Buffer
Model
- Model objects (optional)
Context
Parser
- Parser collection (optional)

The Buffer

The buffer class is where whatever text you need to parse is held. The buffer exposes basic functionality for traversing the text (the parser is forward only, with some exceptions). Much like a database table it operates with a BOF, EOF and Next() cursor mechanism (you call the Next() method to move forward by one character). All methods are inclusive, meaning that reading or comparing always includes the character where the imaginary cursor is located. The current character can be read via the current: char property.

The basic buffer class implements a rudimentary functionality. This is further expanded in TQTXParseBuffer with more elaborate behavior.

The idea of ‘inclusive reading‘ might sound like a trivial thing, but you wont believe the mess if you have functions that deviate from that exact same behavior. It seem absurdly obvious, but when writing loops and recursive code, finding that one scenario where you called Next() once to often is tricky.

The buffer builds up more complex behavior such as Read([length]), ReadWord(), ReadQuotedString(), ReadUntil(), Skip(count), Peek() and Compare(char|string). The most common comparisons such as equal, punctum, space, PasQuote, CQuote, less, more and so on – are isolated in a child class. This means you can write code that is quite readable and easy to maintain:

// move to the first char
buffer.first();

// Check that we are not BOF
// That would mean there is nothing to work with
if not buffer.BOF then
begin
  repeat
    if buffer.common.equal() then
    begin
      // we found the equal char
    end;
    buffer.Next();
  until buffer.EOF;
end;

The above mechanism of EOF / BOF works well with Pascal strings that start at index 1. BOF evaluates as offset < 1, while EOF evaluates as offset > length(text).

The buffer operates with a few sub concepts that should be known:

Ignore set
Line ranges (optional)
Bookmarks
Control characters

These are very easy concepts:

Ignore characters are set to [space + tab], this means the buffer will just skip them when you call buffer.ProcessIgnore().
Line ranges means that the parser keeps track of CR+LF and register where each line of text starts and ends within the text (the offset into the string). The text you assign to the buffer is kept as a stock string field. There was little to gain by using pointers since an integer offset into a string is pretty fast.
Bookmarks stores the current cursor position, column and row, and the current line ranges. Bookmarks are typically used when reading ahead to check something, so that you can quickly return to the previous state.
Control characters are set to [#13,#10] (on Windows, Linux operates with #10 by default), the buffer will skip these and update the line-ranges when you call buffer.ProcessCRLF.
Both ignore characters and control characters can be changes (type is TSysCharSet)

The model

Where the buffer provides the essential means of navigating through the text, you also need a place to store the result. For example, if you are parsing pascal code then you would identify keywords, parameters, brackets, semicolon and so on – and store these as tokens somewhere (the tokenizing stage of the compilation process). This ‘somewhere’ is where the model object comes in.

Example:

With ini-files the two entities we collect are groups, such as “[settings]”, and name-value-pairs, like “port=8090”. Differenciating between these are simple, as group names are within brackets [], while value assignments contains a = character beyond column position 1. Checking that the criteria for a value is met means reading ahead and looking for “=”. The name will be to the left of the equal char, and the value is whatever is beyond it. CR|LF is a natural stop, making inifiles simple and elegant to parse.

The main logic of the TQTXInifile parser is fairly straight forward and muc smaller than you might expect

You are expected to inherit from TQTXCustomModel and setup whatever data structure you need there. So the parsers that I already implemented, such as TQTXInifileParser, TQTXCommandlineParser, TQTXDFMParser and so on – all have a specific model, context and parser classes.

The context

This is where things become more interesting. When you are dealing with truly complex formats, be it source-code or something of equal complexity, you can write several parsers that deal with a particular chunk of the format. So instead of having one massive parser that is supposed to deal with everything under the sun – you divide the work across several parsers instead.

To make this easy and fast, both the model and the buffer is isolated in the context object. The context object is the only piece that is shared between parser instances. When you create a parser, you must pass a context instance to the constructor. Since the parsers use the same context (and consequently buffer), you can create parsers on demand and continue in your main loop without problems.

The parser

The parser class is where you implement the logic for either an entire format, or a piece of a format. You use the methods of the buffer to work your way through the text, store whatever results you have in the model – and create sub parsers should you need them. Since no data is stored in the parser itself, not even the position in the buffer, they can be recycled easily.

For example, if you were parsing a pascal unit, you would have a parser for the interface section, one for the uses clause, enum parser, record parser, class parser, interface parser, function and procedure parser, var parser, comment parser – and then use them at the correct places.

Recycling parsers this way is why we have a parsing collection available.

Parsing collections (optional)

Sub parsers (as mentioned above) can be held in a dictionary like class, TQTXParserCollection, so you can query if a parser exists for a given name or token.

var lKeyword := context.buffer.ReadWord();
if fParsers.GetParserInstanceFor( lKeyword, lSubParser) then
  lSubParser.Parse( context )
else
  raise EParseError.Create('Syntax Error');
end;

Availability

The latest parser framework will be available in the next update. It has been re-implemented from scratch to work with QTX (first and foremost), Delphi and Freepascal.

As of writing the use of the code outside of QTX is reserved for backers / customers, we might consider making it open source together with the Quartex Native Framework after the release of QTX.

Align: TAlign and PreAlignBounds

One of the coolest features that has been added to Quartex Pascal’s RTL in the past few months, is most likely support for Align: TAlign for TQTXWidget. This greatly simplifies form layout, and also doing layout when writing your own custom controls.

Rules and differences

HTML5 is not the same as WinAPI or GTK, and as such there are sometimes subtle and small differences in how things are done. Aligning widgets is fairly straight forward (from a user’s perspective), but behind the scenes it’s actually a bit more complex than how the VCL or LCL frameworks operates.

The challenge is not technique or anything special we do in our code, but rather how the browser calculates things – and when it does it. For example, a change in content of a widget can trigger what is known as a reflow. This is when the browser decides to completely re-calculate the layout for a branch in the DOM (document object model). The browser can even force a full reflow, meaning that it recalculates the entire layout for the whole DOM. A reflow is not a problem for QTX, but if you are doing a lot of animations, resizing and content changes – non aligned content might jitter while the browser performs its calculations.

To avoid this, TQTXWidget cache’s the bounds-rect it had before the Align property was set. This cache is used to calculate the order in which widgets occupy horizontal and vertical space – and how much space they are allowed to use in the overall layout.

For example: If you have two labels that are both aligned left, which one of them should be placed first? Obviously the one that is closest to the left-edge at the time the widget was aligned. To avoid reflows jittering widgets during a complex layout operation (e.g the user resizing the browser window excessively), even for a millisecond, the RTL cache the original bounds.

This way, regardless of what the browser might do – widget layout remains consistent.

Adjusting pre align bounds

Alignment is very important when writing your own custom controls

If you try to change the width of a widget that is aligned left or right at runtime, you will notice that nothing happens. This is because altering the width property does not alter the cache’d boundsrect values. It’s the cache that has to be adjusted, because that’s what TQTXWidget uses as the basis for it’s calculations.

In other words, you have to alter the properties inside the TQTXWidget.PreAlignBounds property if you want to change size proportions or order of aligned widgets.

In the present build this can only be done from within a TQTXWidget (it can only adjust itself). However that is about to change. In the next build, TQTXBounds exposes SetLeft(), SetTop(), SetWidth, SetHeight and SetBounds() methods. This greatly simplifies altering proportions at runtime.

So the correct approach would be something like this:

Procedure TMyControl.Resize(Orientation: TQTXOrientation);
begin
  if fChild.Align <> alNone then
    fChild.PreAlignBounds.SetWidth( round(Height * 1.5) )
  else
    fChild.Width := fDefaultSize;

  // Allow the RTL to do it's resize calculations
  inherited Resize(Orientation);
end;

March update and beyond

It has been a little while since we issued an update, so last week we decided to ship another “snapshot” build for the initial backers. There have been a lot of changes so listing them all would cover several A4 pages, needless to say – we are hitting the last tickets at record speed.

Ragnarok protocol designer

While visual component features such as Align: TAlign and linear constructors were much needed additions (earlier constructors were asynchronous), the main focus of the latest update was to finish the Ragnarok protocol designer. Adding support for enum types and collection types in the process.

One of the first things you will notice when writing client-server applications regardless of what transport methodology you use (REST, UDP, TCP or WebSocket), is that you need uniformity. If you are writing a complex server that deals with 20, 30 or more messages (read: commands), you must have standards with regards to message structure, field datatypes and how data is serialized. You also need to differentiate between client messages (sent to a server) and response-messages (returned to the client as a response).

Ragnarok protocol designer is built into the IDE, you can edit as many protocols as you like

Ragnarok solves all of this. It builds on a message-envelope (read: JSON structure) that is tried and tested. Adding to this is the visual designer that takes the grunt out of defining complex message structures. You can define your own complex structures, collection types, enumeration types – which can all be used side by side with intrinsic types when constructing your message objects. There is also clear differentiation between client messages and response messages.

Adding a protocol is as simple as clicking “New File” and selecting “Protocol”.

Multiple protocols

Protocol definition files have the extension *.RRP and can be added to your project at any time. You can also have as many such protocol definition files as you like in a single project.

When the compiler notices such a file in your project folder, it automatically generates ready-to-use object pascal files that you can use from your code. You can define what should be generated within the protocol editor. A typical client-server project would consist of a node.js websocket server and a HTML5 websocket client, but you can also generate client code for node.js – which is handy when creating services that communicate in a cluster.

You can check what code should be generated for the protocol-file. Protocol definition files are JSON

Ragnarok can also be used between a HTML5 application and running web-workers (threading). The uniformity of the generated protocol classes simplify communication regardless of endpoint types (networking, message channels, named pipes et-al).

Easily extended

Since we are aiming for our first launch of the product shortly, support for Ragnarok will be limited to Quartex Pascal. However, the generated pascal code – and consequently the JSON emitted by the message objects when serialized, is standard JSON. Meaning that these messages can easily be supported by other languages.

Once we start working on version 1.1 of Quartex Pascal, support for generating code for JavaScript or TypeScript, as well as Delphi / Lazarus and C# is a natural next step. There is no black-magic involved, just a well crafted message-envelope scheme and code-generators that saves you days of monotonous boilerplate coding.

Switching to Lazarus

The first version of Quartex Pascal is expected to be out this summer (hopefully before). The initial version is written completely in Delphi. However, starting from version 1.1 we will have switched wholesale to Lazarus and Freepascal.

This will have a huge impact on our demographic, and will allow us to target more exotic platforms such as Raspberry PI 4 and 5, ODroid N2+, Rock Pi 5b and other popular single board computers. Raspberry PI is especially interesting since it’s very popular for school and education. We feel Quartex Pascal is a perfect fit for education. The language is easy to learn, students get instant visual feedback, and they can target a HTML5 and work with internet technology directly. Something which for kids growing up today is second nature.

We already have a working builds for MacOS x86 and ARM, Linux x86, Windows x86 – both 32bit and 64bit builds. These builds use TrollTech QT as their UI framework, including their exciting WebView engine.

The transition from Delphi to Lazarus / Freepascal will take some time though, since Quartex Pascal is a large and complex piece of software that covers a lot of different technologies. We have few dependencies, but it’s imperative that the IDE and compiler behaves identical across platforms.

We will keep you posted on our progress!

Changes, fixes and debugger

These are exciting days and Quartex Pascal is being polished, refactored and optimized on a daily basis. For the past week or more my focus has been on the RTL, while Kjell is busy with the IDE and making our toolchain more platform independent.

I could probably list a wide range of tickets we have closed, but I want to focus on the RTL changes since that is at the heart of things. To sum up the most important changes in the RTL this week:

Align: TAlign now works brilliantly
Bugs with regards to async constructors are gone, the source for the strange inconsistensies was simply that people (myself included) had tried to achieve too much inside callback mechanisms, causing synchronization and timing issues. It is very tempting to adopt some of the flamboyance of JavaScript when working with QTX, but the framework defines rules that are there for a reason.
Moved execution of the constructor callback to AFTER the widget has been injected into the DOM, but before ObjectReady() is invoked. This is actually a massive change, since it radically changes what you can do in the constructor callback. It also affecs nested constructors and when they finish (read: when widgets are available in the DOM and can be accessed).
By getting rid of the issues above, Javascript JIT kicks in and is able to optimize the code far better than before. I was also able to remove bloat and fine-tune things now that ready-state is no longer a factor.
TQTXSyncWidget will be depricated since there is no longer any difference between TQTXWidget and TQTXSyncWidget.
Sourcemaps and debugging!

As you might remember from previous articles and posts, the difference between Delphi or FPC’s component constructor and our’s, is that we have a second, optional parameter. An anonymous procedure (a.k.a callback) that allows you to daisy-chain constructors and together – and thus initialize properties immediately, as a part of the constructor itself, rather than afterwards.

The benefit of this technique is that you can “recursively nest” work together, forcing JavaScript to create the entire UI in a single swoop. This technique is something that JSVM (JavaScript virtual machine) excels at and agressively JIT optimize -which is ultimately why QTX code is blistering fast. Even for large and complex UI designs containing hundreds, if not thousands of widgets.

The downside has been that you have been limited to two things:

Create child widgets or objects
Initialize properties

If you tried to do anything else, like accessing a widget further up the chain (which might not be ready yet due to the reverse nature of daisy chained constructors) you would run into access violations.

To make this more clear, imagine you daisy-chain three constructors together to create three widgets as quickly as possible:

First widget
- Second widget
  - Third widget

In the above sequence, the third widget would in fact be available first, since that is at the end of the recursive sequence. This means that if you tried to access the first or second widget from the third, you would cause an access violation. A widget is by nature only ready to be accessed after it’s callback had finished.

Making it easier

Since this clearly has been counter intuitive we decided to move where we invoke the callback. We have moved it to a spot after the widget is ready to be used, but before ObjectReady() is invoked. This means that child elements will already be in the DOM when the callback executes, and that elements further up in a nested sequence will be available.

Code such as this will then be perfectly valid:

var fHeader:= TQTXHeader.Create(self, procedure (Header: TQTXHeader)
begin
?Header.ButtonLeft.Caption := "Back";
?Header.ButtonRight.Caption := "Forward";
end);

In the new RTL timing is more natural and intuitive

The above code failed in previous versions of the RTL, because there was no guarantee that TQTXHeader had in fact created it’s child buttons. The ButtonLeft and ButtonRight child objects would have been nil.

Since we know the callback and it’s limitations has been confusing and counter intuitive, especially the limitations — we decided it was worth the effort to change it. Most people will immediately assume that they can access widgets and objects there, so it makes sense to meet that expectation as best we can.

We do still suggest you override the InitializeObject() and FinalizeObject() methods rather than the constructor, but these methods are just time-savers since you dont have to type the entire constructor syntax and callback type.

Consequences

The consequences of this is surprisingly little and only affects the RTL and component writers. In 99% of the cases I have encountered so far, the only consequence is that creation-order is flipped. Like outlined above, the last constructor would finish first in the old RTL, but now they finish first-to-last (in the order you have nested them).

The only UI change is (so far) with regards to zIndex. I am sure there will be edge cases, but updating code written especially for the older RTL is very simple. We are talking one-liners or, at most, re-arranging the sequence you create child elements. You can still use WhenReady() to line up work that needs to happen when ObjectReady() is invoked, just like before. So the change is abysmall.

Since Align: TAlign is now supported out of the box for all TQTXWidget’s, you dont need to override Resize() to manually deal with layout. So all in all writing your own complex widgets is now easier than ever.

Sourcemaps and debugging

We still have some way to go before Chrome’s debugger protocol is fully integrated with the IDE, but we are making excellent headway. The compiler now emits the source-maps, and the IDE clones out the pascal code (which is needed by the browser debugger if you wish to use Chrome’s tools directly) into it’s own debug folder.

Once we get more tickets done the goal is to talk directly with Chrome (or Edge) so you can set breakpoints directly in the pascal editor, inspect variables and parameters, and basically work with the code like you do in Delphi or Lazarus. The protocol is already wrapped and ready, so this will be awesome!

Support for Linux, MacOS and ARM

We have already gotten the compiler to spin on Linux, MacOS and Windows from the same codebase, and likewise for ARM Linux (suitable for Raspberry PI, ODroid, Rock Pi 5 and similar SBC [single board computers]) so that is exciting!

I want to add Risc-V to the list of platforms, but we have to research what that entails or how well Freepascal supports the platform. From what we understand so far, support is fairly complete and it should be possible to compile for Linux on Risc-V once our codebase is 100% platform independent.

The goal ofcourse is to make both the IDE and compiler 100% platform independent (the compiler already running on said targets and chipsets), so that it compiles both under Delphi and Freepascal. This will require a substancial refactor of our codebase, but it’s something we want to do. This will have significant benefits for us both in terms of product demographic and adoption, but also for schools and educational institutions where affordable ARM devices is used for IoT. Being able to work on such devices is definitively a great bonus!

Stay tuned, QTX is coming!

Quartex is now a company

It is with great pleasure that I can inform everyone that Quartex Pascal is now a company with a fully funded three year runway. This means that Quartex Pascal as well as the desktop project will see full-time development the next three years.

As of such, the Patreon project will shut down mid December. Don’t remove your backing just yet, i need time to register each name and the tires so that backers can receive appropriate licenses for having backed the project. I will inform everyone in good time.

Having sacrificed more or less every weekend for the past four years, it feels good to have arrived at a place where i can focus more deeply on the tasks, rather than jumping in and out with a full work-week between sessions. I can barely remember a weekend I did not work on QTX in some capacity. I must admit I had hoped the Patreon model would have seen stronger backing, but at least I gave the alternative routes a fair chance.

I think products like this can only be realized through traditional, commercial mechanisms. That is ultimately the lesson I have learned after four years of alternative paths. C# and Miguel de Icaza’s story is somewhat similar. He barely made any money on donations and backing (and that was in California, at the heart of Silicone Valley) and it was ultimately thanks to Novell investing in C# and asp.net for Linux that Xamarin Studio (commercial mono, much better than VS) came into existence. Xamarin Studio was later bought by Microsoft, and the mobile development framework and compiler in VS is ultimately Xamarin. Without commercial investment, Xamarin would never have been a success.

Anyways, updates will appear on Patreon for December, and after that everything will happen via the official website here. The Facebook group will remain active, and it will become an open user-group like Delphi developer, FPC Developer and C++Builder Developer – which are the developer groups I manage on Facebook.

There will no doubt be some pause in January as I implement the license server so updates etc. can be automated, but news will be posted here starting with the release of version 1.0 of Quartex Pascal.

So there is a lot to be happy for right now! Finally I can work full weeks on QTX and have normal weekends again

Now for the third chapter in a very exciting journey