Update to latest DWS in the works

Like I mentioned in the previous posts in Patreon (backers only), we have precious little tickets left! And I have started on the last big change, which is to update DWS (Delphi Web Script) to the absolute latest, bleeding edge version.

This brings us a number of advantages, most notable is the fancy new async and await keywords. So you will be able to define a procedure as purely async, and further define code that waits for that particular condition to finish or fail. The mechanism behind this is called a promise, which is a common JavaScript technology by now – implemented at the lowest levels of JSVM both for the browser and Node.js.

This will have a profound effect on how we write wrapper code for external libraries. At the moment we have to use a bit of inline javascript in ASM sections to pascalify these mechanisms — so it will be a great to finally be able to work directly with promises as a part of the pascal language.

A pseudo example is when downloading a file via the fetch() browser method:

await fetch(url, procedure (data: JByteArray)
begin
// Success code here, bytes in the data param
end,
procedure (error: JError)
begin
// Error handling here
end);

Timeframe

Since we forked DWS some 3 years ago, there have been a lot of internal changes to the compiler and infrastructure. This includes name changes that, while superficial, affects more or less every inch of the IDE code. I cant just do a mass-rename inside the source-code either, since there have been subtle changes in a few public elements that are now protected.

In short, this will take a week more to get to grips with — but it will be worth the wait!

Website, forum and gearing up for RC1

As we are nearing RC1 with only a handful of tickets left, there are about to be some serious changes to our infrastructure.

So far we have operated with Patreon as our means to finance the development, and Facebook as our daily driver for live status, quick feedback and general community talk. This model has served the project well, except that Facebook really is a terrible place to run a group that is 100% dependent on a single person.

Facebook supports multiple account, and you would imagine that you could establish a business account and a personal account – and that these would be completely separate from each other. Sadly that is not the case, and should my personal account be blocked for whatever reason the Facebook A.I decides that day, the business account becomes locked as well.

It was never a long term plan to stick with Facebook, so it was time to bite the bullet and purchase a professional installation of WordPress with a forum plugin. And this is exactly what I have done. I was aiming to wait with that step until after version 1.0 was out the door, but Facebook has made it difficult to have both a personal life and professional life (which I consider separate) on the same platform. I have outlined at length how absurd the Facebook A.I operates, how it mistakes Norwegian words for English (with disastrous results) and so fourth. You can read that article here.

New model

Before x-mas I did a couple of polls in our Facebook group regarding how we should proceed. Regardless of the above mentioned debacle, Facebook as a platform is really only suited for user-groups and keeping track of friends. There is also the financial aspect of the product to consider. We need a stable income to ensure stable development of future versions. If we manage to secure enough, I would be able to work full time on this rather than just weekends. That would be much more effective for everyone and I would be able to expand the range and power of the product a thousand fold compared to what we have today.

The result of the polls were quite simple:

  1. We continue to use Patreon for handling backing and donations
  2. The Facebook group will become public (read: anyone can join, not just backers) much like Delphi Developer, FPC Developer and the other groups I run in the same family.
  3. New binaries will exclusively be issued via Patreon
  4. Community edition will be available on this website, but that will only be major releases (LTS). Those that wish to have running updates, patches and cutting edge documentation must subscribe to a Patreon tier. This is what everyone else is doing and it makes more sense. There has to be an incentive to get the cutting edge stuff (and yes, the tiers will be adjusted to better suit this model).

To give you some idea of cost, a year worth of $25 a month backing, which gets you binary updates (including docs) and articles is exactly $300. Keep in mind that this also includes the cluster environment, a ton of new components I will write, and all the fancy stuff that is lined up for version 1.1 and 1.2 (like database components, filesystem over WebSocket and so on). The hard work with the IDE is done, and we can polish and make sure the details get better – but all in all the IDE infrastructure is ready to rock. Focus after version 1.0 will be more on content, demos, articles, ready to use project templates, new packages, utility features and drop-in protocols (yes you read that right). This is where the fun begins.

This is a product that we can aggressively expand to consume pretty much whatever we want of the JSVM / React / Node.js world. We don’t have to wait for Idera or Embarcadero or anyone else. QTX is a free agent and answers to nobody but the backers. Once we agree on a tech, I get to work and make it a part of the IDE and RTL. And this also includes exciting stuff like bytecode assembly, WASM and baking native executables via Phonegap and React directly from the IDE.

In plain English: We can’t keep going like we have up to now, because we are financially nowhere near where I can jump over full time (or 50% for that matter). The only way to get anywhere close to a 50% position, is to get everyone over on Patreon. Even if it’s just the lowest Tier (I will adjust the tiers) that would radically change things. It’s a long term goal, but it’s there.

There is also something to be said about simplicity. Having to re-post the same news in 3 places is a bit much. At least now we have this website as the main focal point and Patreon for backers and “subscriptions”. So feel free to post to the forums here on this website (you probably need to register first).

QTX reaches code freeze

I am happy to report that Quartex Pascal has reached code-freeze. This is the stage where we stop researching and adding new features, and instead focus on what has already been added. Basically this is where we polish, refactor and make the codebase production ready.

Its been a journey but I am finally getting close to RC1

During the research phase of the development system there were several tickets that I left in semi-working condition. Features that I know how to solve and roughly how much time they would take to complete. The worst thing you can do is to push the difficult and truly challenging features towards the end of a project, because you risk ending up with a product that simply don’t work as expected.

Code-Freeze is the timeframe where all the unknown factors have been solved, and I systematically go through the project step by step, completing the tasks that I left in semi-working condition. It is also this phase where backers and testers report findings back to me so they can be corrected.

Tickets and latest fixes

There have been a ton of fixes since the last post on this website. If you want to get up-to date information, please see the Patreon website. Backers get access to both Patreon downloads and our Facebook user-group where we post daily and weekly. Below are the highlights for the past 4 months:

  • Code suggestions
  • Parameter hints
  • Mouse-over hints
  • Project templates
  • Externalized graphics
  • Delegates
  • Automatic form creation
  • Ragnarok protocol designer
  • Remaining tickets
Code suggestion

Two features that is considered common for all IDE’s (integrated development environments) is without a doubt automatic code suggestion and parameter hints. No matter what language you work with you expect the IDE to perform live code analysis and present you with available members, properties, fields and bindings at language specific junctions. In Object Pascal whenever you type a fieldname or variable followed by “.” (punctuation), the IDE presents you with a pop-up dialog with suggestions.

Code suggestions now work brilliantly, and very fast I might add
Parameter hints

The same is true for parameters. Whenever you have typed a function or member name, followed by “(“, the IDE is expected to present a hint box that displays the parameters of that member. If the member is overloaded and have multiple parameter options – it should display them in a top-down list fashion. It is also common that these hints remain visible while you type, highlighting the parameter item you are typing.

Parameter hints now work as expected

These two features are actually very, very difficult to create. They require that the IDE parses and maintains an AST (abstract symbol tree) in the background that is queried live with regards to relevance and visibility of whatever symbol you are addressing. Without getting into the nitty gritty of it all, these features were quite difficult to get right, but they are finally in place.

Mouse-over hints

I also finished the mouse-over information, where you move your mouse over a keyword or symbol and information about it is displayed in a standard tool-tip. These features all rely on more or less the same functionality, so once I had a working model it was relatively easy to kill three birds with one stone.

Mouse-over hints are now in place. This feature has room for future improvement, but its a solid start
Project templates

The IDE operates with project templates much like Embarcadero Delphi or Lazarus. This means that I have prepared a particular project which is registered with the IDE when it starts. When you create said project, the folder that makes up that project is cloned and saved as your new project.

The project cloning tool simplifies creating your own project types

Since the RTL (runtime library) spans both HTML5 (browser) and Node.js (server side scripting), the IDE recognizes different project types. When a template is registered the type of project is very important, otherwise the IDE wont be able to distinguish the rules for each type. For example, a form based HTML5 project cannot contain code that is designed to run on the server, and obviously server-side code wont work in the browser.

This is further complicated by the distinction between forms and windows. A window is technically a form, but it has a title-bar, close, minimize and maximize gadgets, can be resized by the user and moved around. Without a correct template type during registration this system would be a complete mess.

The project templates have been made up-to-date. There were a few mistakes that had sadly been replicated between dynamic, tabbed and windowed project types, but there are now fixed.

The IDE also allows anyone to create their own project types without leaving the IDE. If you right-click a project in the “New project” dialog you can select “clone” and you are presented with a dialog that helps you. This feature is handy if you have wrapped a JavaScript API and want to simplify projects based on that API.

Externalized graphics

At the moment we are using free glyphs (icons) from Icon8 which look good in both light and dark theme (imho). But having these hardcoded into the IDE executable seems less than ideal considering that we support themes for the IDE. As such I have implemented a glyph manager which loads the graphics from disk when the IDE starts.

The glyphs and icons used by the IDE are now purely loaded from disk and can be replaced

This means that people can replace these glyphs themselves if they so want to. The IDE now supports both 24×24 and 48×48 pixel glyphs. By default the IDE uses the 24×24 glyph size. The larger glyphs can be handy for people that suffer from poor eyesight, which is why I included them.

The icons for the code-suggestion dialog is likewise loaded from disk. As of writing these are 16×16 pixel in size, but the IDE operates with a row-height of 24 pixels (smaller glyphs are vertically centered), so these glyphs too can be replaced.

Delegates

The IDE now supports visual delegates for all widgets (visual controls). Delegates is a concept that Delphi and Freepascal developers might not be too familiar with, the closest thing in the pascal world is an earlier COM concept called multi-cast events.

All visual widgets can have delegates added to them, as many as you like per-event

In traditional Object Pascal each event can only have one handler procedure. This is because pascal events are essentially just a pointer that glues caller and called together. Delegates however are objects that wrap an event and offer multiple handlers to be registered. In short, if you want 10 event handlers to fire in response to your OnClick event, you can register 10 handlers to the same delegate and they will fire in the sequence they were added.

Note: Quartex Pascal still supports ordinary pascal events, but these are used purely by code. They are still the fastest in terms of execution and very efficient. But since HTML5 and Node.js relies heavily on delegates – it makes much more sense to use that for UI elements.

Adding 3 separate delegate handlers that all fire in response to an OnClick event

When you add a delegate to a widget on your form or window, you can double-click the delegate handler and the IDE injects a procedure handler for you (just like Delphi does for common events). If you double-click on an existing delegate handler, the IDE switches to the edit-tab, locates the handler, and places the cursor on the first line.

Automatic form creation

This was added quite early but due to a bug in how I encoded information we had a build where the form-path was stored rather than the form name. Needless to say this caused havoc as no forms were created when applications started.

This was easily fixed and the projects affected by it were cleaned up. You can check each of the forms you wish the application to auto-create when it starts inside the project options.

Project options will list the forms in your project and you can check the ones you want created on startup
Ragnarok protocol designer

Ragnarok is a JSON based message format that simplifies message dispatching between client and server, as well as routing messages in a cluster architecture. It is lightweight and simple in terms of data.

As is the case with all communication, it is the scaffolding that is time consuming. The code that parses a message, that recognizes it, that delegates the message to its handler – and that makes sure the response is issued to the correct recipient. Keeping track of messages in an asynchronous environment can be tricky if you dont build the correct infrastructure — which is where Ragnarok comes in.

When you add a protocol to your project, you can double-click and edit the protocol in the IDE

The IDE now has a built-in protocol designer. This is where you can define datatypes and messages that will be shipped between client and server (or between processes for that matter).

Once designed, the IDE will automatically convert the protocol file (*.rrp) to pascal code. Which means you don’t have to waste time writing message envelope code and instead focus on populating and working with the messages. These messages are exceptionally well suited for WebSocket, but the RTL also comes with an UDP server (node.js) and client that can likewise be the medium.

Ragnarok as a format is not bound to the actual transport of the messages. You can ship these messages via any medium you like, be it HTTP, WebSocket, UDP, TCP or IPC (inter process communication). The Quartex Media Desktop actually uses Ragnarok as the default protocol system between host (desktop) and applications. All “soft-kernel” calls are done purely via messages.

Note: This system will grow considerably as Quartex Pascal is further developed. There is a lot of room for expansion and we can implement some amazing technology here.

Remaining tickets

As of writing we have only a few tickets left. There will always be more as we dig into the codebase, small things that we have forgotten – or sloppy mistakes like not setting the tab-stop correctly, but all in all we are in good shape!

Right now I am working on the Besen2 Javascript provider class. The IDE has a provider class for each filetype it supports. This provider class deals with parsing and maintaining the TTreeview (unit overview).

After that it’s time to finally give the Search & Replace dialog some love, this has been waiting for quite some time. But after that, it’s pretty much done in terms of core features. There will always be minor adjustments and tickets to fill, but based on what we aimed to deliver in version 1.0 — we have reached our goal.

These are the tickets remaining

Better information pages!

This website is meant as an information page for those interested in Quartex Pascal or the Quartex Media Desktop projects. The idea of this website is that, once version 1.0 is out – this is where people can go to download binaries and the community edition. But until that is the case, news is posted weekly to Patreon and ad-hoc to our Facebook group.

The Facebook group is for backers only at this point, but once version 1.0 is out we will open it up for everyone. It will become an interest group much like Delphi Developer or many of the other language groups on Facebook.

Anyways, I have spent some time writing more about Quartex Pascal and it’s features here. I also updated the Quartex Media desktop so that this has it’s own page and menu item too, making it easier to get familiar with both.

I will continue to write and add more and more information (and split the page into sub-sections rather than one massive document) as we move forward.

Getting involved

If you find Quartex Pascal interesting and want to back it, head over to Patreon and sign up there. After you have signed up, head over to Facebook and join our group there for chat and questions. New binaries are published mostly weekly, but some times it can take two weeks between drops. The latest binary is always pinned at the top of our Facebook group, and you will find it on Patreon as well.

Cheers!

This week’s fixes and build

A few bugs were reported on the last build, which have now been promptly fixed. I have also updated our to-do list so that everyone knows where we are with regards to the first beta release. It might look much, but keep in mind that most of these task take 1 hour tops each. Only the most elaborate tasks will take longer.

Note: This post is 3 days late. I came down with a nasty cold on thursday and pretty much spent the entire weekend in bed. A national holiday started on monday, and it was my time to take the kids on a trip. So it’s been a less than productive weekend, but that’s just how it has to be. I always make up for lost time.

Ok, let’s go through the tasks for this build.

Bugfix list

  • A bug where starting two instances of the IDE caused the latter to throw an access violation
  • A strange case where the resize grab-handles for a selected widget suddenly would not render
  • The caption property was not set on windows (TQTXWindow) in windowed projects. This turned out to be quite critical
  • Windowed project file was missing the {$I “app::form.init”} compiler injection point, meaning that forms would not be automatically created. Instead the windowed default project operated with code that created windows manually

Multiple instances of the IDE

It was suggested that we use a Mutex to make sure only one instance of the IDE is running. However, the IDE should be able to run in multiple instances by design. The reason for this is because it simplifies writing client / server solutions on the same machine.

After the fixes mentioned below, you can now run multiple instances of the IDE simultaneously

The IDE is initially designed to handle multiple projects simultaneously, but to save time I decided to push that feature into the future. It’s not a critical thing at this point. Not as critical as getting the first version out the door in good condition is.

Log4Delphi

When inspecting what caused the problem I discovered that it was in fact the logging system that was the first culprit. We use Log4Delphi which is a 1:1 conversion of Log4Java. All in all a very simple system that I could have written better myself, but again – time is a factor here since we are overdue by almost 2 months. Hence I just picked a ready to use logging system that we could use without much hassle.

Turns out that Log4Delphi opens a permanent filestream without any sharing flags (meaning that it locks the file for read and write access by other processes). Since the Log4Delphi is open-source I simply went in and added the fmShareDenyNone if the log is opened in either fmOpenRead or fmOpenWrite modes.

The IDE now creates logfiles with filenames based on date and time, each instance thus operates with its own logfile

I decided to make it more bullet-proof, so I implemented a filename function that generates a filename based on date and time. This ensures that two instances will never try to own the same file – at the cost of each IDE operating with separate logfiles. Not a huge deal I feel, since the logfile is not a permanent fixture – we will be logging to standard WinAPI after the release candidate.

TZipfile and packages

The next culprit that prevented multiple instances to run was the package loader. Quartex Pascal uses ordinary zip-files as the package format. Zip-files are almost universal and natively supported by Windows, so using that for packages makes a lot of sense. This way developers can easily create package-files for their components ready for distribution.

QTX packages are just ordinary zip-files with whatever resources your widgets need. Widgets are registered via Attributes.

The problem was, again, that TZipFile’s Open() method does add the fmShareDenyNone flag, effectively locking the package for the entire IDE session. Since TZipFile is a part of the Delphi RTL I can’t simply copy the code and change it to better suit our needs. But as luck would have it there was an overload for Open() that takes a TStream. So I just modified the Zipfile filesystem driver to open the package file via a TFileStream with fmShareDenyWrite flag (allowing reading but refusing other processes to write to the file).

With this change, you can now run multiple instances of the IDE without problems.

Form designer grab handles

Before the previous release I more or less re-coded the selection logic for our form-editor. A form editor that supports the features we do is not easy to write, especially since the widgets are not Delphi TCustomControl’s (like most other designers operate with), but are drawn and represented purely via TCanvas.

The code had reached a level of complexity where, if we continued, it would have grown into a spectacular mess. So I decided to refactor the entire selection system, which is spread out over the 3 common mouse handling methods (mousedown, mousemove and mouseup).

Grab handles for widgets and the form is now back. The logic for these will likewise be refactored

For some reason I had forgotten to create the TProxyResizeIndicators instance in the selection situation code (all actions you can do like select a single widget, multiple select, add or remove from a selection — these all make up specific “situations” that the designer must check for and respond to). I also needed a call to __ImmediateRedraw(). Once these were in place, the resize-bars were back as they should be.

The missing caption

When I implemented the code for the last build I was quite tired, but how I managed to miss this puzzles me. The codegen that converts the prototype (a “prototype” is an object that contains all the published properties for widget. The IDE scans each widget when it boots, and whenever you drop a widget on a form, it clones out the properties that the inspector can work with) for a widget into code at compile-time, a process which is recursive and has zero room for errors.

For some strange reason the TViewImplementationCodegen was missing a call to EmitPropertiesFor() where the properties for the form was generated for the TQTXForm or TQTXWindow itself. This is obviously due to the fact that we waited with that part of the RTL until we had delegates added to the codegen.

The properties for TQTXWindow and TQTXForm are now written in full, solving the caption issue

It is a good thing we noticed this, because the code was only emitting properties like width, height and name. This meant that the Caption property was effectively never set! It would also mean that none of the other properties would be generated. A bit sloppy, but the code we are talking about here is very dense. Recursive code on this level is easy to get wrong, believe you me. You have to take account for every situation, and distinguish between subtle differences.

For example, if the project is of type dynamic, the height property should never be written (otherwise the page would have fixed sizes on the content and not scale). Windowed projects are different and must have width, height etc. set early in the constructor code — so there are a lot of levels to this type of code.

Updates the Windowed project template

The code for windowed projects is old and was written just before we got delegates into the mix. At that time the IDE did not recognize TQTXWindow as a form, and would only deal with TQTXForm. As such the windowed project type would have code to manually create a TQTXWindow, and there was no DFM file to house the design – and consequently no way for the IDE to automatically create windows added to the program.

With the IDE now aware of TQTXWindow, the standard “$APP::form.init” injection point works as expected

This has now been updated and the new windowed application type works exactly like a normal dynamic project.

What is left before we can enter beta?

There are quite a few minor additions and changes we have to implement before we go into beta. The “bugs” above are good examples of smaller tasks that can be found here and there throughout the codebase. Nothing critical or overly complex, but they have to be dealt with.

Right now I am operating with the following task list:

  1. Delegate event code. When you double click on a delegate the IDE should either inject the event handler (like Delphi does) if it does not exist, and consequently switch to the code-tab and goto the procedure.
  2. Delegate add dialog. The “new delegate” must have a name-field and also be simplified, right now the design makes little sense.
  3. Delegate edit dialog. This is essentially the same dialog as above, but without the code for adding anything, it should be edit only, allowing you to change the delegate class-type and name. This is more elaborate since it also requires the form code to be changed automatically by the IDE.
  4. The TRagnarokProtocolImplementationCodegen class must be completed. Right now the codegen for ragnarok protocols emits the interface, but not the implementation. Again this type of code can be complex due to its recursive native, but I estimate a full work-day to get it done and ready for testing.
  5. Ragnarok project types: we need to make it easy to create ragnarok based projects. This includes node.js servers, node.js system service, dynamic and windowed client. Ragnarok does not impose itself, so you can add a protocol to any project and the IDE will spit out the code for it, but having ready to rock project types will make it easier to get to grips with.
  6. The preferences dialog has a lot of options that must be finished. Also, the layout needs a refactoring, especially for the SynEdit color palette stuff. It is less than intuitive.
  7. The HTML dialog must be refactored. Right now we are using the default HTMLComponents editor with toolbars, and the toolbars dont really work well in a dialog form. A traditional toolbar must be implemented. This should not take long since all toolbar features are isolated as TActions
  8. The IDE must have dialogs for all supported dialog types (around 10), right now we have 2. Thankfully a lot of this is boilerplate and functionality is largely displaying options and writing them to the widget prototype
  9. TQTXMainMenu editor. Not critical but I think this is a very important feature. Being able to create windowed applications easily for HTML5, mobile, embedded etc. requires a good menu editor. We might push this to version 1.1 is we dont have time.
  10. Probably 100 little things and tweaks, but thankfully nothing fancy or huge, one-liners mostly and smaller graphical adjustments.

As always, the latest build will be available on Patreon and for our backers on Facebook in Quartex Developer.

New Build 0.15.6.9b

A somewhat curious bug had appeared in the latest build, where *.wwr files (web worker files) were no longer recognized by the IDE. Turned out I had somehow typed “.wrk” when refactoring the file-type association code, resulting in the IDE delegating the task to WinAPI.

This has been promptly fixed and a new build can be download from Patreon.

New website, easier access

Quartex Pascal has been in active development for the past 18 months. The project is overdue by a few months due to covid and associated factors, but we are quickly closing in on the release of version 1.0 (hurray!)

Since Quartex Pascal has been a sort of sub-community-under-the-radar kind of project for so long (bordering on a skunkworks project), becoming more public and easily available is more than overdue. It is important to make things easier for new backers and users as they come along, and having an official website achieves that.

Patreon and Paypal

The project is still a Patreon project, just to underline that. This website here has no means to accept funding or donations, so if you feel like helping out then we have two ways you can do that:

  1. Visit the Patreon page and pick a tier to support
  2. For donations and single-shot gifts, use Paypal (if you donate 100€ or more, you have full access to our invite-only Facebook group. A thriving community of backers and developers. Simply PM me on Facebook afterwards to get access).

While version 1.0 will be out in a little while, that is by no means the end of Quartex Pascal. We have a lot of technology just waiting to be included in the IDE and RTL! Things like data aware controls, a full database editor, REST and SOAP protocol support, full NAS OS (the Quartex Media Desktop) deployment, ASM.JS and Webassembly, building native applications for desktop and mobile via Phonegap and similar technology — and much, much more!

Update for Customers Using PhoneGap and PhoneGap Build | by Adobe I/O |  PhoneGap
We have so much cool stuff lined up for the next versions, it will make mobile and embedded development ridiculously easy

The way I look at it, Quartex Pascal version 1.0 represents the foundation stone. It is the foundation that we construct better and more elaborate technologies on. The world of web technology has so many exciting avenues that object pascal developers can make use of – the sky literally is the limit here.

Easier access to material

This website will also serve as an easy access point for media. I have several presentations that I have held regarding QTX and it’s associated cluster technology, and putting everything in one place for easy access is the way to go. We are slowly moving from our garage days into a more office like modus operandi, so bear with me as go forward – it will be awesome once all the dots align.