Home » Company » Sample applications » “Some Degree of Introspection”

“Some Degree of Introspection”

Danger, Will Robinson!

This is not a subject for the uninitiated.

Unless you have delved into that ancient grimoire of forbidden lore “Understanding FMAC“, penned in the deeps of time by the legendary black magician Jeffery B. Jarvis, you might want to look away now!

Evil face

This is where it all started really.

One evening in February 2009 Sture Andersen and I were in his flat in Copenhagen, doing what we often do: having a few beers and discussing things.  As founder members of the small informal group of VDF developers John Tuohy had christened “The JSON-heads” (AKA “The JSONites“, although, as Klaus Berthelsen later pointed out at Synergy in New Orleans, this was not quite correct, since properly the followers of Jason were called “The Argonauts” after their ship, the Argo), we were trying to work out how we could build on VDF’s newly acquired capability (in VDF 15.0, then still in an alpha-test version) to return web service responses in JSON format to deliver a full “JSON-in/JSON-out” interface.

John Tuohy had wriiten a little JSON parser in VDF, so that should allow us to interpret any JSON-formatted data we received.  The parser returned the data in a “ValueTree” structure: a very simple (but recursive) data-type made up, at each level, of a value and an array of child ValueTrees.

I was aware, from working extensively with the internals of the VDF web service client implementation, of (and John had also pointed out) the existence of a command – the awkwardly named “ValueTreeDeserializeParameter” – which could take a ValueTree and move its contents into a struct variable.

The problem was that for this to work, the data in the ValueTree had to be in exactly the same form as the elements of the struct variable (and John warned that really bad things – up to crashing the web server – might happen if it wasn’t)… and this could not be deduced from the incoming JSON data, because JavaScript makes no guarantees about what order the properties of an object will be enumerated in, so that order might easily differ from implementation to implementation (or even from enumeration to enumeration in the same implementation, although this seems less likely).

We needed the JSON parser to be able to work out the names of each of the elements of a struct definition, so that it could place the appropriate decoded JSON elements in the right order into the ValueTree (and indeed discard any inappropriate ones if that were required).

This would involve being able to “see into” a struct definition at run-time – a technique known as “reflection“, or “type introspection“.

What we need is some degree of introspection,” said Sture… then he got a faraway look in his eyes and muttered something like: “We could modify the struct command to keep a track of its members at run-time.…”

Sture is never really comfortable more than a few metres from a keyboard, so inevitably he started trying things and, although he didn’t get there that evening, once he gets excited about something, he usually finds a way to make it work… as he did in this case.

The Visual DataFlex compiler directive pair #COMMAND / #ENDCOMMAND (they don’t need to be in uppercase – it is just an old habit) allow for the creation of custom commands.  This is a technique that had much greater applicability in the old days before the 1991 release of DataFlex 3.0 introduced procedures and – even more importantly – functions into the language, but has been largely deprecated since, so that it is now neither recommended nor supported by Data Access.

Although #COMMAND is normally used to create new commands, what is less well known is that it can also be used, in conjunction with the #REPLACE directive, to redefine, or overwrite, existing commands, and it was this aspect that Sture was thinking of when he talked of “modifying” the struct command.

That however turned out not to be possible.  It seems that some commands enjoy immunity from being redefined in this way (Sture calls them “holy“) and “struct” turns out to be one of those, defeating his initial approach to redefining it in program code.

But… there is another way.

All of the core VDF commands are defined in a text file called FMAC, which still ships with Visual DataFlex.  FMAC itself is not referenced by the VDF compiler at compile time, rather a binary version of it called Flex.cfl is where the complier gets its command definitions.  You can find both of these files in the “lib” subdirectory of any VDF installation.

Convering the textual FMAC into the binary Flex.cfl is the job of a program called DFPack – a character-mode utility which does not ship with Visual DataFlex: Data Access really does not want people interfering with the core of the language in this way.  Even in the days when creating new commands was commonplace (or relatively so), making changes to FMAC (and subsequently “packing” them into custom versions of Flex.cfl) was very strongly discouraged – and for good reason.

However we had come to an impasse: we needed the struct command to do more than it currently did and it could not be altered by redefinition in our program code.

So Sture made a very small change to FMAC, then used a copy of the DFPack utility (still available in the DataFlex 3.2 character-mode product and still compatible with VDF) to pack that into a custom version of Flex.cfl.  (Here I should also point out that Sture was working on workspace-local copies of FMAC and Flex.cfl, not the master versions in his VDF installation “lib” directory.)

The changes he made were as follows:

#COMMAND _STRUCT_ATTACK // New command. Does nothing.   Move Windowindex to Windowindex // <- This is nothing. Could be removed#ENDCOMMAND// note that this command also creates a symbol _struct_XYZ where XYZ is the...// be used with #ifdef to see if a struct is defined#COMMAND STRUCT R   #IFDEF __@INSIDE_STRUCT@__      #ERROR DFERR_COMP_ILLEGAL_CODE_PLACEMENT "STRUCT command within a STRUCT...   #ELSE      #STRU !1      #SREP __@INSIDE_STRUCT@__   |CI1   // lets other commands check that they...      _STRUCT_ATTACK !1 // <- This line is added by Sture   #ENDIF#ENDCOMMAND

As you can see, the code first defines a new command “_STRUCT_ATTACK” (which really does nothing – the single line in it could be removed), then inserts that command into the STRUCT command proper.  In effect, this does nothing at all.

Having placed this “hook” into the struct command, Sture was then able to create his “SomeDegreeOfIntrospection” package.  In that he was able to redefine his new “_STRUCT_ATTACK” command (and also a number of other commands relating to struct members) so that it did things.  Exactly what is done there is too complex a topic to discuss in detail here, but the net result is that it builds a global array of struct definitions which other code (crucially the JSON parser) can reference at run-time to determine the names and positions of struct members.

This gave us the vital link between the inbound JSON data and the complex data-types we needed to copy it into in order to pass it on to the Ajax web service functions (or indeed those of any other web service).

Health Warning!

We should note that in his own company’s production code, Sture does not use this method, deeming modification to FMAC as too risky a thing to expose his customers to.  Anyone considering using this technique should take that judgement – from its inventor – on board.

Sture makes these points regarding modifying FMAC:

  1. It adds an uncomfortable step when installing VDF Studios.
  2. I find it impossible to bug-track.
  3. Even in support incidents not involving it, it will attract suspicion. DAE once gave up using it in a project because of side-effects.
  4. And when it came to it, I didn’t dare put it into production. That perhaps is the worst recommendation of them all.