The Lib folder/directory contains tools maintained and and periodically updated by MPE. The contents of Lib differ between the Windows, Linux, OS X and DOS versions as some of the tools are operating system specific.
Cross reference information helps you to manage your source code. When LIB\XREF.FTH is loaded you can use XREF <name> to find out in which other words <name> is used. You can also find out which words you defined but did not use. XREF is precompiled in the Studio version of VFX Forth but not in the base version.
The compiler generates cross references by building a chain of fields including LOCATE format (link:32, xt:32, line#:32) in a separate area of memory. Links and pointers are relative to the start of the XREF memory area.
Two chains are maintained. The first produces a chain of where a word is used, so that the user can find out where (say) DUP is used. The second produces a chain of which words and literals are called in order. This is the basis of decompilation and debugging.
XREF is initialised by the switch +XREFS and is terminated by -XREFS. You must use +XREFS to turn on the production of cross reference information.
By default 1Mb of cross reference memory is allocated from the heap. If you need more than this for a very large application, use the phrase <n> XREF-KB to set the size of the cross reference memory, where <n> is in kilobytes.
Because the VFX code generator optimises so heavily, there is no direct relationship between the binary code and the source code. Consequently DIS and DASM use disassembly and special cases, but cannot produce a good approximation to the original source code.
The cross reference information includes a decompilation chain. When you use SHOW <name> the cross reference information is used to produce a machine decompilation. This includes none of the comments from the original source code, and is machine formatted.
The decompilation produced by SHOW is mostly default and automatic. However, some words such as string handling take in line data which would not be displayed by SHOW without special handling.
SHOW can be extended by adding items to the DCC-SWITCH chain. The stack effect of the action is: addrx -- addr ; where addrx is the offset of the cross reference packet in the cross reference information memory. See the /REF[X] structure in LIB\XREF.FTH for details of the structure of this data packet. The example below is for a word X" which takes an in-line string like S".
|
Note that unlike previous VFX Forth decompilers, SHOW is based on cross reference information which references the source word without knowledge of what it compiles. The only reasons for special cases are control of the decompilation layout and display of associated data to reconstruct source code.
: dump(x) \ offset len --
Displays the specified contents of the XREF table. Note that the
given address is an offset from the start of the XREF table.
: init-xref \ --
Initialise XREF memory and information if not already set up.
: term-xref \ --
Free up XREF memory.
: save-xref \ -- ; save XREF memory to file
Save the cross reference memory to disc. Unless the
file name has been changed by XREF: <filename> the
file will be called XREF.XRF.
: load-xref \ -- ; reload XREF file from disc
Load the cross reference memory from disc. Unless the
file name has been changed by XREF: <filename> the
file used will be XREF.XRF.
: xref: \ "filename" -- ; enable XREFs
Use in the form XREF: <filename> to define the file that
SAVE-XREF and LOAD-XREF will use.
: xref-kb \ n --
Specifies the size of the cross reference memory in kilobytes.
By default this is 1024 kb, or 1Mb.
: +xrefs \ -- ; enable XREF
Initialises the cross reference system if it has not already
been initialised, and enables production of cross reference
information.
: -xrefs \ -- ; disable XREF
Stops production of cross reference information, which can
be restarted by +XREFS.
Cross reference memory is not erased or released. Thus,
restarting with +XREFS will retain information.
To release all previous information use TERM-XREF
before +XREFS.
: xref-report \ -- ; display XREF information
Displays some statistics about cross reference memory
usage.
: WalkXref \ xt1 xt2 -- ; XREF of XT1 using XT2 to display.
Used by application tools to walk the XREF chain for XT1.
The structure offset for each step in the chain is handled
by XT2 ( offset -- ). Because writing XT2 requires use of
the internal XREF structure, you must expose the XREFFER
module: EXPOSE-MODULE XREFFER to get access to the words
in Lib\XREF.FTH.
: (show) \ xt -- ; show/decompile words used by this XT
Given an XT, produces a machine decompilation of the word
using the cross reference information. If cross referencing
is not enabled, no action is taken.
: $show \ $addr --
Given a counted string, it is looked up as a Forth word name
and (SHOW) produces a machine decompilation of the word
using the cross reference information. If cross referencing
is not enabled, no action is taken.
: show \ -- ; SHOW <name>
The following name is looked up as a Forth word name
and (SHOW) produces a machine decompilation of the word
using the cross reference information. If cross referencing
is not enabled, no action is taken.
: hasXref? \ xt -- flag ; true if word has XREF info
produces TRUE if xt has XREF information
otherwise FALSE is returned.
: hasXDecomp? \ xt -- flag ; true if word has XREF decompilation info
produces TRUE if xt has XREF decompilation information
otherwise FALSE is returned.
: WalkDecomp \ xt1 xt2 -- ; DECOMP of XT1 using XT2 to display.
Used by application tools to walk the decompilation chain for XT1.
The structure offset for each step in the chain is handled
by XT2 ( offset -- ). Because writing XT2 requires use of
the internal XREF structure, you must expose the XREFFER
module: EXPOSE-MODULE XREFFER to get access to the words
in Lib\XREF.FTH.
: FindXrefInfo \ pc xt -- info | 0 ; finds xref packet corresponding to PC
Given the current PC and the XT of the word the PC is in,
FindXrefInfo returns a pointer to an XREF packet if the
PC is at an exact compilation boundary, otherwise it returns
zero.
: FindXrefNearest \ pc xt -- info|0
Given the current PC and the XT of the word the PC is in,
FindXrefNearest returns a pointer to the Xref packet
for the address at or less than the PC. If no Xref
information is available for the word, zero is returned.
: GetXrefPos \ info -- startpos len line addr
Given a pointer to an XREF packet, GetXrefPos returns the
position, name length, line number of the source text in the source file, and
the value of HERE at the time of compilation.
: NextXref \ info1 -- info2
Steps to the next info packet, given the offset of the previous.
: xref \ -- ; XREF <name>
Use in the form XREF <name> to display where <name> is used.
: uses \ -- ; synonym for XREF
A synonym for XREF above.
: xref-all \ -- ; cross reference all words
Produces a cross reference listing of all the words with
cross reference information. This information is often too
long to be directly useful, but can be pasted from the
console to an editor for sorting, printing, and other
post-processing.
: xref-unused \ -- ; cross reference all words
Produces a cross reference listing of all the unused words with
cross reference information. This information is often too
long to be directly useful, but can be pasted from the
console to an editor for sorting, printing, and other
post-processing.
This optional wordset found in /Lib/StringPk.fth contains the following definitions to aid in the manipulation of counted strings.
: $variable \ #chars "name" --
Create a string buffer with space reserved for #chars characters
: $constant \ "name" "text" --
Create a string constant called "name" and parse the the closing
quotes for the content.
: ($+) \ c-addr u $dest --
Add the string described by C-ADDR U to the counted string at
$DEST. This word is now in the kernel.
: $+ \ $addr1 $addr2 --
Add the counted string $ADDR1 to the counted buffer at $ADDR2.
This word is now in the kernel.
: $left \ $addr1 n $addr2 --
Add the leftmost N characters of the counted string at $ADDR1 to
the counted buffer at $ADDR2.
: $mid \ $addr1 s n $addr2 --
Add N characters starting at offset S from the counted string at
$ADDR1 to the counted buffer at $ADDR.
: $right \ $addr1 n $addr2 --
Add the rightmost N characters of the counted string at $ADDR1 to
the counted buffer at $ADDR2.
: $val \ $addr -- n1..nn n
Attempt to convert the counted string at $ADDR1 into a number. The
top-most return item indicates the number of CELLS used on stack to
store the return result. 0 Indicates the string was not a number, 1
for a single and 2 for a double. $VAL obeys the same rules as NUMBER?.
: $len \ $addr -- len
Return the length of a counted string. Actually performs C@ and is
the same as COUNT NIP.
: $clr \ $addr --
Clear the contents of a counted string. Actually sets its length
to zero. Primarily used to reset buffers declared with $VARIABLE.
: $upc \ $addr --
Convert the counted string at $ADDR to uppercase. This acts in
place.
: $compare \ $addr1 $addr2 -- -1/0/+1
Compare two counted strings. Performs the same action as the ANS
kernel definition COMPARE except that it uses counted strings as input
parameters.
: $< \ $1 $2 -- flag
A counted string equivalent to the numeric < operator. Uses
$COMPARE then generates a well - formed flag.
: $= \ $1 $2 -- flag
A counted string equivalent to the numeric = operator. Uses
$COMPARE then generates a well - formed flag.
: $> \ $1 $2 -- flag
A counted string equivalent to the numeric > operator. Uses
$COMPARE then generates a well - formed flag.
: $<> \ $1 $2 -- flag
A counted string equivalent to the numeric <> operator. Uses
$COMPARE then generates a well-formed flag.
: $instr \ $1 $2 -- false | index true
Look for an occurance of the counted string $2 within the string
$1. If found then the start offset within $1 is returned along with
a TRUE flag, otherwise FALSE is returned.
A CHAIN is an extensible version of the CASE..OF..ENDOF..ENDCASE mechanism. It is very similar to the SWITCH mechanism described in the Tools and Utilities chapter.
: case-chain \ -- addr ; -- addr MPE.0000
Begin initial definition of a chain
: item: \ addr n -- addr ; MPE.0000
Begin definition of a conditional code block
: end-chain \ addr -- MPE.0000
Flag the end of the current block of additions to a chain
: in-chain? \ n addr -- flag ; MPE.0000
Return TRUE if N is in the chain beginning at ADDR
: exec-chain? \ i*x n addr -- j*x true | n FALSE MPE.0000
Run through a given chain using TOS as a selector. If a match is
made execute the relevant code block and return TRUE otherwise
the initial selector and a FALSE flag is returned.
|
More items can be added later:
|
The data structures are as follows:
CASE-CHAIN <foo> generates a variable that points to the last item added to the list.
|
Binary overlays are pieces of the dictionary that have been compiled and saved with relocation information. They can be reloaded as needed and released on demand. Binary overlays are useful when you want to ship tools that are only needed during development, or if you have a large application whose memory footprint you want to reduce by only loading parts of the application when needed.
The binary overlay utility is not part of the kernel, but can be compiled from LIB\OVLVFX.FTH. As of build 3.40.0808, there has been major change in the way overlays are constructed. This change removes many restrictions that were present in earlier builds. To use the new overlay handler, all overlays must be rebuilt.
An overlay is generated by MAKEOVERLAY
|
the file <sourcename> is compiled twice. Relocation information is extracted and saved to the overlay along with the raw binary information. If any previously loaded overlays are needed by this overlay, their names are saved in the overlay and they will be automatically reloaded if necessary. After the overlay has been generated, the overlay code is removed. Overlays can be tested by compiling <sourcename> conventionally, and then finally generating the overlay when you are satisfied with it. MAKEOVERLAY preserves and links all vocabularies including SOURCEFILES. Overlay files are saved by MAKEOVERLAY in the current directory. The compiler imposes the following initial condition before the overlay file is compiled:
|
MAKEOVERLAY releases all previously loaded overlays. As a consequence, if the overlay to be compiled requires other overlays, you must load them explicitly by specifying them as dependencies before using MAKEOVERLAY. A dependency list is defined by the word [DEPENDENCIES followed by a list of overlay file names as required by LOADOVERLAY below. The list is termininated by DEPENDENCIES]. Use in the form:
|
This will cause MAKEOVERLAY to load the dependent overlays PRIMOVL.OVX and SECOVL.OVX and so on.
When an overlay is reloaded by LOADOVERLAY
|
the binary code and relocation information are loaded. If the overlay file references other overlays, these are loaded before the relocated binary code is installed. Overlay code is loaded into memory allocated from the Windows heap, and are linked in reverse load order, so that the last loaded is found first. The result of this is that the overlays are always loaded in dependency order, and releasing a "leaf" overlay will not affect the dependencies of other previously loaded overlays.
Although overlay files are saved by MAKEOVERLAY in the current directory, LOADOVERLAY will look first in the current directory and then in the directory from which the application was loaded. This allows all overlays and the main executable to reside in the same directory regardless of the current directory, but maintains convenience during development.
An overlay can be released by the use of RELEASEOVERLAY.
|
All loaded overlays can be released by RELEASEALLOVERLAYS
|
A word can be set to excute whenever the overlay is loaded from file or released. These words permit the overlay to allocate and free resources such as memory buffers.
|
Note that these settings should be in the overlay load file. The stack effect of <load-action> and <release-action> must be neutral, i.e. take nothing and return nothing [ -- ].
From VFX Forth v3.4 onwards, the naming conventions have been changed.
The binary overlay files have a ".OVX" extension. The word MAKEOVERLAY creates the overlay for you as follows:
|
If the source file name does not have an extension, the rules of INCLUDED will be followed, checking for files with extensions ".BLD" ".FTH" ".F" ".CTL" ".SEQ" in that order. If the destination file name does not have an extension ".OVX" will be used. If the destination file name is not provided, the source file name is used with a ".OVX" extension. Thus, just typing MAKEOVERLAY FOO will compile FOO.FTH to create FOO.OVX. The overlay name held by the system is the output file specification as given or created by MAKEOVERLAY, converted to upper case. This is important when reloading the overlay.
If no extension is provided for LOADOVERLAY, a ".OVX" extension will be added to the file name. Thus LOADOVERLAY FOO will check if an overlay called FOO.OVX has been loaded, and will load from file FOO.OVX. Similarly, LOADOVERLAY FOO.OVX will check if an overlay called FOO.OVX has been loaded, and will load from file FOO.OVX.
Each overlay contains VFX Forth information, and overlays cannot be loaded by a version of VFX Forth other than the one that built it. A user defined version string can be added to the version control information using SETOVLVER, which takes the address of a counted string. The format of the string is entirely user defined, the overlay handler simply checks the strings for identity.
Note that this version of LIB\OVLVFX.FTH requires VFX Forth build 3.40.0808 of 15 March 2002 or later.
The following system state is preserved and restored by the overlay handler.
|
If you generate other system-wide chains, these will NOT be preserved. To preserve them, modify the code in LIB\OVLVFX.FTH using the xxxIMPORTLINK words as a model. Future versions of this code may support a chain of chains model, but this will require that ALL such chains are anchored in the VFX Forth kernel/application before any overlays are either generated or reloaded.
N.B. If you modify this code, please pass it back to MPE so that it can be incorporated in later builds. This will reduce your maintenance work, our technical support load, and you will benefit from the work of others.
The overlay is produced by comparing two versions of the binary at different addresses, and generating relocation information from any differences. If a relocation value does not correspond to another overlay or the VFX Forth kernel, the build of the overlay will cause an error. Such errors can be caused by anything that inadvertently changes the data or code generation of the two versions being compared.
If data space in the dictionary is not initialised at compile time, it may contain random data. Compare:
|
The initial conditions of directives that affect code generation must be the same for each build. At least the following directives should be considered:
|
Similarly the starting codition of BASE should also be considered. The compiler imposes the following initial condition before the overlay file is compiled:
|
When compiling an overlay strict control of the initial search order is often necessary, especially because of redefinitions. We recommend that overlays are constructed from a build file which ensures that other required overlays are installed.
A sign of bad search order control is that the overlay can be correctly built with the source inliner turned off, but will not build with it on.
You cannot use file names with spaces, even though GETPATHSPEC is used to input the file names, because the file names are internally used as Forth word names.
There are occasions when a four-byte code sequence matches an address in another overlay, causing false relocation data to be generated. The result will be code that is corrupt after loading.
This situation has been drastically improved by the overhaul of 14 March 2002, but the warning has been left in until we are confident that all situations have been covered.
defer ovl-init-compile \ -- ; set initial state
A DEFERred word to set the initial compilation state for
both compilations of the overlay source code. The default
condition is:
|
Do not rely on this word being present in future releases. It is only present for experimental use with very large overlays.
: [dependencies \ -- ; set up dependency list
This word is used before MAKEOVERLAY below to define a list
of overlays required by the overlay to be made. It is followed
by a list of overlay file names as required by LOADOVERLAY below.
The list is termininated by DEPENDENCIES]. Use in the form:
|
: $MakeOverlay \ c-addr1 u1 c-addr2 u2 --
Use the first string as the source file name and the second string
as the overlay name. This word constructs a MAKEOVERLAY string and
EVALUATES it. $MAKEOVERLAY is provided for the construction of
higher level overlay management functions.
: MakeOverlay \ "src" ["dest"] -- ; MAKEOVERLAY <buildfile> <overlay>
Creates an overlay by loading an input file, which can itself load
other files, and producing an output file. If the source file name
does not have an extension, the rules of INCLUDED will be followed,
checking for files with extensions ".BLD" ".FTH" ".F" ".CTL" ".SEQ"
in that order. If the destination file name does not have an extension
".OVX" will be used. If the destination file name is not provided,
the source file name is used with a ".OVX" extension. Thus, just
typing MAKEOVERLAY FOO will compile FOO.FTH to create FOO.OVX.
The overlay name held by the system is the output specification
as given. This is important when reloading the overlay.
The compiler imposes the following initial condition before the
overlay file is compiled:
|
: SetOvlLoadHook \ xt -- ; ' <load-action> SETOVLOADHOOK
This word sets the action to be performed whenever the overlay
is loaded from the file. This action is NOT called by LOADOVERLAY
if the overlay is already loaded. SETOVLLOADHOOK must be included
in the overlay load file.
: SetOvlReleaseHook \ xt -- ; ' <release-action> SETOVLRELEASEHOOK
This word sets the action to be performed when the overlay
is released. SETOVLRELEASEHOOK must be included
in the overlay load file.
: SetOvlVer \ c-addr --
Sets the address of a counted string added to the version control
information. All overlay loads will be checked against this string.
SETOVLVER must be used before MAKEOVERLAY. The string can be reset
at any time by 0 SETOVLVER.
: $OvlLoaded? \ c-addr u -- start true | 0 0
Converts the string to upper case and tests whether or not
the overlay has been loaded, returning its start address
in memory and true if loaded, or two zeros if not loaded.
See MAKEOVERLAY for a discussion of overlay names.
: $LoadOverlay \ c-addr u -- start|ior end|-1
Uses the given string as an overlay name, and reloads the
the overlay if not already loaded. If the overlay name does
not have an extension, ".OVX" will be used. Any other required
overlays will be loaded before the requested overlay. The start
and end+1 address of the overlay code after installation are
returned. $LOADOVERLAY is provided for the construction of
higher level overlay mangement functions.
On error, the start and end values are replaced by ior and -1.
: LoadOverLay \ "name" -- ; LOADOVERLAY <name>
Load an overlay whose name follows in the input stream.
See $LOADVERLAY for more details.
: .overlays \ -- ; display loaded overlays
Shows the names of the the loaded overlays.
: lo \ "name" -- ; LO <name>
A synonym for LOADOVERLAY.
See $LOADVERLAY for more details.
: mo \ "src" ["dest"] -- ; MO <buildfile> <overlay>
A synonym for MAKEOVERLAY.
: $ReleaseOverlay \ c-addr u -- ior
Release the overlay of the given name, returning a non-zero code if
the overlay was not loaded. The name is converted to upper case before
the comparison is performed. $RELEASEOVERLAY is provided for the
construction of higher level overlay mangement functions.
If the overlay was loaded when OVL_IN_DICT was set FALSE (the default),
overlays loaded after the specified one will also be removed.
If the overlay was loaded when OVL_IN_DICT was set TRUE, the overlay
is in the 'kernel' area of the dictionary, and any code compiled or loaded after
the overlay will also be removed. Overlays dependent on this one will
be removed.
: ReleaseOverlay \ "text" -- ; RELEASEOVERLAY <name>
Uses $RELEASEOVERLAY to release the overlay whose name follows.
See $RELEASEOVERLAY for more details.
: ro \ "text" -- ; RO <name>
A synonym for RELEASEOVERLAY.
See $RELEASEOVERLAY for more details.
: ReleaseAllOverlays \ --
Releases and unhooks all overlays. Executed automatically by
the Exit chain.
: ovl_in_dict \ -- addr ; true to load overlays in dictionary ; SFP022
Set this variable to TRUE to load overlays at the end of the
dictionary, rather than in memory allocated from the heap.
This is only required in special circumstances.
After overlays have been built, restore OVL_IN_DICT
to FALSE.
Application configuration can be done in a number of ways, especially under Windows.
Registry |
A user nightmare to copy from one machine to another |
INI |
files Very slow for large configurations (before mpeparser.dll) |
binary |
Usually incompatible between versions |
database |
Big and often similar to binary |
Forth |
Already there, needs changes to interpreter. Independent of operating system. |
A solution to this problem is available in Lib/ConfigTools.fth. Before compiling the file, ensure that the file GenIO device from Lib/Genio/FILE.FTH has been compiled.
The Forth interpreter is already available, but we have to consider how to handle incompatibilities between configuration files and issue versions of applications. The two basic solutions are:
The abort on error solution is already available - it just requires the caller of included to provide some additional clean up code.
|
In VFX Forth, INTERPRET is used to process lines of input. INTERPRET is DEFERred and the default action is (INTERPRET). The maximum line size (including CR/LF) is FILETIBSZ, which is currently 512 bytes. If we restrict each configuration unit to one line of source code, we can protect the system by ignoring the line if an error occurs. We also have to introduce the convention in configuration files that actions are performed by the last word on the line (except for any parsing). This action has to be installed and removed, leading to the following code.
|
: CfgInterp \ --
A protected version of (INTERPRET) which discards any
line that causes an error.
: CfgIncluded \ caddr len --
A protected version of INCLUDED which discards any
line that causes an error, and carries on through the
source file.
: [SaveConfig \ caddr len -- struct|0
Starts saving a configuration file. Creates a configuration
file and allocates required resources, returning a structure
on success or zero on error. On success, the
returned struct contains the sid for the file at
the start of struct.
: SaveConfig] \ struct --
Ends saving a file device by closing the file, releasing
resources and restoring the previous output device.
: SaveConfig \ caddr len xt --
Save the configuration file, using xt to generate the
text using TYPE and friends. The word defined by
xt must have no stack effect.
We chose to support five type of configuration data:
All numeric output is done in hexadecimal to save space, and to avoid problems with BASE overrides. All words which generate configuration information must be used in colon definitions.
: \Emit \ char --
Output a printable character in its escaped form.
: \Type \ caddr len --
Output a printable string in its escaped form.
: .cfg$ \ caddr len --
Output a string in its escaped form, characters in the
escape table being converted to their escaped form. The
string is output as Forth source text, e.g.
s\" escaped text\n\n"
: .sint \ x --
Output x as a hex number with a leading '$' and a trailing
space, e.g.
$1234:ABCD
Single integers are saved by .SintVar and .SintVal.
' (SintVar) SimpleCfg: .SIntVar \ "<name>" --
Saves a single integer as a string. <name> must
be a Forth word that returns a valid address. Generates
$abcd <name> !
Use in the form:
.SIntVar MyVar
' (SintVal) SimpleCfg: .SIntVal \ "<name>" --
Saves a VALUE called <name>. Generates
$abcd to <name>
Use in the form:
.SIntVal MyVal
Double integers are saved by .DintVar.
' (DintVar) SimpleCfg: .DIntVar \ "<name>" --
Saves a double integer as a string. <name> must
be a Forth word that returns a valid address. Generates
$01234 $abcd <name> 2!
Use in the form:
.SIntVar MyVar
Counted strings are saved by .C$CFG.
' (c$cfg) SimpleCfg: .C$var \ "<name>" --
Saves a string <name> must
be a Forth word that returns a valid address. Generates
s\" <text>" <name> place
Use in the form:
.C$Var MyCstring
Zero terminated strings are saved by .Z$var.
' (z$cfg) SimpleCfg: .Z$var \ "<name>" --
Saves a zero terminated string at <name> which must
be a Forth word that returns a valid address. The output
consists of one or more lines of source code, following
lines being appended to the first.
s\" <text>" <name> zplace
s\" <more text>" <name> zAppend
...
Use in the form:
.Z$var MyZstring
Memory blocks are output by
.Mem <name> len
<Name> must be a Forth word that returns a valid address. Len must be a constant or a number. The output takes one of three forms, depending on len.
bmem <name> num $ab $cd ...
wmem <name> num $abcd $1234 ...
lmem <name> num $1234:5678 $90ab:cdef ...
A block of memory is output by
.Mem <name> len
<Name> must be a Forth word that returns a valid address. Len must be a constant or a number.
: BMEM \ "<name>" "len" --
Imports a memory block output in byte units by .Mem.
: WMEM \ "<name>" "len" --
Imports a memory block output in word (2 byte) units by .Mem.
: LMEM \ "<name>" "len" --
Imports a memory block output in cell (4 byte) units by .Mem.