Copyright (C) 2003-2005 by David J. Hardy.
All rights reserved.


=====================
MDAPI Developer Notes
=====================

David J. Hardy, February 2004

These notes are provided for developers and maintainers of the MDAPI
library.  The intention is to clarify details about the implementation,
to tell how and why the MDAPI does what it does.

The current MDAPI implementation is single-threaded.  This fact is most
apparent in the notes regarding mdsync.c - the synchronization routines
needed for nonblocking semantics presently have trivial implementations.

Please email document corrections and improvements to dhardy@ks.uiuc.edu.


Contents
--------

1.  Source Files
2.  External Dependencies
3.  Naming Conventions
4.  Type Definitions
4.1.  Predefined Types
4.1.1.  Primary Types
4.1.2.  Derived Types
4.2.  MDAPI Types
5.  Control Flow of Front End and Engine Code
6.  Subsystems
6.1.  Setup and Cleanup Routines (mdsetup.c)
6.2.  Type Creation and Management (mdtypes.c)
6.3.  Data Array Creation and Access (mddata.c)
6.4.  Setup and Execute Callbacks (mdcback.c)
6.4.1.  Overview of Callback Processing
6.4.2.  Registering Callbacks
6.4.3.  Processing Callbacks
6.5.  Routines for Running the Simulation (mdrun.c)
6.6.  Synchronization of Nonblocking Calls (mdsync.c)
6.7.  Error Reporting and Handling (mderror.c)


1.  Source Files
----------------

The MDAPI header files intended for inclusion by external code are:

  mdtypes.h - predefined MD type definitions, constants
  mdfront.h - front end API definitions and prototypes
  mdengine.h - engine API definitions and prototypes

The other header files are:

  mdcommon.h - definitions and constants common to both APIs
  mddef.h - internal definitions and prototypes

The header file inclusion graph is:

             mddef.h
             /     \
     mdfront.h     mdengine.h
             \     /
            mdcommon.h
                |
            mdtypes.h

This means that all of the source files include mddef.h in order to
see all prototypes and definitions.

The single-threaded version of MDAPI is separated into the following
"subsystems":

  mdsetup.c - setup and cleanup routines
  mdtypes.c - type creation and management
  mddata.c - data array creation and access
  mdcback.c - setup and execute callbacks
  mdrun.c - routines for running the simulation
  mdsync.c - synchronization of nonblocking calls
  mderror.c - error reporting and handling

The code in each of these files is generally divided into front end
API definitions, engine API definitions, and internal definitions.


2.  External Dependencies
-------------------------

The implementation depends on two MDX modules:  the ADT library
(e.g. libadt.so) and the debug.h header file.

The debug header file defines useful assertion macros - these perform
conditional tests when expanded (by defining DEBUG_SUPPORT to the
preprocessor) and otherwise vanish (leaving empty statements).  See
the debug.txt document for details.

The ADT (abstract data type) library defines two large data structures:
a resizable array (ADT_Array) and a hash table (ADT_Table).  The
ADT_Array class provides access to an array container which allocates
memory as indexed and provides an explicit resize function.  The
standard way of using the container is to append single elements to
the end of the array (or resize by taking away single elements).
In this case, the array allocates memory the "right way" by always
doubling the allocation when needing more space.  The memory allocation
shrinks by halving the allocation whenever the used array length drops
to one-quarter of the maximum space.  This means that appending and
removing a single element at the end of an array has O(1) amortized
cost.  The ADT_Table provides a hash table (i.e. associative array)
that stores an integer value to be retrieved by a string key.  The use
of ADT_Table by MDAPI is always to perform quick array indexing through
the key string, where the integer value ends up being the integer index
for the array.  The ADT_Array and ADT_Table data structures form the
fundamental building blocks for the MDAPI implementation.  See the
adt.txt document for more details about these classes.


3.  Naming Conventions
----------------------

The MDAPI externally visible symbols (function names with global
linkage, type tags, typedefs, enum constants, and macro names - note
that there are no global variables) are all prefixed with "MD_".
Function names after the prefix are lowercase, possibly containing
words separated by underscores (e.g. MD_readsub, MD_exec_callback).
Typedefs begin each word after the prefix with a capital letter with
words run together (e.g. MD_Interface, MD_AtomPrm).  Type tags are
named the same as the corresponding typedef with an added "_tag" suffix
(e.g. type "struct MD_Engdata_tag" is typedef "MD_Engdata"), and all
structs have a corresponding typedef.  (The only exception to this
type-naming convention are "int32" and "uint32" defined in mdtypes.h.)
All integer constants are declared as enums (but none of the enum
definitions are tagged).  The enum constants and macro names are all
upper case, with different words separated by underscores (e.g.
MD_INT32_MAX, MD_SIZEOF, MD_ERR_ENGINE).  (The only exception are
the four API calls that expand as macros - MD_init, MD_setup_engine,
MD_read, MD_write - and the version strings - MDAPI_VERSION, MDAPI_STRING
- defined in mdtypes.h.)

The MDAPI internal symbols (e.g. static functions and types defined
in mddef.h) drop the "MD_" prefix.  The other conventions remain.

The ADT library functions and types are all prefixed with "ADT_".
The debug.h file should be included only within .c source files.
The DEBUG_WATCH mode (set if you #define DEBUG_WATCH before including
debug.h) defines a static variable DEBUG_COUNTER which is incremented
by "variable watch" macros.  The main debug.h macros seen will be
ASSERT, COND, ERRCOND, and BUG.


4.  Type Definitions
--------------------

There are predefined types intended to be used for engine data during
the simulation.  All of these are defined in mdtypes.h.  There are also
MDAPI specific types which are needed by the APIs - some of these
structs are visible to the front end and/or the engine with directly
accessible members, and others should be considered opaque.  There are
a few MDAPI types visible only internally, defined in mddef.h.

4.1.  Predefined Types

Predefined types are for use in engine data arrays.  There are two
categories:  primary types and derived types.  Derived types are structs
comprised of members which themselves are primary or previously defined
derived types - these members might be singletons or fixed-length one-
dimensional arrays.  All of the predefined derived types have primary
members, however engine-defined types (all considered derived) are able
to use any previously defined type as a member.

Each of these "data" types has a type number.  Internally, the type
numbers are each of type int32 and have an encoded bitwise form:

  bit position:        31  -  19       |    18  -  16    |    15  -  0

  function:      TypeInfo array index  |  flip constant  |  size in bytes

TypeInfo is an MDAPI type used to store information about the defined
"data" type.  The "flip constant" indicates byte reordering information
needed for communicating this data type with a different endian machine.
Having this encoding constrains types to be no larger than 65535 bytes
(i.e. just under 64KB) and constrains the number of defined types to be
no more than 8192 - neither of these limits should cause a problem.
All of the predefined types have a corresponding enum constant giving
the type number.  The MD_SIZEOF() macro takes a type number and returns
the size in bytes (as an int32).

4.1.1.  Primary Types

The primary types are the fundamental building blocks for the more
advanced derived types.

  type name    size    type number    purpose
  -----------------------------------------------------------------
  char         1       MD_CHAR        use in array for string
  int32        4       MD_INT32       integer
  float        4       MD_FLOAT       floating point
  double       8       MD_DOUBLE      floating point
  MD_Fvec      12      MD_FVEC        3D vector of float
  MD_Dvec      24      MD_DVEC        3D vector of double
  MD_Name      8       MD_NAME        atom name (length <= 4 chars)
  MD_String    64      MD_STRING      shorter (keyword) strings
  MD_Message   512     MD_MESSAGE     longer (message) strings

The C-types char, float, and double are simply used as is (and encoded
as predefined types).  The int32 type is defined to ensure that the
integer type used is 4-bytes long across various architectures.

Even though MD_Fvec and MD_Dvec are defined as structs, they are
considered to be primary types - the byte flipping for these types
is considered as its own special case to improve efficiency.

The MD_String and MD_Message types date back to an earlier version of
the MDAPI - they are no longer as useful since engine-defined types
can now include a fixed-length array (see the section on type creation
and management below).  So these are candidates for removal.

mdtypes.h also defines uint32 as an unsigned 4-byte integer type, but
it is not made into a "predefined data type" - the uint32 is used to
store length information by the ADT_Array class.

4.1.2.  Derived Types

The derived types are created to store force field parameter (include
"Prm" in type name) and molecular topology information.

  type name    size    type number    purpose
  -----------------------------------------------------------------
  MD_AtomPrm   40      MD_ATOMPRM     params for atom type
  MD_BondPrm   32      MD_BONDPRM     params for bond type
  MD_AnglePrm  56      MD_ANGLEPRM    params for angle type
  MD_TorsPrm   56      MD_TORSPRM     params for torsion type
  MD_NbfixPrm  56      MD_NBFIXPRM    fix nonbonded params for pair
  MD_Atom      40      MD_ATOM        define each atom
  MD_Bond      12      MD_BOND        define each bond
  MD_Angle     16      MD_ANGLE       define each angle
  MD_Tors      20      MD_TORS        define each torsion
  MD_Excl       8      MD_EXCL        define each nonbonded exclusion

These types date back to the X-Plor molecular software package.  NAMD
(and X-Plor) distinguish types of torsion angles as being either dihedral
or improper - since both of these "types" are identical in NAMD, we can
and probably should keep separate data arrays named "dihedral" and
"improper", both of type MD_Tors (with the parameters stored as MD_TorsPrm).
The MD_Excl defines nonbonded exclusions beyond those already indicated by
the exclusion type:  1-2, 1-3, 1-4, or 1-4scaled; this means that MD_Excl
is rarely used.

4.2.  MDAPI Types

The table below presents the MDAPI types needed for implementing the
front end and engine APIs.  The "file" column below indicates the header
file in which the type is defined.  The "use" column indicates whether
the members of that particular structure are to be accessed by either the
front end or engine, where "accessible" indicates that access is intended,
"depends" indicates that only some members are to be accessed, "opaque"
means that there should be no access of members, and "invisible" means
that the type is not seen externally.

  type name     file        use         purpose
  -----------------------------------------------------------------------
  MD_Attrib     mdcommon.h  accessible  data array attributes
  MD_Cbdata     mdcommon.h  depends     data "arguments" to callback
  MD_Member     mdcommon.h  accessible  describes defined-type member
  MD_Error      mdcommon.h  opaque      describes error condition
  MD_Interface  mdcommon.h  opaque      "MDAPI-object" defining its state
  MD_Engine     mdfront.h   opaque      renamed MD_Interface
  MD_Front      mdengine.h  opaque      renamed MD_Interface
  MD_Engdata    mdengine.h  depends     describes an engine data array
  MD_Callback   mdengine.h  depends     describes a callback
  TypeInfo      mddef.h     invisible   info about a defined-type
  MsgcbInfo     mddef.h     invisible   info about a message callback
  TypeInit      mdtypes.c   invisible   initialize predefined data types

The MD_Interface struct is the main "MDAPI-object", defining the state
of the MDAPI.  It is renamed to "MD_Engine" in mdfront.h, so that the
front end views it as the handle to an engine; similarly, it is renamed
"MD_Front" in mdengine.h, so that the engine views it as the handle to
the controlling front end.  The MD_Callback type is used to describe both
standard and force callbacks.  Since message callbacks entail much simpler
bookkeeping than the others, they are given as a different type (that
happens to be internally defined).  The members of these types are
presented later in the subsystem to which they relate.

Any arrays passed through the API calls by either front end or engine
are only shallow copied by MDAPI, but are expected to persist unless
otherwise specified.  These persistent array items passed by the front
end include the engine name string (which might indicate the name of a
module to be dynamically loaded) and the arrays of MD_Cbdata used to
configure data arguments to a callback.  For the engine, the persistent
array items include name strings for data arrays, message strings for
error definitions, arrays of MD_Member used to define new types (along
with the member name strings referenced within each MD_Member element),
and the type name strings.


5.  Control Flow of Front End and Engine Code
---------------------------------------------

This is a condensed guide discussing the control flow of the front end
and engine codes and their interaction through the MDAPI layer.  Although
this repeats concepts found in the the MDAPI specification document, it
is worth a brief discussion here.

The front end declares a variable of type MD_Engine (really MD_Interface),
either on its stack or allocates its space on the heap.  In any case,
this memory must persist through the entire use of MDAPI, from the initial
MD_init() call through the completion of MD_done().  A pointer to this
variable is passed in all front end API calls.  A pointer to this variable
is also passed to the engine provided routines and used by the engine
(this time as MD_Front *) for engine API calls.

The "dialogue" presented below shows the control flow.

  front end:
    declares or allocates an MD_Engine (MD_Interface struct)
    invoke MD_init()

  MDAPI MD_init():
    setup each subsystem by calling MD_init_error(), MD_init_types(),
      MD_init_data(), MD_init_callback() (all are internal routines)
    invoke engine_init()

  engine engine_init():
    setup and further configuration of MD_Front (MD_Interface struct)
    invoke MD_setup_engine()
    possible calls to MD_engdata(), MD_engdata_buffer(),
      MD_engdata_manage(), MD_engdata_alias(), MD_new_type(),
      MD_new_error()
    invoke MD_setup_run()
    return

  MDAPI MD_init():
    check error status
    return

  front end:
    initialize simulation data arrays, involves zero or more calls to
      MD_namelist(), MD_idnum(), MD_name(), MD_attrib(),
      MD_setlen(), MD_setmax(), MD_resize(), MD_read(), MD_readsub(),
      MD_write(), MD_writesub(), MD_share(), MD_unshare(), MD_direct(),
      MD_setmod(), MD_firststep()
    possibly setup callbacks using MD_callback(), MD_fcallback(),
      MD_msgcallback()
    might need to examine engine defined types using MD_type_namelist(),
      MD_type, MD_type_name(), MD_type_memberlist(), MD_type_member()
    error handling done using MD_errnum(), MD_errmsg(), MD_reset()
    run simulation with MD_run()

  MDAPI MD_run():
    set some status flags
    invoke run() (routine provided by engine)

  engine run():
    invoke MD_engine_data() to obtain engine data handle
    perform simulation
    must invoke MD_incrstep() when appropriate
    must process callbacks according to API specification, involves
      calls to MD_ready_callback(), MD_exec_callback(),
      MD_ready_fcallback(), MD_exec_fcallback()
    any messages can be sent via calls to MD_ready_msgcallback(),
      MD_exec_msgcallback()
    might also involve further manipulation of engine data arrays
      through calls to MD_engdata_setlen(), MD_engdata_setmax(),
      MD_engdata_resize(), MD_engdata_ackmod(), MD_engdata_cbsetup()
    callback data argument discovery can be performed through calls to
      MD_callback_list(), MD_fcallback_list()
    error status can be set using MD_error()
    return after finishing simulation steps

  MDAPI MD_run():
    reset some status flags
    return

  front end:
    perform post-analysis or save state if needed, possibly through
      further data array manipulation calls
    invoke MD_done() when finished with simulation

  MDAPI MD_done():
    invoke engine_done()

  engine engine_done():
    cleanup memory allocations
    possible call to MD_free_data()
    return

  MDAPI MD_done():
    cleanup each subsystem by calling MD_done_callback(), MD_done_data(),
      MD_done_types(), MD_done_error()

Not included above are the synchronization routines:

  for front end:
    MD_test(), MD_wait()
      - used to synchronize MD_init(), MD_read(), MD_readsub(),
        MD_direct(), MD_update(), MD_run()

  for engine:
    MD_test_callback(), MD_wait_callback()
      - used for MD_exec_callback()

    MD_test_fcallback(), MD_wait_fcallback()
      - used to synchronize MD_exec_fcallback()

    MD_test_msgcallback(), MD_wait_msgcallback()
      - used to synchronize MD_exec_msgcallback()

The front end is permitted to only have one of these nonblocking
routines in flight at a time.  The engine has separate synchronization
contorls for the three kinds of callbacks, because each type of callback
processing is allowed to be active.  In the single threaded MDAPI
implementation, all routines run to completion before returning, so the
synchronization routines have trivial implementations.  In multi-threaded
MDAPI, there could be as many as five active threads going at one time:
one for the front end, one for the engine run() routine, and one for each
of the three callback processing routines.

The rationale for allowing overlap between standard and force callbacks
is to enable better overlap of computation of force and integration with
communication back to the front end.  The caveat is that the engine should
not permit CBWRITE and/or CBSHARE access on a data array buffer in
combination with FCBWRITE and/or FCBSHARE access on the same buffer, since
consistent writes to that buffer cannot be guaranteed.  However, the MDAPI
does not prevent an engine from assigning these access rights together.

A high performance engine should synchronize right before it needs to
update shared buffers.  Pseudocode for an engine run() routine using
leapfrog integration is presented below.

  engine data arrays (i.e. established via MDAPI)
    pos        access:  READ, WRITE, CBREAD    # position array
    vel        access:  READ, WRITE, CBREAD    # velocity array
    pos_int    access:  FCBREAD                # internal position array
    f_ext      access:  FCBWRITE               # external force array

  internal engine arrays
    vel_int    # internal velocity array
    f          # force array

  run(int32 numsteps) {
    # see if callbacks need to be processed for step 0
    MD_exec_callback()     # read pos, vel

    # compute initial force
    pos_int := pos
    MD_exec_fcallback()    # read pos_int, write f_ext
    compute force          # read pos_int, write f
    MD_wait_fcallback()    # finish using pos_int, f_ext
    f := f + f_ext

    # leapfrog integration for numsteps
    while (numsteps-- > 0) {
      vel_int := vel + dt/2 * f/m      # half kick
      pos_int := pos + dt * vel_int    # drift
      MD_exec_fcallback()              # read pos_int, write f_ext
      compute force                    # read pos_int, write f
      MD_wait_fcallback()              # finish using pos_int, f_ext
      f := f + f_ext
      MD_wait_callback()               # finish using pos, vel
      MD_incrstep()                    # increment MDAPI step counter
      vel := vel_int + dt/2 * f/m      # half kick
      pos := pos_int
      MD_exec_callback()               # read pos, vel
    }
    MD_wait_callback()     # finish using pos, vel
  }

The pos_int and vel_int are internal position and velocity arrays,
respectively, allowing undisturbed standard callback access to pos and
vel arrays during force computation and integration.  The "compute force"
step above contains most of the computational work involved in each step.
See the later section on the data array subsystem for further information
on the meaning of data array access permissions.


6.  Subsystems
--------------

The following sections below discuss implementation specific details
regarding the subsystems of the MDAPI.  This is not intended to be a
comprehensive description of all API calls, rather it discusses the
how and why concepts behind the implementation.

6.1.  Setup and Cleanup Routines (mdsetup.c)

In the discussion below as well as in the source code, the MD_Interface
pointer in the implementation of MDAPI routines is always called "s".
This dates back to the earliest versions of MDAPI in which the main
object was type MD_Sim, with the pointer named "sim" or "s" for short.
The MD_Interface type name is used in defining internal routines, and
the type is renamed MD_Engine for front end API definitions and MD_Front
for engine API definitions.

The setup subsystem directly initializes the following members of
MD_Interface:

  MD_Interface {
    ...
    void *engine;
    const char *engname;
    int (*engine_init)(struct MD_Interface_tag *, int32 flags);
    void (*engine_done)(struct MD_Interface_tag *);
    ...
  };

The "engname" string identifies the engine (in some sense).  The
"engine_init" and "engine_done" routines serve as the constructor and
destructor for the engine.  All three of these are initialized when the
front end calls MD_init().  The "engine" void pointer is the handle to
the engine state, initialized when the engine calls MD_setup_engine().

The front end calls MD_init() to setup MD_Interface and intialize the
engine.  However, MD_init() is really implemented as a macro that
expands into a call to MD_init_version() that passes the MDAPI_VERSION
string in the call.  The first task is to compare the argument
mdapi_version with the MDAPI_VERSION - this validates that the front
end has included the correct version of the MDAPI header files as was
used when compiling libmdapi.so.  (For now, strcmp() is used to check
that the strings are identical, but in the future this test might be
relaxed to allow different minor revision numbers.)  If the test fails,
then MD_error_version() is called (see the error handling subsystem
for details).  After validating the version number, the setup routines
are called for each subsystem:  MD_init_error(), MD_init_types(),
MD_init_data(), MD_init_callback().  After this, the engname string
is stored (shallow copied) and the engine_init() routine is invoked.
Note in the case that engine_init() fails, the MD_ERR_ENGINE error is
set (which is a fatal error).  In a later implementation, the engname
might be somehow interpreted as an engine plugin (dynamically loaded
module - using dlopen() on most architectures).

The MD_done() routine is called by the front end to conclude the use
of the engine and cleanup the MD_Interface object.  Note the early
exit if errnum == MD_ERR_VERSION.  Otherwise, the cleanup reverses
the initialization process.  First engine_done() is invoked, then
MD_done_callback(), MD_done_data(), MD_done_types(), and finally
MD_done_error().

The MD_setup_engine() routine is also implemented as a macro that
expands into a call to MD_setup_engine_version().  The purpose here
is to again validate the version number, this time from the engine
side, and if successful store the engine state using a void pointer.
This pointer is the engine handle to its data, retrieved through the
MD_engine_data() routine.

6.2.  Type Creation and Management (mdtypes.c)

The MD_Interface members used for type handling are:

  MD_Interface {
    ...
    ADT_Table typenum;
    ADT_Array typename;
    ADT_Array typeinfo;
    ...
  };

The "typenum" table is a mapping of type names to their corresponding
numbers.  Every "defined" type has a name and a number (see the "Type
Definitions" chapter regarding predefined types).  The string name for a
type is intended to be identical to the name used in the source code to
declare a variable of that type, where typedef has been used to rename
structs to a one-word name (e.g. "struct MD_Dvec_tag" is renamed "MD_Dvec"
which is also the string name for this type).  Using a number to
represent a type provides a more concise way to handle types, providing
a simple way to store a type (MD_Attrib contains an int32 "type" member),
easy comparison to check if two types are the same, and a quick method
to discover the size of a given type (using the MD_SIZEOF macro).  The
table container is used by front end MD_type() to quickly obtain the
type number for a given type name.

The "typename" array is a list of all type names.  This array of strings
is returned by front end MD_type_namelist() for the list of all defined
data types, including both the predefined types and the engine-defined
types.

The "typeinfo" array is a list of the internal TypeInfo type that stores
the information about each defined type.  There are three main reasons
for having a detailed encoding for defined data types.  The first reason
is to allow a simple way to get the type size - this permits the generic
implementation of data array manipulation routines to read, write, and
resize the buffer.  The second reason is to enable the interface to be
extended beyond the predefined data types - the engine has an elegant
mechanism for defining its own types, and the front end has routines for
discovering information about these types, such as the type name, the
names of its members, the member types, and the byte address of some
member within an object of the given type.  The third reason is to
facilitate the communication of data between a remote front
end and engine - byte flipping information is stored for each type to
efficiently support communication between big-endian and little-endian
architectures.

The TypeInfo struct defined in mddef.h is:

  TypeInfo {
    int32 typenum;
    const char *typename;
    const MD_Member *memberlist;
    int32 memberlistlen;
    ADT_Table *member;
    int32 *byteoffset;
    int32 *flipoffset;
    int32 flipoffsetlen;
  };

with the MD_Member type defined in mdcommon.h as:

  MD_Member {
    int32 type;
    int32 len;
    const char *name;
  };

Imagine a "defined data type" to be almost comparable to a C-struct.
The type itself has a name and will cluster together one or more members
that are of previously defined types.  The type name is recorded in
"typename", and, upon the definition of this type, the type number is
assigned and stored as "typenum".  The "memberlist" is an array of the
members.  Looking at the definition of MD_Member shows that each member
has a "type" (number) and a "name" - in this case the name of the member
field, not the name of the type.  The "len" argument takes into account
that for a C-struct, a member could be a fixed length array, where
explicit support is provided only for one-dimensional arrays.  Looking
again at TypeInfo, "memberlistlen" is the length of the member array.
The "flipoffset" array of length "flipoffsetlen" encodes byte reordering
information to support the communication of engine data arrays between
big-endian and little-endian architectures.  The "member" table pointer
is initially NULL.  This table is filled out when needed, mapping the
member names to the byte address offsets (from 0) of the location of
that member within an instance of that type.  The "byteoffset" array
is also initially NULL, to be allocated and filled when needed.  This
array parallels the "memberlist" array (also containing "memberlistlen"
elements) and provides the byte offset for each member, counting from 0.

The MDAPI performs a TypeInfo encoding, not just for engine-defined
types, but also for all of the "predefined data types" from mdtypes.h.
The TypeInit type with the static constant TypeInitList array provides
the initialization data used by MD_init_types() to encode all of the
predefined data types using the MD_new_type() routine, the same routine
that the engine uses to define new types.  The "primary types" are
treated by the MD_new_type() routine as a special case, since they do
not depend on previously defined types.  After the encoding of the first
MD_NUMBER_PRIMARY_TYPES (from mdtypes.h) number of types, all subsequent
types are treated as "derived types" meaning that they all contain one
or more members of some previously defined type.  (Although predefined
derived types contain only primary types as members, a member can be of
any previously defined type.)  As a validity check, the type number
constants given in mdtypes.h (MD_CHAR, MD_INT32, etc.) for predefined
types are tested against those type numbers assigned by MD_new_type() to
make sure that they match.

There are three important conditions that should be satisfied by the
members of a new type (besides the obvious one that the members are
all of previously defined types).  First, each member should be named
with a nonempty (and nil-terminated) string.  The only exception is
for the primary types, in which case there is exactly one member
with name "" (the empty string).  Second, the members must be properly
word-aligned on 4-byte and 8-byte word boundaries.  This means that a
member composed of some 4-byte numeric type (e.g. int32, float, and
MD_Fvec) must be aligned on a 4-byte word boundary.  Similarly for
8-byte numeric types (e.g. double and MD_Dvec).  Third, the length
of the type must be divisible by the largest word alignment that is
used within the type.  For example, MD_Atom has an "int32 notused"
padding member so that the length is divisible by 8 since it also
contains members of type double.  Note that MD_Fvec, MD_Bond, and
MD_Tors have length divisible only by 4, but this is all that is
needed since they are all based on 4-byte numeric types.

The "typenum" (repeated from the "Type Definition" chapter) assigned
to an encoded type is an int32 bit-field:

  bit position:        31  -  19       |    18  -  16    |    15  -  0

  function:      TypeInfo array index  |  flip constant  |  size in bytes

The "flipoffset" array elements have a similar looking bit-field:

  bit position:        31  -  19       |    18  -  16    |    15  -  0

  function:            (unused)        |  flip constant  |   byte offset

where each byte offset indicates where the next flip is to occur, and
the flip constant tells what kind of flip to perform.  The flip constants
are declared as enum constants in mdtypes.h:

  enum {
    MD_FLIP_NONE    = 0,
    MD_FLIP_4BYTE   = 1 << MD_SHIFT_FLIP,
    MD_FLIP_8BYTE   = 2 << MD_SHIFT_FLIP,
    MD_FLIP_12BYTE  = 3 << MD_SHIFT_FLIP,
    MD_FLIP_24BYTE  = 4 << MD_SHIFT_FLIP,
    MD_FLIP_DERIVED = 5 << MD_SHIFT_FLIP,
    MD_FLIP_MASK    = 7 << MD_SHIFT_FLIP
  };

where MD_SHIFT_FLIP = 16 (obviously).  The _12BYTE and _24BYTE constants
are for MD_Fvec and MD_Dvec types, respectively (recall that they are
considered to be primary types).  Rather than having MD_FLIP_DERIVED
appear in the "flipoffset" array, any derived types have been recursively
expanded in terms of its primary type components, so the "flipoffset"
array might be longer than the "memberlist" array.

The most complicated routine in mdtypes.c is MD_new_type().  This routine
takes the name of the prospective type, the member list, and the number of
bytes (using "sizeof").  The routine uses this information to determine
the TypeInfo encoding for the new type and append it to the MD_Interface
"typeinfo".  During the TypeInfo encoding, the validity of the new type is
checked.  Also, the type name is appended to the MD_Interface "typename"
array, and the type number is stored in the MD_Interface "typenum" table
to be accessed through the type name.  The return value of MD_new_type()
is the new type number assigned.

The first MD_NUMBER_PRIMARY_TYPES number of types are treated as special
cases to create the predefined primary types.  Additional types are
treated as derived types.  In order to validate the correctness of the
new type while encoding it, two internal data structures are employed:
a stack and a queue (both are of ADT_Array).  First, the member types are
pushed backwards onto the stack.  Then while types remain on the stack,
the next type is popped off.  If the type involves no flipping, then its
length is accumulated as the offset to the next flip byte position.  If
the type is a primary type that involves flipping, then the difference
between the current flip offset and the previous flip offset is ORed with
the flip type and then enqueued.  If the type is a derived type involving
flipping, then the members of this type are pushed backwards onto the
stack.  Before each flip offset enqueuing, the required word alignment is
validated.  At the end of the loop, the total byte count is checked
against the required word alignment, and the flip offset queue becomes
the TypeInfo "flipoffset" array.

The internal method MD_flipdata() is used to reverse the byte ordering on
data arrays of some defined type, necessary when communicating the array
between big-endian and little-endian machines.  MD_flipdata() accepts a
pointer to the data buffer along with its type number and performs a byte
reordering in place.  After looking up the TypeInfo encoding in the
MD_Interface "typeinfo" array, flipping is applied to each element.  An
array of some primary numeric type requires a single-nested loop over
each element.  If the indicated flipping is instead MD_FLIP_DERIVED, then
a double-nested loop first over each element and then over "flipoffset"
array is required.

The front end method MD_type_member() provides quick access to a member
field based on the name of the member, returning a pointer to that
member field when given a pointer of an instance of that type.  A copy
of the corresponding MD_Member array element can also be returned to
the front end.  The first invocation of this for a particular type will
create the TypeInfo "member" table, associating each member name with the
corresponding index into the "memberlist" array.  Also created is the
TypeInfo "byteoffset" array, which parallels the "memberlist" array,
containing the byte offset from 0 for each member field in the type.
Subsequent invocations take much less time, performing a simple hash
table lookup in the "member" table for the requested information.  The
created "member" table and "byteoffset" arrays persist until MD_done()
is called on MD_Interface object, with memory freed by MD_done_types().

6.3.  Data Array Creation and Access (mddata.c)

Most of the API routines deal with the transfer of simulation data
between front end and engine.  The engine is reponsible for establishing
each data array of some defined type (see preceding subsystem), and the
front end is generally responsible for providing initial information to
the arrays and, after the simulation, obtaining results.  The MDAPI
supports a number of alternative mechanisms for creating and accessing
the array buffer space, which accounts for the number of routines.

There are a few key concepts that should be understood before presenting
the details.  In order to provide a front end with an extensible interface
to some engine, data arrays are given string names for their access.
(There is an agreed upon naming convention for basic molecular dynamics
functionality, with everything beyond considered to be an extension.)  
The string names are awkward for continued access, so these are for the
front end immediately replaced by identification numbers (int32) that
provide a handle to refer to the data array, similar to the type number
described earlier.  The engine receives a pointer to each data array
(pointer to MD_Engdata).  The memory management for the array buffer is
generally performed by the interface layer, although there is support for
statically allocated array buffers by the engine and buffer space (either
dynamically or statically allocated) shared by the front end.  The engine
establishes particular access permissions on each individual array that
affect how the array may be used by the front end.  The interface layer
sets internal status flags for each individual array that are transparent
to the front end but sometimes useful to the engine.

The MD_Interface members for data handling are:

  MD_Interface {
    ...
    ADT_Table idnum;
    ADT_Array name;
    ADT_Array pengdata;
    ...
  };

The "idnum" table is a mapping of engine data array names to their
corresponding ID numbers.  The MD_idnum() routine performs a lookup in
the table to return the ID number for the given name string.

The "name" array is a list of all of the engine data array names.  This
list is explicitly provided to the front end through the MD_namelist()
routine.

The "pengdata" array is an array of pointers, each to a unique MD_Engdata
object.  The engine itself also receives each of these pointers as it
creates the engine data arrays.  Note that keeping a list of MD_Engdata
would invalidate the pointers as the list grows and the memory is resized.
The MD_Engdata members are:

  MD_Engdata {
    void *buf;
    void *(*reallocmem)(void *ptr, size_t size);
    const char *name;
    MD_Attrib attrib;
    int32 idnum;
    ADT_Array cback;
  };

The "buf" field points to the array buffer.  The "reallocmem" contains
a pointer to the memory allocation routines which must have the same
semantics as realloc() - by default this pointer actually points to
realloc().  "name" is the data array name.  "idnum" is the ID number
assigned to the data array.  "cback" is an array of pointers to the
callbacks that receive this data array (discussed in greater detail in
the following section).

The "attrib" member of type MD_Attrib contains the array attributes.
The internals of MD_Attrib are:

  MD_Attrib {
    int32 type;
    int32 len;
    int32 max;
    int32 access;
  };

These, respectively, store the type number of the data array, the length
of the array (i.e. how many elements of the array are in use with well-
defined values), the maximum number of elements allocated to "buf",
and the access permissions (and status flags) on the array.  "access"
is a bit field - presently the bit positions are:

  bit position:        31  -  16        |    15  -  0

  function:      internal status flags  |  access flags

The access flags are defined in mdcommon.h as:

  enum {
    MD_READ     = 0x00001,  // read access
    MD_WRITE    = 0x00002,  // write access
    MD_CBREAD   = 0x00004,  // read during callback
    MD_CBWRITE  = 0x00008,  // write during callback
    MD_CBSHARE  = 0x00010,  // shared data from front end during callback
    MD_FCBREAD  = 0x00020,  // read during force callback
    MD_FCBWRITE = 0x00040,  // write during force callback
    MD_FCBSHARE = 0x00080,  // shared data from front end during force cb
    MD_SETLEN   = 0x00100,  // front end allowed to set length
    MD_SETMAX   = 0x00200,  // front end allowed to set maximum allocation
    MD_RESIZE   = MD_SETLEN | MD_SETMAX,
    MD_ESETLEN  = 0x00400,  // engine allowed to set length
    MD_ESETMAX  = 0x00800,  // engine allowed to set maximum allocation
    MD_ERESIZE  = MD_ESETLEN | MD_ESETMAX,
    MD_DIRECT   = 0x01000,  // direct access
    MD_NOSHARE  = 0x02000,  // do not permit front end shared buffer
    MD_NOTIFY   = 0x04000,  // notify engine of front end updates
  };

The internal status flags are defined in mdcommon.h as:

  enum {
    MD_SHARE    = 0x10000,
    MD_UNSHARE  = 0x20000,
    MD_MODIFY   = 0x40000
  };

where MD_SHARE and MD_UNSHARE specify whether or not the front end
is permitted to supply (or take back) a shared buffer, and MD_MODIFY
specifies (to the engine) whether a buffer has in some way been modified
by the front end (either by resizing or writing to it).  It is likely
that more status flags will be added to support later implementations,
for instance, remote communication between front end and engine.

The only MD_Engdata members that should normally be accessed directly by
the engine are "buf" to directly read and write the array buffer,
"attrib.len" to obtain the number of elements in the array (when set by
the front end), and "attrib.access" to check whether or not MD_MODIFY
flag is set.  Engine resizing of an MD_Engdata buffer should be done
through MD_engdata_setlen(), MD_engdata_setmax(), and MD_engdata_resize()
routines which will update the internal flags accordingly.  The
MD_engdata_ackmod() routine clears the MD_MODIFY flag, acknowledging
to the interface layer that the engine has updated its own state with
respect to the data array modification made by the front end.

Engine data arrays are initially created during the engine_init()
routine.  MD_engdata() is the typical routine for creating a resizable
engine data array - it establishes a NULL "buf" to be resized using
realloc().  MD_engdata_buffer() is used for supplying a fixed-size
buffer (either statically or dynamically allocated) - "buf" is set
to point to the buffer space passed to the routine, "reallocmem" is
set to NULL, and MD_RESIZE and MD_ERESIZE access is explicitly cleared.
MD_engdata_manage() is used to establish the use of an alternative
memory management routine.  All of these routines call the internal
routine engdata_alloc() which validates array attributes, enters the
array name and ID number into the MD_Interface "idnum" table, appends
the array name to the MD_Interface "name" list, allocates a new
MD_Engdata object, appends the MD_Engdata pointer to the MD_Interface
"pengdata" array, and returns the MD_Engdata pointer.  The engine API
routine MD_engdata_alias() can be used to add an additional array name
and ID number into the MD_Interface "idnum" table and append the name
to the MD_Interface "name" list - this creates an alias name for the
array without modifying the MD_Engdata object.  The value assigned
to the ID number for a new array is simply the index of that MD_Engdata
pointer appended to "pengdata".

Rather than allow the engine to specify an MD_SHARE flag to permit the
front end to share buffer space for an array using MD_share(), this is
instead set as an internal status flag for the array.  The MD_SHARE
status is set only for MD_engdata() arrays - "buf" must be NULL and
"reallocmem" must be equal to realloc() - also, the access must permit
MD_RESIZE and must not have MD_NOSHARE or MD_ERESIZE set.  Note that this
status will later be cleared by the interface if the array is resized so
that "buf" is no longer NULL, and will again be reinstated if the buffer
is resized to have 0 max allocation.  A front end call to MD_share()
succeeds only if MD_SHARE is set - this call results in setting the buffer
to have a fixed max size, clearing MD_SHARE and MD_SETMAX, setting
MD_UNSHARE, and assigning "reallocmem" to NULL.  The MD_unshare() call
undoes this process and yields the buffer back to the front end.

During the engine_done() cleanup, the engine is responsible for freeing
any buffer allocations performed for MD_engdata_buffer().  If the engine
state is storing information for some "reallocmem" routine established
by MD_engdata_manage(), then the engine should call MD_free_data() in
order to cleanup buffer allocations before the engine frees itself.

6.4.  Setup and Execute Callbacks (mdcback.c)

6.4.1.  Overview of Callback Processing

Callbacks into the front end are invoked during the simulation started
through MD_run().  There are three types of callbacks.  The simplest type
is a message callback which receives an informational text string from
the engine.  Standard callbacks and force callbacks are able to receive
subarrays of any engine data arrays that permit that type of callback.
The difference between standard and force callbacks is when they are
invoked.  Standard callbacks are meant to be invoked at the completion
of some step.  Furthermore, the frequency can be set as to how often the
callback should be invoked, measured in number of steps.  The purpose of
standard callbacks is to communicate quantities such as trajectory,
energy, and temperature during a running simulation.  Unlike standard
callbacks, force callbacks are invoked every time the force is evaluated.
Their processing occurs at least once every step, and, depending on the
integration method, might be more than once every step.  The purpose of
force callbacks is to provide external forces to the simulation, for
instance, in an interactive simulation.

The MD_Interface members for handling callbacks are:

  MD_Interface {
    ...
    ADT_Array pcbq;
    ADT_Array pfcb;
    ADT_Array msgcb;
    ...
  };

where "pcbq" and "pfcb" are arrays of pointers to MD_Callback and "msgcb"
is an array of (internal type) MsgcbInfo.

The MsgcbInfo type is defined in mddef.h as:

  MsgcbInfo {
    int32 (*msgcb)(void *, const char *msg, int32 stepnum);
    void *info;
  };

where "msgcb" points to the message callback routine and "info" has been
provided by the front end to deliver state information to the routine.
Anytime the engine calls MD_exec_msgcallback() to process the message
callbacks, each of the message callback routines stored in MD_Interface
"msgcb" array are invoked in order.  It probably serves no real practical
purpose to permit registration for multiple message callbacks, but since
it is so easily implemented, it is allowed for consistency with the other
types of callbacks.

The MD_Callback type defined in mdengine.h is more complicated:

  MD_Callback {
    union {
      int32 (*cb)(void *, MD_Cbdata *c, int32 clen, int32 stepnum);
      int32 (*fcb)(void *, MD_Cbdata *c, int32 clen, int32 step, double ds);
    } funcptr;
    void *info;
    MD_Cbdata *cbarg;
    int32 cbarglen;
  // needed only for standard callbacks
    int32 stepincr;
    int32 nextstepnum;
    int32 cbinfonum;
  // status flags
    int32 flags;
  };

The "funcptr" union is used to store and invoke either a standard
callback "funcptr.cb()" or a force callback "funcptr.fcb()".  The "info"
is provided by the front end to deliver state information to the routine.
The engine data subarrays are accessed through the MD_Cbdata array, stored
in MD_Callback as "cbarg" with length "cbarglen".  The next three members
"stepincr", "nextstepnum", and "cbinfonum" are used only for standard
callbacks, where "stepincr" stores how often the callback routine is to be
called, measured in number of steps.  The "flags" provide status
information using a bit field with enum constants defined as:

  enum {
    MD_CALLBACK_INIT  = 0x001,
    MD_CALLBACK_SETUP = 0x002,
    MD_CALLBACK_SHARE = 0x004
  };

The MD_Interface member "pcbq" is used to store standard callbacks within
a priority queue, implemented as a binary heap, in order to provide
immediate access to the next callback to be processed, which is always
stored at the head node of the heap (i.e. the first element of the array).
The heap ordering of MD_Callback pointers is based on "nextstepnum" (the
value of the step number when the callback should again be invoked) and
"cbinfonum" (a unique value assigned to each callback to specify the order
in which the callbacks are registered by the front end).  After a standard
callback is invoked, "nextstepnum" is incremented by "stepincr" and the
heap is readjusted by performing percolate down on the head node,
requiring no more than logarithmic time in the number of callbacks.  The
"cbinfonum" is compared only if the "nextstepnum" of two different
callbacks match, so as to ensure that the callback registered earliest is
processed first.  The use of "cbinfonum" with "nextstepnum" imposes a
total order relation on the standard MD_Callback nodes.

The MD_Interface member "pfcb" is a list of force callbacks.  In this
case, there is no need for a priority queue since the entire list of
callbacks must be invoked for every force evaluation.  Building the list
by appending each force callback as it is registered ensures their in
order processing.

6.4.2.  Registering Callbacks

The front end registers callbacks through MD_callback(), MD_fcallback(),
and MD_msgcallback(), for standard, force, and message callbacks,
respectively.  A called back routine can be registered multiple times,
with each callback receiving different engine data subarrays through
the array of MD_Cbdata.  The MD_Cbdata struct is defined in mdcommon.h:

  MD_Cbdata {
  // set by front end for initialization
    int32 idnum;
    int32 access;
    int32 nelems;
    int32 first;
  // provided to the engine
    MD_Engdata *engdata;
  // provided to the callback routine
    void *data;
    MD_Attrib attrib;
  };

Each MD_Cbdata element defines a subarray of an engine data array.  As
stated above, the front end initializes the first four members of each
MD_Cbdata element.  The "idnum" member is set to the identification number
of the engine data array.  The "access" member establishes the desired
access to the array, by some ORing of flags MD_CBREAD, MD_CBWRITE, and
MD_CBSHARE for standard callbacks or flags MD_FCBREAD, MD_FCBWRITE, and
MD_FCBSHARE for force callbacks.  Note that the attributes for the
indicated engine data array must already permit the access requested.
The length of the subarray is indicated by "nelems" with the first index
indicated by "first".  Setting "nelems" to -1 specifies receiving up
through the last element in the array, with "first" still indicating the
first index of the subarray.  The routine that is called back will
receive the same MD_Cbdata array used when the callback was registered,
but with "data" and "attrib" members set.  The "data" member will point
to the first element in the specified subarray (not necessarily the first
element of the engine data array).  The "attrib" member will provide a
copy of the current engine data array attributes, which is most important
if the data array length is not known in advance.  The "engdata" member
is provided for the convenience of the engine, enabling the engine to
examine the the registered callbacks through calls to MD_callback_list()
and MD_fcallback_list() so as to determine what parts of which data
arrays are being passed to callback routines.

MD_callback() registration calls internal function cbdata_validate() on
each of the MD_Cbdata array elements in order to validate the engine data
array access request with the subarray indexing ranges; the "engdata"
pointer is also initialized.  If everything returns successfully, then a
new MD_Callback node is allocated, initialized, and appended to the
callback list.  The MD_Callback status "flags" are set to MD_CALLBACK_INIT.
Before MD_callback() returns, the internal function callback_setup() is
invoked on the MD_Callback node to setup for each MD_Cbdata array element
the "data" and "attrib" members.  MD_CALLBACK_SHARE status will be set for
the MD_Callback "flags" if any of the MD_Cbdata elements has "access" set
to either MD_CBSHARE or MD_FCBSHARE (see notes on callback processing for
more details on callback buffer sharing semantics).  Note that since
MD_CALLBACK_INIT status is set, a call is made to engdata_append_callback()
for each engine data array.  This appends the MD_Callback pointer to that
MD_Engdata "cback" list.  The purpose of each MD_Engdata object keeping a
list of callbacks involving that engine data array is to allow a fast
reverse lookup.  Note that if the buffer is resized, the MD_Cbdata "data"
pointer is no longer valid.  The data array manipulation routines
MD_setlen(), MD_setmax(), MD_resize(), MD_share(), MD_unshare(),
MD_engdata_setlen(), MD_engdata_setmax(), and MD_engdata_resize() all
conclude with a call to MD_engdata_cbsetup().  (This list includes all
data manipulation calls that might involve a change to either the data
buffer location or to the engine data array length attributes.)  The
MD_engdata_cbsetup() routine simply iterates through the MD_Engdata list
of "cback" and sets each of the MD_Callback "flags" to MD_CALLBACK_SETUP
to signify that callback_setup() must again be performed before invoking
the callback routine.  Also note that the MD_engdata_cbsetup() routine is
part of the engine API to permit explicit callback flag setting in the
case where the engine performs data buffer manipulations, for example,
swapping a data buffer.

MD_fcallback() registration performs almost the same as MD_callback().
The additional actions of MD_callback() not done by MD_fcallback()
include validating and initializing "stepincr", initializing
"nextstepnum" and "cbinfonum", and setting MD_CBUPDATE for the global
MD_Interface "flags" (see notes for mdrun.c discussing MD_Interface
status flags).  The "cbinfonum" member is set to the length of the
MD_Interface "pcbq" priority queue to designate the order in which the
callbacks were registered.  The "nextstepnum" is initialized to
MD_INT32_MIN to signal its need to be reset before callback processing.

Callbacks can be un-registered by the front end through calls to
MD_callback_undo(), MD_fcallback_undo(), and MD_msgcallback_undo() for
the standard, force, and message callbacks, respectively.  The function
pointer passed to the *_undo() routines is matched against the list of
callbacks of that type, and all callbacks to that routine are removed.
If NULL is passed for the function pointer, then the entire list of
callbacks is deleted.  When removing just a single function from being
called back, all three of the *_undo() routines compact their respective
MD_Interface callback array, keeping the relative ordering between the
remaining callbacks the same.  The standard and force callbacks invoke
callback_remove() on each MD_Callback node to be removed and then free it.
The callback_remove() routine iterates through the MD_Cbdata array for
the given callback, invoking engdata_remove_callback() on each MD_Cbdata
"engdata" pointer.  The engdata_remove_callback() routine updates the
MD_Engdata "cback" list by removing the MD_Callback pointer.

The MD_callback_undo() routine must also adjust "cbinfonum" for each
of the callbacks.  To maintain the "cbinfonum" ordering, the iteration
through the MD_Callback list subtracts the original list length from
"cbinfonum", and then after the array compaction a second iteration
through the new MD_Callback list adds back the new list length to
"cbinfonum".  Note that this could fail if there are more than
MD_INT32_MAX + 2 cumulative deletions from the MD_Callback list, but
this procedure should work fine in practice.  Note also that subsequent
calls to MD_callback() to register additional callback routines will
continue to assign larger "cbinfonum" numbers than those already assigned
to other callbacks.  The MD_callback_undo() routine concludes by setting
the MD_CBUPDATE status to the global MD_Interface "flags" to signal that
the heap must be rebuilt before the next processing of standard callbacks.

6.4.3.  Processing Callbacks

The engine is responsible during its run routine for processing callbacks.
Force callbacks need to be processed whenever the force is computed.
Standard callbacks might need to be processed before any integration steps
are taken (if MD_CBFIRST was passed to MD_run() as a run flag) and then
possibly after each completed step.  Message callbacks allow the engine to
communicate any interesting textual information to the front end.

The routines MD_ready_callback(), MD_ready_fcallback(), and
MD_ready_msgcallback() return true if there is some callback of that type
ready to be processed.  MD_ready_fcallback() and MD_ready_msgcallback(),
in practice, just report whether or not there any callbacks of that type
are registered.  MD_ready_callback() indicates whether the standard
callback at the head of the priority queue should be processed at the
current step number.  Note also that MD_ready_callback() returns true if
either of MD_CBRESET or MD_CBUPDATE global flags are set, so that the
priority queue will be immediately initialized (even if no callbacks
actually need to be processed on the current step number).

The engine processes callbacks of the three respective types by calling
MD_exec_callback(), MD_exec_fcallback(), and MD_exec_msgcallback().  All
of these routines have nonblocking semantics, so there are synchronization
routines MD_test_callback(), MD_wait_callback(), MD_test_fcallback(),
MD_wait_fcallback(), MD_test_msgcallback(), and MD_wait_msgcallback()
(see mdsync.c notes for details).  For force and message callbacks, the
entire list of callback routines is processed in sequence.  For standard
callbacks, callback routines whose "nextstepnum" value equals MD_Interface
"stepnum" are processed in sequence.  However, the MDAPI semantics permit
the engine to simultaneously process the three different callback types.
This means that the engine should probably not permit MD_CBWRITE and/or
MD_CBSHARE engine data array access to be combined with MD_FCBWRITE and/or
MD_FCBSHARE.  In case some engine array permits both standard and force
callback modify access, the engine should make sure that standard and
force callbacks are not processed simultaneously, so as to avoid
conflicting updates to the array.

The MD_exec_fcallback() routine to process force callbacks loops over all
callbacks in the MD_Interface "pfcb" list.  If MD_CALLBACK_SETUP is set
on a given callback, then callback_setup() must be called to correctly
initialize, for the MD_Cbdata array, the "data" buffer pointers and the
"attrib" values.  Next, the callback routine is invoked.  A nonzero return
value from the callback is reported by raising MD_ERR_CALLBACK error and
returning a fail state to the engine, which is then obligated to terminate
its run routine.  Otherwise, if the MD_CALLBACK_SHARE flag is set on the
callback, then callback_share() is invoked to complete the buffer copying.

The MD_CALLBACK_SHARE flag will be set for a callback if any of the
MD_Cbdata array elements have established MD_CBSHARE or MD_FCBSHARE
access.  The callback shared buffer semantics are different from read
and write semantics.  Instead of the MD_Cbdata element receiving a
"data" pointer that points to some subarray of an engine data array for
reading and/or writing, "data" is set to NULL and "nelems" and "first"
are both set to 0.  The callback is responsible for setting the "data"
buffer with the number of elements indicated in "nelems" and the index
where it should be copied into the engine data array given by "first".
The front end retains control of the buffer that is returned through
the MD_Cbdata element by the callback - the buffer will again be
available the next time that same callback is issued, and the front end
is responsible for freeing the memory.  The callback_share() routine
will, for each MD_Cbdata shared buffer, copy the buffer contents into
the corresponding engine data array.  If the engine data array is not
long enough to accommodate the buffer length and/or the indicated first
index, an attempt is made to resize the buffer - success then depends on
having MD_SETLEN and possibly MD_SETMAX access permission, with an error
reported if resizing is not permitted.  The purpose of a callback shared
buffer is to allow the front end to send arbitrary length arrays to the
engine during a simulation.  Note that shared buffer access is
incompatible with callback read or write access (and this is explicitly
checked in cbdata_validate() routine).  However, it is possible to have,
say, two different MD_Cbdata elements passed to a callback, one with
MD_CBREAD access on a particular engine data array and the other with
MD_CBSHARE access on the same array (assuming that the engine permits
both types of access on the array).  Also note that there are performance
considerations associated with callback shared buffer semantics, which
include an iteration over the entire MD_Cbdata array after each callback
and a copy from the front end provided buffer into the engine data array
buffer.

The MD_exec_callback() routine performs the same as MD_exec_fcallback()
on the leading MD_Callback nodes in the "pcbq" priority queue that require
processing.  Additional work is required to maintain the priority queue.
The priority queue is implemented as a binary heap.  (Detailed discussion
of binary heaps can be found in any reasonable data structures textbook.)
For our purposes, the binary heap sits within the allocated array - no
extra space is required.  The head node is, with respect to our ordering,
always in the front of the array, available in O(1) time.  An update to
the head node (modifying the ordering) requires O(log N) time to update
the heap order property (where N is the number of callbacks).  The entire
heap can be built into correct ordering in O(N) time.  The heap ordering
employs the following recipe, where 'a' and 'b' are two MD_Callback nodes:

  if (a->nextstepnum < b->nextstepnum
      || (a->nextstepnum == b->nextstepnum && a->cbinfonum < b->cbinfonum))
    // then we consider 'a' < 'b'

Note that since the MD_Callback "cbinfonum" assignments are unique, the
order relation is a total ordering of the MD_Callback nodes.  This order
relation with appropriate choice of "cbinfonum" guarantees that all
callbacks processed on the same step number are executed in the order in
which they were registered by the front end.

MD_exec_callback() first checks the MD_Interface global flags to see if
either MD_CBRESET or MD_CBUPDATE are set.  (See notes on mdrun.c for more
details.)  If either is set, update_nextstepnum() is called to initialize
some or all of the MD_Callback "nextstepnum" values.  Then heap_build()
is called to arrange the nodes into some proper ordering.  This callback
initialization needs to be performed at most once, on the first invocation
of MD_exec_callback(), during the engine run routine.

Next, MD_exec_callback() checks the head node to see if "nextstepnum"
is less than MD_Interface "stepnum".  If this is the case, it indicates
that the engine has not correctly processed callbacks when it should, so
an error is reported back to the engine run routine.

The MD_exec_callback() processing loop occurs while the head MD_Callback
node "nextstepnum" is equal to "stepnum".  Then the processing for that
callback is identical to that done for a force callback.  After the
callback processing, the head node is reordered by incrementing
"nextstepnum" by "stepincr" and then calling heap_update() to perform
a percolate down operation on the head node, a logarithmic time update
to correct the heap order property.  Note that since "stepincr" is
guaranteed to be positive, this processing loop will terminate.

6.5.  Routines for Running the Simulation (mdrun.c)

The MD_Interface members used for the run subsystem are:

  MD_Interface {
    ...
    int32 (*run)(struct MD_Interface_tag *, int32 numsteps, int32 runflags);
    int32 flags;
    int32 stepnum;
    ...
  };

The "run" function pointer stores the engine run routine.  The value of
this pointer is initialized during "engine_init()" through a call to
MD_setup_run().  The "run" routine is later invoked through a front end
call to MD_run().

The "flags" indicate the global status of the MD_Interface object.  The
enum constants defining the flags in mdcommon.h are:

  enum {
    MD_UPDATE   = 0x1000000,
    MD_CBFIRST  = 0x2000000,
    MD_CBRESET  = 0x4000000,
    MD_CBUPDATE = 0x8000000
  };

Run flag values passed to MD_run() that are less than or equal to 0xFFFF
are reserved to the engine to define engine-dependent flags.

The MD_UPDATE flag is set internally to signal the engine that the front
end has modified an engine array that had its MD_NOTIFY access flag set.
Front end modifications to an engine array are indicated by setting the
MD_MODIFY local status flag on the buffer access attribute.  Modifications
include array length or buffer resizing from MD_setlen(), MD_setmax(), and
MD_resize().  Modifications also include front end changes to the data
contents of an engine array resulting from MD_write(), MD_writesub(),
MD_share(), and MD_unshare().  The front end can also explicitly indicate
modifications to a buffer accessed through MD_direct() by calling
MD_setmod().  The engine receives the MD_UPDATE signal through the flags
parameter passed to the engine "run" routine.  It is then the job of the
engine to acknowledge all engine data array modifications by calling
MD_engdata_ackmod() on each modified buffer.  MD_run() concludes by
calling MD_clear_update(), which loops through the entire engine data
array list and clears the global MD_UPDATE status only if each buffer
access does not have both MD_NOTIFY and MD_MODIFY set (i.e. all modified
buffers with MD_NOTIFY access have been acknowledged as such by a call
during the engine run routine to MD_engdata_ackmod()).

The MD_CBFIRST flag is set by the front end as part of the "runflags"
passed to MD_run().  This indicates that any standard callbacks that
require initialization should be setup to be invoked before taking the
next time step.

The MD_CBRESET flag is set internally.  It indicates that the MD_Callback
"nextstepnum" for every standard callback needs to be reset.  (If the
MD_CBFIRST flag is also set, then "nextstepnum" is set to the current
step number; otherwise, it is set to the current step number plus the
"stepincr".)  MD_CBRESET flag is set only by MD_init_callback() (as part
of MD_init()) and by any front end call to MD_firststep() to reset the
step number.  MD_CBRESET also causes rebuilding of the standard callback
heap.

The MD_CBUPDATE flag is set internally.  It indicates that there is some
MD_Callback "nextstepnum" that needs to be reset.  (If the MD_CBFIRST
flag is also set, then "nextstepnum" is set to the current step number;
otherwise, it is set to the current step number plus the "stepincr".)
MD_CBUPDATE flag is set only by calls to register a standard callback with
MD_callback() or to remove a standard callback with  MD_callback_undo().
MD_CBUPDATE also causes rebuilding of the standard callback heap.

The MD_Interface "stepnum" is the global step number counter.  The step
number counter defaults to 0, but can be set by the front end by calling
MD_firststep() - this has a side effect of resetting standard callback
step numbering, as discussed above.  The engine, during its "run" routine,
is responsible for calling MD_incrstep() after completing an integration
step, which increments "stepnum".  The engine is also responsible during
its "run" routine to appropriately process callbacks (see the notes
regarding the callback subsystem for more details).  Both front end and
engine can call MD_stepnum() to see the value of "stepnum".

The engine "run" routine returns an int32 exit status, with zero
indicating success and nonzero indicating failure.  If there is a
nonzero exit status with no error having been reported through
MD_error(), then MD_ERR_RUN is reported.  The conclusion of MD_run()
checks "stepnum" against what should be the final step number after
processing the engine run routine; MD_ERR_RUN is reported if these
values do not match.

6.6.  Synchronization of Nonblocking Calls (mdsync.c)

The front end API defines nonblocking calling semantics for routines
MD_init(), MD_read(), MD_readsub(), MD_direct(), MD_update(), MD_run().
Note that MD_write() and MD_writesub() do not have nonblocking calling
semantics, because the engine data array writes are buffered on the front
end side, with array modifications not guaranteed to be visible to the
engine until a call to either MD_update() or MD_run().  The routines with
nonblocking semantics require the use of MD_test() and MD_wait() to
complete the calls before any other MDAPI routine is called.

The engine API defines nonblocking calling semantics for routines that
process callbacks.  MD_exec_callback() is completed by MD_test_callback()
and MD_wait_callback(), MD_exec_fcallback() is completed by
MD_test_fcallback() and MD_wait_fcallback(), and MD_exec_msgcallback() is
completed by MD_test_msgcallback() and MD_wait_msgcallback().  All three
types of callbacks can be processed simultaneously - however, during this
callback processing the engine should not call MD_engdata_setlen(),
MD_engdata_setmax(), MD_engdata_resize(), MD_engdata_ackmod(),
MD_engdata_cbsetup(), MD_error(), or MD_incrstep().

For the single-threaded MDAPI, all routines run to completion, so the
MD_test*() and MD_wait*() routines are trivial.  MD_test*() routines all
return 1 to indicate the asynchronous call has completed, and MD_wait*()
routines all return 0 to indicate that no additional errors occurred.

No state storage for nonblocking routines is presently needed within
MD_Interface.

6.7.  Error Reporting and Handling (mderror.c)

The error numbers provided by the MDAPI are defined as enum constants
in mdcommon.h.

  enum {
    MD_ERR_NONE = 0,           /* Success                         */
    MD_ERR_VERSION,            /* Inconsistent version number     */
    MD_ERR_MEMALLOC,           /* Memory cannot be allocated      */
    MD_ERR_INIT,               /* Engine initialization failed    */
    MD_ERR_RUN,                /* Engine run simulation failed    */
    MD_ERR_CHECK,              /* Consistency check failed        */
    MD_ERR_NEWDATA,            /* Cannot create new engine data   */
    MD_ERR_NEWTYPE,            /* Cannot create new type          */
    MD_ERR_NAME,               /* Unrecognized name string        */
    MD_ERR_IDNUM,              /* Invalid data ID number          */
    MD_ERR_TYPENUM,            /* Invalid type number             */
    MD_ERR_RANGE,              /* Value is out of range           */
    MD_ERR_ACCESS,             /* Access is denied                */
    MD_ERR_CALLBACK,           /* Callback routine failed         */
    MD_ERR_CBSHARE,            /* Callback shared buffer failed   */
    MD_ERROR_LISTLEN              /* not an error */
  };

Each error number has name prefixed by "MD_ERR_".  The last value in
this enum collection MD_ERROR_LISTLEN is not intended to be an error
number, rather it gives the list length.  There is a corresponding
static array of constant MD_Error named ErrorList given in mderror.c
that must have the same order as the enum constants.  The MD_Error type,
defined in mdcommon.h, has two fields

  MD_Error {
    const char *msg;
    int32 isfatal;
  };

where "msg" points to the error message text and "isfatal" is 0 if
the error can be reset or -1 if the error is fatal, meaning that use
of this MDAPI object must be terminated with MD_done().  The definition
of ErrorList follows.

  static const MD_Error ErrorList[MD_ERROR_LISTLEN] = {
    { "Success",                           0 },  /* MD_ERR_NONE     */
    { "Inconsistent version number",      -1 },  /* MD_ERR_VERSION  */
    { "Memory cannot be allocated",       -1 },  /* MD_ERR_MEMALLOC */
    { "Engine initialization failed",     -1 },  /* MD_ERR_INIT     */
    { "Engine run simulation failed",     -1 },  /* MD_ERR_RUN      */
    { "Consistency check failed",         -1 },  /* MD_ERR_CHECK    */
    { "Cannot create new engine data",    -1 },  /* MD_ERR_NEWDATA  */
    { "Cannot create new type",           -1 },  /* MD_ERR_NEWTYPE  */
    { "Unrecognized name string",          0 },  /* MD_ERR_NAME     */
    { "Invalid data ID number",            0 },  /* MD_ERR_IDNUM    */
    { "Invalid type number",               0 },  /* MD_ERR_TYPENUM  */
    { "Value is out of range",             0 },  /* MD_ERR_RANGE    */
    { "Access is denied",                  0 },  /* MD_ERR_ACCESS   */
    { "Callback routine failed",           0 },  /* MD_ERR_CALLBACK */
    { "Callback shared buffer failed",     0 },  /* MD_ERR_CBSHARE  */
  };

When changing the error list, it is sufficient just to insert or remove
corresponding entries in both the enum constant list and ErrorList.

The members of MD_Interface for error reporting and handling are

  MD_Interface {
    int32 errnum;
    MD_Error error;
    ADT_Array engerror;
    ...
  };

with "errnum" and "error" purposely given as the first two members of
MD_Interface.  "errnum" is used to record the error number, and "error"
receives a copy of value ErrorList[errnum].  The advantage to listing
them first is that they can record error messages even if the initial
validity check on the MDAPI version number fails (in this case, it could
be that the user code is using an inconsistent declaration of MD_Interface
from that of the linked library, however, it is still possible to record
the error value and ErrorList[MD_ERR_VERSION] into MD_Interface so that
the front end can still use the API to correctly detect error and cleanup).

Engine errors are stored in "engerror", which is initially empty.  The
MD_new_error() routine will append the error to engerror array and return
the newly defined error number (MD_ERROR_LISTLEN + old length of engerror).
MD_error() is used to raise an error condition based on the errnum.  If
the given errnum is less than MD_ERROR_LISTLEN, then the error is copied
from ErrorList[errnum], otherwise the error is copied from the
(errnum - MD_ERROR_LISTLEN)th element in engerror.  The MDAPI routines
also call MD_error() to report errors.  MD_error() by design always
returns MD_FAIL, so for MDAPI calls that return int32, it is convenient
to report an error using a line like:

  return MD_error(s, MD_ERR_MEMALLOC);

Front end (and engine) routines will generally receive MD_FAIL from an
unsuccessful MDAPI call.  The front end can retrieve the errnum using
MD_errnum(), the error message from MD_errmsg(), and can attempt to
reset a nonfatal error using MD_reset().  MD_reset() for a nonfatal
will call MD_error(s, MD_ERR_NONE) to reset the error state, then
return 0.

There are three internal functions that have global linkage:
MD_init_error(), MD_done_error(), and MD_error_version().  The setup
routine MD_init_error() is called by MD_init() (after validating the
version but before calling anything else) in order to set the initial
error state (errnum = 0, error = ErrorList[0]) and to intialize the
engerror array.  MD_done_error() cleans up the engerror array.  The
MD_error_version() routine handles the special case in which an
inconsistent version has been detected - in this case, the length of
contents of MD_Interface beyond the first two members cannot be trusted,
so this routine must be used instead of MD_error() to report the error.
