→ ↺

MGL-PAX documentation for clips

Table of Contents

• 1 Motivation and concept
• 2 Documented interface
  • 2.1 The clip-1994 ASDF System
• 3 Loading CLASP format
  • 3.1 The clip-1994/loader ASDF System
• 4 Clip classes
• 5 New locatives
  • 5.1 The clip-1994/doc ASDF System
• 6 Demos
  • 6.1 The clip-1994/demo ASDF System
  • 6.2 Documentation of the simple agent experiment
  • 6.3 Documentation of the agent experiment
  • 6.4 Documentation of the super agent experiment
• 7 Background

[in package CLIP/DOC]

← ↑ → ↺

1 Motivation and concept

From the original manual:

We collect information from software systems for many reasons. Sometimes, the very purpose of the system is to produce information. Other times, we collect data for debugging, feedback to the user - in general, for understanding the system's behavior. Unfortunately, there seem to be few general tools available for this task. Instead, we often find ad hoc, domain-specific code, spread throughout a system, varying from component to component, despite strong similarity in the requirements for any data collection code.

Clip, the Common Lisp Instrumentation Package, standardizes data collection. It is named by analogy with the "alligator clips" that connect diagnostic equipment to electrical components. By standardizing data collection, we can more easily add and delete instrumentation from our systems. We can reuse a piece of data collection code, called a CLIP in other systems we are studying. We can use general tools to analyze the collected data. Instrumentation in a system becomes more easily used and understood, because the basic functionality of the system is separated from the functionality of the instrumentation. We designed Clip to be used for scientific experiments, in which a system is run under a number of different conditions to see whether and how the behavior changes as a function of the condition. Consequently, we view an experiment as a series of trials, varying the settings of experiment variables and collecting information. Generally speaking, an experiment comprises the following steps:

1. Creating clips and inserting them into the system to be studied. Generally, clip measures one particular aspect of a system, but you can also define clips that combine many measurements, including other clips. Often, the clips you need will already be defined.

2. Define the experiment, in terms of what system is to be run, any necessary initialization code, and the conditions under which it is to run (different input parameters or environmental conditions).

3. Run a series of trials, saving the output into a Clasp format data file. This format is described in the Clasp manual section Data Manipulation Functions, although it isn't necessary to understand the file format. Clasp can read what Clip writes.

Clip can also write data files in standard tab or space delimited format used by most other statistical packages, databases and spreadsheets.

Instead of CLASP, CLIP/LOADER::LOAD-MEASUREMENTS can be used to load the data as lisp-utils frames.

• [glossary-term] CLIP (Common Lisp Instrumentation Package)

  A tool for automating experimentation and data collection. Clip was originally designed to be used in combination with the Common Lisp Analytical Statistics Package (CLASP), though it can be used as a standalone instrumentation package for Common Lisp applications.

  The original 1.4 version was released in Jan 1994. This version was modified to run on Common Lisp systems in 2023.

  Clasp is no longer easy to obtain and run in 2023, so a loader was also added.

• [glossary-term] CLIP

  There are basically only a small number of ways to instrument a software system. These are: adding special purpose code to collect data, interrogating global (usually objects with state) data structures or using advice.

  The first way is to build in special purpose code to collect data about the state of the system while the system is running. We used this technique in the Phoenix testbed to keep information about the execution time of timeline entries and also message traffic patterns. The Transsim simulator also uses this technique when it keeps track of daily costs, ship locations, demon ring intervals. The key point here is that the only reason for adding the piece of code was to allow an experimenter to analyze the behavior of the system. The simulation itself does not use the information. This method increases the complexity and reduces the readability and maintainability of the software system. In code that is highly instrumented it is often difficult to determine what is intrinsic to the running of the system and what is used to instrument.

  Another method involves interrogating global data structures to determine the state post hoc. Examples include most everything we collected about Phoenix (i.e., fireline built, first plan tried, bulldozer digging times, etc.). This technique is fine if the information is hanging around after you have finished running the simulation, but this is not always the case. Also, there is some information that is time-sensitive, and therefore must be collected on a periodic or event-driven basis. Collecting this type of information often involves resorting to the first method -- altering the code itself.

  Alternatively, one can use the advise facility available in many lisp environments to non- intrusively collect information when functions are called. This is a nice modular approach, but requires a good deal of knowledge about the advise facility and the software system itself. Unfortunately, the advise facility is not standardized across Lisp implementations. The DEFCLIP macro encapsulates the code necessary to instrument a software system and separates it from the system itself. It avoids the pitfalls of adding special purpose code directly to the software system while at the same time allowing periodic and event-driven data collection. It also allows collection to be done by perusing global data structures.

← ↑ → ↺

2 Documented interface

• [package] "CLIP"

• [macro] DEFCLIP NAME ARGS &REST BODY

  Defines a method for managing a set of functions used to collect data.

  :COMPONENTS is list of clips that are associated with this clip; ie., they are collected and reported as a group by collecting or report this clip.

  :MAP-FUNCTION provides a list of items to map the `components' over.

  :REPORT-KEY allows overriding of the default key output to the data stream.

  :PERIOD provides the time interval between collections, if this is not specified collection will be done immediately prior to the report is output to the data stream.

  :TRIGGER-EVENT is a function which triggers collection

  :ENABLE-FUNCTION, :DISABLE-FUNCTION, :RESET-FUNCTION, :DISPLAY-FUNCTION are code that will be called to enable, disable, reset and display the clip.

  :REPORT-FUNCTION can be used to override the default report function.

  For example:

  Another simple one with code used to report a value:

     (defclip number-of-dead-bulldozers ()
       (:report-key "# of Dead bulldozers")
  
       (length (bds-that-died)))

  An example showing one with components:

    (defclip methods-each-bd (bulldozer)
      "Salient info for each instance of applying a recovery method:"
      (:components (trial-number agent-name method-type failure-type calculate-recovery-cost
                    method-succeeded-p order-of-application)
       :map-function (gather-recovery-method-instances (name-of bulldozer)))
      ;; This code executes before the map-function is executed.
      (send (fire-system) :set-frame-system (name-of bulldozer)))

  This one reports no values it simply provides an interface to some metering code:

     (defclip scorched-earth-meter ()
       "Records amount of torched terrain."
       (:enable-function  (send (fire-system) :turn-on-scorched-earth-metering)
        :disable-function (send (fire-system) :turn-off-scorched-earth-metering)
        :reset-function   (send (fire-system) :reset-scorched-earth-meter)
        :display-function (send (fire-system) :display-scorched-earth-meter)))

• [macro] DEFINE-SIMULATOR NAME &KEY DESCRIPTION SYSTEM-NAME SYSTEM-VERSION RESET-SYSTEM START-SYSTEM STOP-SYSTEM SCHEDULE-FUNCTION DEACTIVATE-SCHEDULED-FUNCTION SECONDS-PER-TIME-UNIT TIMESTAMP

  Define the interface to a simulation. The following args are recognized:

  :SYSTEM-NAME system-name

  :SYSTEM-VERSION function or form that handles same args as :BEFORE-EXPERIMENT

  :START-SYSTEM function or form that handles same args as :BEFORE-TRIAL

  :RESET-SYSTEM function or form that handles same args as :BEFORE-TRIAL

  :STOP-SYSTEM function or form that handles same args as :BEFORE-TRIAL

  :SECONDS-PER-TIME-UNIT value (default is 1)

  :SCHEDULE-FUNCTION (lambda (function time period name) . body) - interface to system event scheduling mechanism

  :DEACTIVATE-SCHEDULED-FUNCTION (lambda (scheduled-function-object) . body) - interface to system event unscheduling mechanism

  :TIMESTAMP a function that returns the current simulation time

• [macro] DEFINE-EXPERIMENT NAME ARGUMENTS &BODY BODY

  Defines an experiment named `name' that can be run using the RUN-EXPERIMENT function.

  :SIMULATOR should be the name used in a previous define-simulation form.

  :BEFORE-TRIAL and :AFTER-TRIAL are called with the current values of the independent variables and the arguments.

  :BEFORE-EXPERIMENT and :AFTER-EXPERIMENT are called with the just the arguments. The value of all of these should be a form that refers to the arguments or a function that handles the correct number of arguments.

  :INSTRUMENTATION is a list of names defined with defclip' which will be enabled during the experiment. Usewrite-current-experiment-data' in the after-trial code to write the data to the appropriate output files.

  :IVS ({(var exp) | (var )}*) - define independent variables Example: ((iv1 from 1 to 6 by 5) (iv2 '(a b)))

  :LOCALS ({(var exp) | var}*) - bind variables in the context of the experiment

  :SCRIPT ({(descriptive-name initial-time [next-time] form) | script-element-name}*) initial-time' can be a string, a number or form to evaluate.next-time' is a time interval and should be a fixnum or form.

  The way things work:

   Run :BEFORE-EXPERIMENT code with args
   LOOP
     Update IVs for trial
     Run :BEFORE-TRIAL code with IV's and args
     Call :RESET-SYSTEM function with IV's and args
     Instantiate Script Events
     Reset and Enable all the instrumentation
     Call :START-SYSTEM function with IV's and args
     <system runs (possible periodic and event based collection done)>
     Run :AFTER-TRIAL code (possible post-mortem collection done)
   END LOOP when all trials completed
   Run :AFTER-EXPERIMENT code with args

• [function] RUN-EXPERIMENT EXPERIMENT-NAME &KEY ARGS (REPETITIONS 1) NUMBER-OF-TRIALS LENGTH-OF-TRIAL OUTPUT-FILE (ERROR-STREAM *ERROR-OUTPUT*) ERROR-FILE EXTRA-HEADER (SUPPRESS-HEADERS CLIP::*SUPPRESS-HEADERS*) (OUTPUT-FORMAT *OUTPUT-FORMAT*) (STARTING-TRIAL-NUMBER 1)

  Run the experiment named EXPERIMENT-NAME. OUTPUT-FILE is optional, but must be specified if WRITE-CURRENT-EXPERIMENT-DATA is called from within your experiment. ERROR-STREAM can be used to direct output to an open stream. It defaults to the error-output which sends error/debug output to the appropriate place. ERROR-FILE can also be used to direct the error/debug output to a file. EXTRA-HEADER is written at the end of the header of the output file. STARTING-TRIAL-NUMBER can be used to change this value to something other than one (1). If NUMBER-OF-TRIALS is not specified it will be calculated so as to vary all the independent variables across all their values REPETITIONS (default 1) times.

• [function] WRITE-CURRENT-EXPERIMENT-DATA &KEY (SEPARATOR *DATA-SEPARATOR-CHARACTER*) (FORMAT *OUTPUT-FORMAT*) INSTRUMENTATION STREAM

  Causes each experiment instrumentation to write its data to FILENAME. SEPARATOR should be a character which will be used to separate fields. It defaults to the value of data-separator-character. FORMAT should be one of :CLASP which means write a clasp native format data file, :ASCII which means write a standard SEPARATOR delimited data file including column names or :DATA-ONLY which is the same as :ASCII except no column names are included. It defaults to the value of output-format. INSTRUMENTATION can be used to specify a subset of the experiments instrumentation to write to the data file.

• [generic-function] CLIP::COLLECT INSTRUMENTATION &REST ARGUMENTS

← ↑ → ↺

2.1 The clip-1994 ASDF System

• Version: 1.4.1
• Description: Original parts of the CLIP (Common Lisp Instrumentation Package). The name is suffixed with the original release year to prevent collision with another unrelated asdf system that is in the Quicklisp.

← ↑ → ↺

3 Loading CLASP format

• [package] "CLIP/LOADER"

• [function] CLIP/LOADER:LOAD-MEASUREMENTS FILE &REST NAMES

  Load frames from a single CLASP file to variables and packages in NAMES.

  As an example

  (load-measurements
     (asdf:system-relative-pathname "clip-1994" "demos/agent-simulatordata-1-31-94-10-39.clasp")
   'simulator-data)

  creates global variable SIMULATOR-DATA in current package with the data frame, new package "SIMULATOR-DATA", and symbols such as SIMULATOR-DATA::HIGHEST-AGENT.

• [function] CLIP/LOADER:LOAD-CLASP-FRAME NAME STREAM

  Load data from a CLIP experiment to variable and package NAME.

  NAME is upcased and turned to a variable that contains the data frame.

  Aditionally, package of same name is created and a symbol in this package is created for each column of the experiment.

  As second value, return experiment run description.

← ↑ → ↺

3.1 The clip-1994/loader ASDF System

• Version: 1.4.1
• Description: Auxiliary functions for loading of the data in CLASP format.

← ↑ → ↺

4 Clip classes

This is for a reference from clips.

• [class] CLIP::INSTRUMENTATION

  Simple clips have no components and collect data immediately prior to reporting it to the output le at :AFTER-TRIAL time. If they are defined with a :schedule or :trigger-event defclip option their default behavior is store all of the data collected during a trial and report a single value which is the mean of all the values collected.

• [class] CLIP::COMPOSITE-INSTRUMENTATION

  Clips with components, as specified by the :components keyword, generate multiple columns in a data le each time they are reported. Without other options they produce one column per component (composite clips).

  All clips with components are referred to as super clips. For a good example of clips with components and further discussion of their use,

• [class] CLIP::SUPER-INSTRUMENTATION

  Mapping clips are specified using the :map-function and :components option to defclip. They generate multiple columns of data each time they are reported - they produce multiple columns per component.

  All clips with components are referred to as super clips. For a good example of clips with components and further discussion of their use, see Appendix A.1.1.

• [class] CLIP::PERIODIC-SUPER-INSTRUMENTATION

  Clips that have COMPONENTS and SCHEDULE option are periodic time-series clips. They generate multiple data columns in the manner of component clips (which they are) and also multiple data rows. Each row corresponds to a single collection activated periodically. Since time-series clips generate multiple rows, they are generally written to a data le that is separate from the main experiment (summary) data le. The name of the data file associated with a time-series clip is speci ed using the OUTPUT-FILE option to defclip. The SCHEDULE-FUNCTION, SECONDS-PER-TIME-UNIT, and TIMESTAMP keywords to define-simulator must be specified.

• [class] CLIP::FUNCTIONAL-COMPOSITE-INSTRUMENTATION

  Clips that have COMPONENTS and TRIGGER-EVENT option are event based time-series clips. They generate multiple data columns in the manner of component clips (which they are) and also multiple data rows. Each row corresponds to a single collection and is triggered by a particular event. Since time-series clips generate multiple rows, they are generally written to a data file that is separate from the main experiment (summary) data file. The name of the data le associated with a time-series clip is specified using the OUTPUT-FILE option to defclip. Event-based time-series clips require that the Common Lisp implementation provide some mechanism similar to the advise function.

• [class] CLIP::COMPOSITE-CHILD-OF-COMPOSITE-INSTRUMENTATION

• [class] CLIP::COMPOSITE-TIME-SERIES-INSTRUMENTATION

  Time series clip that uses explicit CLIP::COLLECT to collect the data.

• [class] CLIP::FUNCTIONAL-MAPPING-INSTRUMENTATION

  Event based collection with mapping.

← ↑ → ↺

5 New locatives

• [package] "CLIP/DOC"

  Additional locatives for clip components.

• [locative] CLIP

  An instrumentation (clip, alligator clip). Documentation is taken from the docstring in the DEFCLIP.

• [locative] EXPERIMENT

  An experiment. Documentation is taken from :DESCRIPTION of the DEFINE-EXPERIMENT.

• [locative] SIMULATOR

  An instrumentation (simulator, alligator simulator)

← ↑ → ↺

5.1 The clip-1994/doc ASDF System

• Version: 1.4.1

← ↑ → ↺

6 Demos

← ↑ → ↺

6.1 The clip-1994/demo ASDF System

• Version: 1.4.1
• Description: Original demos for the CLIP system. They show different ways to use clips.

← ↑ → ↺

6.2 Documentation of the simple agent experiment

[in package CLIP-USER]

This example collects trial summary data about overall agent-cost and task completion-time. Collection occurs at the end of each trial and is written out to a summary file in Clasp format.

• [simulator] AGENT-SIM-1

• [clip] AGENTS-COST

  Collects the combined cost of all agents at the end of each trial.

  Class: CLIP::INSTRUMENTATION

• [clip] COMPLETION-TIME

  Records the time at which the trial ends.

  Class: CLIP::INSTRUMENTATION

• [experiment] SIMPLE-AGENT-EXPERIMENT-1

  A test experiment.

  Instrumentation: (AGENTS-COST COMPLETION-TIME)

← ↑ → ↺

6.3 Documentation of the agent experiment

[in package CLIP-USER]

This example collects trial summary data about overall agent-cost and task completion-time. Collection occurs at the end of each trial and is written out to a summary file in Clasp format.

• [simulator] AGENT-SIM

• [clip] HIGHEST-AGENT-STATE

  No documentation supplied.

  Class: TLS-SERVER/MEASURE::CALLBACK-INSTRUMENTATION-MIXIN

• [clip] POSTHOC-AGENT-STATE-SNAPSHOT

  No documentation supplied.

  Class: CLIP::SUPER-INSTRUMENTATION

• [clip] EACH-AGENT-STATE-SNAPSHOT AGENT

  Record the state at an agent.

  Class: CLIP::INSTRUMENTATION

• [clip] PERIODIC-AGENT-STATE-SNAPSHOT

  No documentation supplied.

  Class: CLIP::PERIODIC-SUPER-INSTRUMENTATION

• [clip] CHANGE-OF-STATE-PRED

  No documentation supplied.

  Class: CLIP::FUNCTIONAL-COMPOSITE-INSTRUMENTATION

• [clip] CHANGE-OF-STATE AGENT-NAME NEW-STATE

  No documentation supplied.

  Class: CLIP::FUNCTIONAL-COMPOSITE-INSTRUMENTATION

• [clip] CHANGE-OF-STATE-3

  No documentation supplied.

  Class: CLIP::FUNCTIONAL-COMPOSITE-INSTRUMENTATION

• [clip] WILMA

  No documentation supplied.

  Class: CLIP::COMPOSITE-CHILD-OF-COMPOSITE-INSTRUMENTATION

• [clip] BETTY

  No documentation supplied.

  Class: CLIP::COMPOSITE-CHILD-OF-COMPOSITE-INSTRUMENTATION

• [clip] EVENT-BASED-AGENT-STATE-SNAPSHOT

  No documentation supplied.

  Class: CLIP::FUNCTIONAL-MAPPING-INSTRUMENTATION

• [clip] SELF-COLLECTION TWO-ITEM-LIST

  No documentation supplied.

  Class: CLIP::COMPOSITE-TIME-SERIES-INSTRUMENTATION

• [experiment] AGENT-EXPERIMENT &KEY (VERBOSE T)

  A test experiment.

  Instrumentation: (AGENTS-COST ALL-AGENTS-COSTS COMPLETION-TIME HIGHEST-AGENT-STATE CHANGE-OF-STATE-PRED CHANGE-OF-STATE-3 CHANGE-OF-STATE PERIODIC-AGENT-STATE-SNAPSHOT SELF-COLLECTION EVENT-BASED-AGENT-STATE-SNAPSHOT POSTHOC-AGENT-STATE-SNAPSHOT)

• [experiment] SAE

  A test experiment.

  Instrumentation: (POSTHOC-AGENT-STATE-SNAPSHOT PERIODIC-AGENT-STATE-SNAPSHOT)

← ↑ → ↺

6.4 Documentation of the super agent experiment

[in package CLIP-USER]

This example collects trial summary data about overall agent-cost and task completion-time. Collection occurs at the end of each trial and is written out to a summary fole in Clasp format.

• [clip] ALL-AGENTS-COSTS

  Super clip to collect cost of each agent.

  Class: CLIP::SUPER-INSTRUMENTATION

• [clip] EACH-AGENT-COST AGENT

  No documentation supplied.

  Class: CLIP::INSTRUMENTATION

• [experiment] SIMPLE-AGENT-EXPERIMENT-2

  A test experiment.

  Instrumentation: (AGENTS-COST ALL-AGENTS-COSTS COMPLETION-TIME)

← ↑ ↺

7 Background

• [glossary-term] CLASP

  Common Lisp Analytical Statistics Package - another tool of same origin as CLIP.

• [glossary-term] CLASP format

  Format of the data used by clasp and documented in CLASP manual.

• [glossary-term] Phoenix system

  Phoenix was a multi-agent planning system that fights simulated forest-fires. The simulation used terrain, elevation, and feature data from Yellowstone.

  It is used for the demos. See original manual for more specifics.