Classification of Artificial Neural Systems: Is Stochasticity
a Reliable Diagnostic Character?

by Wesley R. Elsberry
2000/06/21 (Notes added 2002/05/06)

William A. Dembski likes to lump in artificial neural systems
with computational approaches like evolutionary computation
and simulated annealing, calling the resulting assemblage
"evolutionary algorithms".  Dembski's basis for doing so is
the claim that all of these are dependent upon stochastic
processes.

[Quote]

By an evolutionary algorithm I mean any well-defined
mathematical procedure that generates contingency via some
chance process and then sifts it via some law-like
process. The Darwinian mechanism, simulated annealing, neural
nets, and genetic algorithms all fall within this broad
definition of evolutionary algorithms.

[End Quote - WA Dembski, 
"CAN EVOLUTIONARY ALGORITHMS GENERATE SPECIFIED COMPLEXITY?",
presentation at the "Nature of Nature" conference,
Baylor University, April, 2000.]

It should come as no surprise that this results in some
apparent inconsistencies.  The field of artificial neural
systems is incredibly broad.  The assertion that all ANS
models depend upon stochastic processes is simply false.

The following is a list of ANS models which do not depend
upon or utilize stochastic processes.  I'm using Patrick
Simpson's "Artificial Neural Systems" as my reference on
this.

  Outstar
  Instar
  Additive Grossberg
  Shunting Grossberg
  Binary Adaptive Resonance Theory (ART1)
  Analog Adaptive Resonance Theory (ART2)
  Discrete Autocorrelator
  Continuous Hopfield
  Discrete Bidirectional Associative Memory (BAM)
  Adaptive Bidirectional Associative Memory (ABAM)
  Temporal Associative Memory (TAM)
  Learning Matrix (LM)
  Drive-Reinforcement (DR)
  Linear Associative Memory (LAM)
  Optimal Linear Associative Memory (OLAM)
  Fuzzy Associative Memory (FAM)
  Brain-State-in-a-Box (BSB)
  Fuzzy Cognitive Map (FCM)
  Avalanche Matched Filter (AMF)

The following is a list of those ANS models which utilize
a stochastic process for initialization of structures or
values:

  Sparse Distributed Memory (SDM)
  Learning Vector Quantizer (LVQ)
  Counter-Propagation Network (CPN)
  Perceptron
  Adaline/Madaline
  Backpropagation [see Note 1]

It should be noted that non-stochastic variants of some of the
above models are possible to construct.  For example, Adaline
implementations work fine starting from a basis of setting the
weights to some fixed non-zero value; the randomization is
actually for the proper function in Madaline ANS.  Another
example is in backpropagation, where random initial weights
are assigned for the purpose of "symmetry breaking".  This can
also be accomplished by setting each initial weight to a
unique non-zero value via a deterministic algorithm.

The following is a list of ANS models which depend upon
stochastic processes as part of the learning procedure:

  Boltzmann Machine (BM)
  Cauchy Machine (CM)
  Adaptive Heuristic Critic (AHC)
  Associative Reward-Penalty (ARP)

This last list is the only group of models within ANS that
legitimately fit Dembski's quoted characterization.  [See
Note 2]

These lists demonstrate the diversity of approaches to ANS
modelling.  The assertion that all ANS depend upon stochastic
processes is false.  Classification of all ANS models on the
basis of a character held only by a small minority of those
models is risible.



References

Simpson, Patrick K.  1990.  Artificial Neural Systems:
Foundations, Paradigms, Applications, and Implementations.
Elmsford, New York: Pergamon Press.


Notes

1. Iain Strachan has pointed out that terminology has changed
since Simpson's volume, and it is now more typical to refer
to what Simpson called "Backprogation" as a "multi-layer
perceptron trained via backpropagation of error".  The
backpropagation technique has itself diversified as well,
with both stochastic and deterministic variants.

2. These models are obligately dependent upon stochastic
processes.  The first set of models had no dependence 
whatever upon stochastic processes.  The second set of
models typically do use stochastic processes in application,
though it is possible to deploy some of them without any
depedence upon stochastic processes.