Backpropagator's Review

What does the reader do when he wishes to see in what the precise likeness or difference of two objects lies? He transfers his attention as rapidly as possible, backwards and forwards, from one to the other. The rapid alteration of consciousness shakes out, as it were, the points of difference or agreement, which would have slumbered forever unnoticed if the consciousness of the objects compared had occurred at widely distant periods of time. What does the scientific man do when he searches for the reason or law embedded in a phenomenon? He deliberately accumulates all the instances he can find which have any analogy to that phenomenon; and by simultaneously filling his mind with them all, he frequently succeeds in detaching from the collection the pecularity which he was unable to formulate in the one alone; even though that one had been preceded in his former experience by all those with which he now at once confronts it.

                       -- William James, 1890
Copyright 1996-2004 by Donald R. Tveter, commercial use is prohibited. This material CAN be posted elsewhere on the net if the posted files are not altered in any way but please let me know where it is posted. The main location is:

Version from January 7, 2004


This document is something of a backprop FAQ, reading list (with a preference for online articles) and a summary of the state of the art. Its aimed at people who want practical advice on using backprop. Note that for the most part I am trying to avoid duplicating information that is already found in the NN FAQ that can be found at SAS Institute, North Carolina or a plain text version is available from the MIT FAQ collection and SAS Institute, North Carolina. In particular I am leaving it to the curator of the NN FAQ to list books and general backprop software.

A Plea for Help

I'm always on the lookout for worthwhile articles and experiences with the various techniques, if you can suggest articles or give your experiences with a technique please let me know. I really want to concentrate on practical methods not theory. If you have any answers that you think should be included please tell me about them. If you have any questions that should be answered please let me know, if nothing else I will create a category with a plea for help. While I am trying to be accurate note that I may make mistakes. Please tell me about them!

A Plea for Programs

There are so many papers being published on backprop that no one can keep up. Reading and understanding the typical backprop paper can take a very long time and programming the method can take even longer. Therefore I want to make a plea to the authors of backprop papers to make their programs available in a form such that users can test the methods on their practical problems without having to understand much about the method. Of course I will be happy to list them here.

Viewing These Pages

Now I am joining the crowd and recommending Netscape although lynx works perfectly well. In Mosaic there is a problem where the program will not jump to an empty anchor:

<a name="xyz"></a>.

To fix this there are asterisks placed in the anchor text:

<a name="xyz">*</a>

This means you will find many highlighted but unselectable asterisks in this document. Some section headings will also show up as selectable when in fact they are not, they are just being used as places you can jump to. (The 0.3b2 Grail browser highlights these in yellow when you go to them.)

If you want a copy of these files as a .zip file for yourself here it is, about 100K.

The Topics


I want to thank the following people for help:

  • Remi Coulom
  • Ofer Melnik
  • Robert Parker
  • Bruno Lienard
  • Fredrik Tengroth
  • Alois P. Heinz
  • Ian Cresswell
  • Randolf Werner
  • Douglas B. Kell
  • Martin Brown
  • C. Lee Giles
  • Patrick van der Smagt
  • Roy Goodacre
  • David Elliott
  • Perry Moerland
  • G. Thimm
  • Steve Lawrence
  • Wolfram Schiffmann

If you have questions or comments, write me.

To Don's Home Page