A alternate way to think the architecture of network
- Start with direct I/O connections only. No hidden units.
- Train output-layer weights using BP or Quickprop.
- If error is now acceptable, quit.
- Else, Create one new hidden unit offline.
- Create a pool of candidate units. Each gets all available inputs. Outputs are not yet connected to anything.
- Train the incoming weights to maximize the match (covariance) between each unit’s output and the residual error:
- When all are quiescent, tenure the winner and add it to active net. Kill all the other candidates.
- Re-train output layer weights and repeat the cycle until done.
Why Is Backprop So Slow?
- Moving Targets
- All hidden units are being trained at once, changing the environment seen by the other units as they train.
- Herd Effect
- Each unit must find a distinct job -- some component of the error to correct.
- All units scramble for the most important jobs. No central authority or communication.
- Once a job is taken, it disappears and units head for the next-best job, including the unit that took the best job.
- This is a very inefficient way to assign a distinct useful job to each unit.
Advantages of Cascade Correlation
- No need to guess size and topology of net in advance.
- Can build deep nets with higher-order features.
- Much faster than Backprop or Quickprop.
- Trains just one layer of weights at a time (fast).
- Works on smaller training sets (in some cases, at least).
- Old feature detectors are frozen, not cannibalized, so good for incremental “curriculum” training.
- Good for parallel implementation.