Neural Networks 2

No loops Activation flows: input-hidden-output

Has loops or feedback Activation flows indefinitely Can model the time course (activity over time) of cognition Ex. An Interactive model of spoken word production is a recurrent neural network

Hopfield network TRACE model of word recognition Simple recurrent network of sentence processing

Retrieval of a whole memory from just a piece of the memory Ex. If we see the piece “to be or not to be,” we know what comes next is “that is the question.”

A model of auto-associative memory or pattern completion Patterns can be anything Example of unsupervised learning Every node is connected to every other node, and weights are symmetrical Nodes have binary activation (usually -1 or +1) At each time step t, pick a random node, and update its activation for time t+1 based on its current input at t Simultaneously updating all neurons is also possible Learning/weight setting is typically done using Hebbian learning

Model of how we learn structure in sequences (sentence processing) Need to remember the earlier parts of the sequence to predict the next unit, which SRN does. Using a context layer to encode for memory of the previous parts The hidden layer encodes for the current input and the history from the context layer Trained using a variant of backpropagation

Simulates activation over time of acoustic features, phonemes, and words An example of interactive processing with multiple layers interacting with each other continuously It is made up of three layers Inputs to features Features to Phonemes Phonemes to Words It has excitatory connections between layers It has inhibitory connections within layers (inhibits other words/phonemes since you can’t hear two words at the same time)

The feature layer receives speech input that is encoded, not actual speech sounds (human voices) Some of the features are power, vocalic, diffuseness, acuteness, consonantal, voicing, and burst Each feature has 9 levels Each phoneme has a particular level for each feature Each feature neuron maps to the phoneme level Phonemes have a peak of activation (around six time cycles in) Phonemes with similar sounds have similar features, and compete more

There is a separate neuron for every phoneme, at every time slice

There is a separate neuron for every word, at every time slice

Features for neighboring phonemes overlap which mimics the way humans speak

Through bottom-up activation (feature - phoneme - word) and top-down activation (word - phoneme)

Bottom-up excitation causes initial activation (feature - phoneme - word) They activate phonemes at the correct time, corresponding to when they should be said Excitation increases with temporal overlap and similarity This leads to partial stimulation provided to the units similar to the original unit.

Top-down activation is when word units feedback to their phonemes Can activate phonemes that were not “heard” (earlier or later) via lexical feedback Can clean up ambiguous input

Words and phonemes inhibit other words and phonemes (respectively) that overlap in time

The rhyme competes with the target, unlike the cohort theory, which says rhyme does not compete. Because in TRACE, words are activated by phonemes that all slots in the word, so it doesn’t rule out words that are inconsistent in the onset with what’s heard. The cohort competitor competes with the target more than the rhyme. The cohort competitor is activated more since the target beaker is going to be less active early on in the input because less input has been heard. So the beaker won’t be able to inhibit the beetle as much, as later it can inhibit the speaker more. Also leads to the speaker being activated later than the cohort competitor

If you present subjects with a word that has an ambiguous sound (at the beginning or end), they are more likely to hear that sound in a way that makes the word into a real word Ex. /?iss/ - /kiss/ The effect goes away under time pressure. The effect is also stronger/faster at word offset (ambiguous sound at the end of the word) than onset.

TRACE theory says that hearing /?iss/ activates /kiss/ due to the lexical feedback down to /k/ phoneme TRACE explains the effect of going away under time pressure by saying that feedback takes time so if you do it under pressure, it does not give it enough time for lexical feedback to happen TRACE explains the effect being stronger/faster for the offset of the word because word-final can be predicted by preceding lexical information

It is not a model of learning, so it does not learn Does not work on real speech Representation of time is unrealistic

Mental information represented by strings of symbols as in Turing Machine

Mental Information is stored in the weights or connection strengths between the units of a neural network

Eliminative and implementational connectionism

Neural networks eliminate the need for symbols

Can implement a neural network in a classical computer Can implement a classical computer in a neural network