Analysis and synthesis of natural language texts with recurrent neural networks
from
Paul Werner
submitted on: 06/18/2008
Main-Goal:
-------------
Analysis of natural spoken Phrases and determining the meaning without
depending on the formulation.
Algorithmic generation of appropriate Learn- and Validationsets
(Example Phrases) to learn and test the neural net.
Specification of Phrase-Meaning:
----------------------
Allocation of semantic roles for selected phrase-parts
Max (Agent) loves (Verb) Lisa (Patient)
Lisa (Patient) is loved (Verb) by Max (Agent)
Semantic roles are not grammatical phrase-parts
remember (Subject, Predicate, Object)
Allocation of semantic roles will be the same, although a changed
phrase-formulation.
semantic roles can defined at one's own discretion
Ambiguity
-------------------------
Problem: Natural Language is formulated ambiguous all too often.
Example: "Time flies like an arrow"
- Time flies so fast, like the arrow flies away.
- Time (to time) flies like an arrow is doing that.
- Time (to time) flies like an arrow lock alike.
- "Timeflies" like (to care for) an arrow.
- Syntactical knowledge is not enough to determine the meaning of a phrase.
- We need contextual knowledge.
- Problem: A parser cannot store or compute the complete
sense of the world
Neural Net
----------------
- Idea: Motivation by the human skills, to recognize phrases (Neuro / Biology)
- Learn a small amount of example-phrases with correct allocated semantic roles.
- supervised learning analog the childish speech acquirement.
- Abstraction and generalization for the correct analysis of unknown phrases.
- Correct allocation of semantic roles for not learned phrases.
Feed-Forward-Net
-----------------
Limitations:
- Input-Layer has a fixed size (Amount of Input-Neurons).
- cannot accept phrases with different lengths (wordcount).
- no "internal memory". so we cannot propagate single words (word by word).
- no commonness to the human language understanding
Elman-Net
---------
- "internal" memory is established by context-neurons (Nr. 4 and 5)
- stores the activation of the hidden neurons the timestep before
depending on the memory-factor LAMBDA
- elman-net is able to determine chronological dependencies in the input-sequence
Idea:
- Phrase is propagated sequential (word by word). Also during the learning-process
- output of the words with (its) allocated semantic role.
Learning-Set / Validation-Set
-----------------------
- Definition of a generative grammar to produce a large set for
learning and valiating (example phrases)
- non-terminals are substituted with the concrete
word-assignment (projector)
- it produces a large language amount (Example: 12x3x3x3x3x3 = 2916 Phrases)
- only a small fraction of this amount has to be put in the learn- und validationset
-> Learn-Density p € [0,1].
-> neural-net has to generalize. only a small amount of
learning-phrases must be enough.
Validation-Set subseteq Complete set MINUS Lerning-Set
Encoding of Net-Input
----------------------
...
Learning-process Active- and Passivephrases
------------------------------------------
Example Active- and Passivephrases
- Total amount of 7488 Phrases.
- Learndensity p=0.01 results 75 Phrases in Learningset and 64 Phrases
in Validationset (round about 1% of the total phrase amount)
- memory-factor LAMBDA=0.2, Learnginrate ETA=0.01, Learningcycles = 1000.
- Elman-Net
- 2 hidden layer
- 50 neurons per layer
On-Road-Test for Active- und Passivephrases
----------------------------------------
Test for all 64 phrases of the validationset
Is is essential: Validationpattern notsubset Learningset !
[TABLE]
- variant role-allocations are learned and abstracted "very well" (>90%)
- invariant role-allocations are learned "perfect", because special words
only appear with a special semantic role (yesterday, today, tomorrow)
Lerning-process for ambiguous Phrases
------------------------------------
Motivation: Determining the semantic role for the last phrase-part-
We learn only phrases of the following kind:
AGENT enchain PATIENT with the chain (instrument).
AGENT enchain PATIENT with the binoculars (attribute).
AGENT observe PATIENT with the chain (attribute).
AGENT observe PATIENT with the binoculars (instrument).
AND
AGENT look at PATIENT with the binoculars (instrument).
Interesting question: How act the neural net with the not learned phrase
AGENT look at PATIENT with the chain (???)
On-Road-Test for ambiguous Phrases
------------------------------------
- Test only with Phrases of kind (37 phrases)
AGENT look at PATIENT with the chain (not learned!)
- What semantic role is allocated to the phrase-part "with the chain" ?
(instrument or attribute)
Conclusion:
- The net has lerned the alternating relation between instrument and attribute
("Chain" and "Binoculars")
- The words "look at" are never learned in relation of attribute !
- Special keywords now behave like operands !