next up previous contents
Next: Contents

Automatical Lexicon Extraction from Aligned Bilingual Corpora



Diploma Thesis


Jörg Tiedemann
Student of Computer Science
Register 1991
Otto-von-Guericke-Universität Magdeburg

tiedeman@sunpool.cs.uni-magdeburg.de


Otto-von-Guericke-Universität Magdeburg
Department of Computer Science
Institution of Information and Communication Systems
Diploma Thesis:
Automatical Lexicon Extraction from Aligned Bilingual Corpora






























Author: Jörg Tiedemann
Supervised by: Erik F. Tjong Kim Sang
  (University Uppsala)
  Prof. Dr. Dietmar Rösner
  (University Magdeburg)
Start of the work: April 01, 1997
End of the work: August 31, 1997
Statistics: 102 pages, 27 figures, 19 tables

Statement by the Author

This thesis is the description of my work in the bilingual lexicon extraction project at the department of Linguistics/Computational Linguistics at the university in Uppsala/Sweden in the summer semester 1997.

The author declares with this statement that this report was completely written by himself. Citations are marked with corresponding reference links. Used papers, literature and other information collections are listed in the bibliography.

This thesis will be deposited at the university library in Magdeburg and will be available under rules of the library. Brief quotations from this thesis are allowable if accurate acknowledgement is provided. Extended quotations and reproductions require the permission of the author or the department of computer science at the ``Otto-von-Guericke-Universität'' in Magdeburg and serve non-commercial use only.







Magdeburg, September 7, 2000 Jörg Tiedemann

Acknowledgement

With this I wish to acknowledge the assistance provided by Erik F. Tjong Kim Sang, who supervised my project at the University of Uppsala. It was an encouraging and pleasant cooperation in the style of good acquaintance more than a typical teacher-student relation. Thanks also to Anna Sågvall Hein for providing interesting suggestions to start my work with. Further, I wish to acknowledge Prof. Dietmar Rösner for the support of my work at the University in Magdeburg. Thanks to Erich Round who polished my English and corrected some of my typical German translations, to Brigitte Grote for the many suggestions and recommendations to improve content and writing style of this thesis, and to Jolene R. Huberty for the tough fight not only with my innumerable commata mistakes. Thanks to everybody in the department of computational linguistics in Uppsala for their patience when I took all the performance from the computer system by counting masses of word pair frequencies. Acknowledgements to the linguistic students who tried to teach me Swedish constantly (unfortunately with only few success). Thanks to my boring corridor mates in Flogsta who never stopped me from working on my project, and also to many other students, especially exchange students, in Uppsala for the motivating discussions about my topic.

Abstract:

There are many research projects for automatical extraction of bilingual lexical knowledge from parallel corpora in the field of information retrieval from large scale text corpora. In this thesis a number of well-known techniques are discussed and evaluated. Several new approaches for bilingual lexicon extraction, which take into account the specific characteristics of the corpus under consideration, were developed and applied.

An alignment reduction method using a given basic bilingual dictionary was applied to obtain new one-to-one word alignments which were considered as new translation pairs. An algorithm for the identification of slightly modified new translations of source expressions stored already in the basic dictionary is presented.

Another approach employs simple similarity measures to find cross-language cognates. Several combinations and evaluations of these similarity scores are described. This method produces many accurate lexical data especially for closely related languages like Swedish and German.

Other techniques make use of frequency counts to find corresponding words in the alignments. One approach applies the Dice coefficient for measuring the correlation between word pairs. This produces good results for the Swedish/German alignments but leads to many incomplete translations for Swedish/English because of the different usage of compounds in these languages.

An algorithm for identification of compounds with separated word translations using similarity measures and bilingual dictionary data is presented.

The last approach uses part-of-speech tags to extract one-to-one word translations. This technique aligns words with identical part-of-speech tags with each other and uses the resulting alignments for lexical extractions.

The different methods can be applied independently of each other to create rough bilingual dictionaries. The differences in their behaviour, dependent on the language pairs, were evaluated and experimental results are presented.

Furthermore, algorithms for automatic dictionary evaluation in the form of adjustable and cascadable filters were developed. These filters focus especially on dictionary entries with multiple translations to identify and exclude incorrect translation pairs using different heuristics and measures.



 
next up previous contents
Next: Contents
Jörg Tiedemann
2000-09-07