In: Antonio Zampolli (ed., 1984): Studies in Honour of Roberto Busa. Linguistica Computazionale, Vol. IV und V., S. 343-359

MACHINE-AIDED INDEXING OF TEXT CORPORA*

HARALD H. ZIMMERMANN

When the first volumes of the computer-aided indices of German literature1 appeared in the sixties, linguistic dataprocessing with the help of the computer was still in its infancy. The same could be said (and it holds true to some extent even today) regarding the preparation of more or less representative text corpora2, such as the LIMAS corpus3, the news corpus of Lund4 and various corpora of the Institute for the German Language5.
What played a major role in the case of these initial attempts at computer-aided text processing were the basic functions of EDP devices pertaining to computation and collation, namely the capacity to sort words, or rather, word forms, (for instance in alphabetical order, spelt forward and backward) to order them according to their frequency (frequency dictionary); and sometimes also the formal function of indicating the stored data in a given context (line or sentence). More often than not such results of text analysis by computer were viewed as end products, and were presented in this raw form to potential users, especially to philologists and linguists, and this is still being done today.
It is, of course, also true of text dictionaries that one should not put everything in a corpus-oriented dictionary of this type that can be accomplished with its help. It is, moreover, true that one cannot foresee all the questions which may be raised in connection with an element or portion of the text.
On the other hand, it cannot be considered as an achievement to have discovered the computer as a substitute for a card-index box: Kaeding6, for example, has processed running texts of nearly 11 million words without the aid of a computer. This material, even today, is useful for many complementary (intellectual as well as machine-aided) evaluations. Reference may be made in this connection to the linguistic statistics of Meier7, and the investigations by the Goethe Institute which make use of the Kaeding material8.
The demands on the computer should, therefore, definitely be somewhat higher in connection with collections of linguistic data. This can especially be done within the field of morphosyntax. Therefore, as early as 1970, in a case of text processing of this kind - namely the indexing of the works of the Austrian poet Georg Trakl - the expectations from a computer-aided product were higher than was usual for literary indices9. In order to provide the possibility of better use, especially by philologists, the following tasks should be accomplished:

1. Morphological «lemmatization» of the word-forms, i.e. their reduction to basic forms
2. Decomposition of compound word-forms into meaningful segments 
3. Determination of derivations (especially suffixes) 
4. Classification of documents according to syntactic categories (word-classes)
5. Semantic disambiguation

1. Lemmatization
By lemmatization in the broadest sense one generally understands the categorisation of word-forms according to key-words which represent them respectively, and which can morphologically, syntactically and semantically be traced back to the same characteristics, for instance one stem morpheme, one part of speech and similar characteristics of meaning10. This definition, if used strictly, would lead in some cases to problems (e.g. in the case of WAR and BIST, morphologically diverging forms of the German verb SEIN): on the other hand, however, it has not been possible till now to find semantic characteristics for sufficient differentiation in meaning which would stand up to every test. Finally, even in the syntactical field, the differentiation of parts of speech is not always to be determined empirically (for instance by means of distribution analyses); on the other hand, clumbing together or establishment of relations between word-forms over and above a particular category could be useful (e.g. SINGEN-GESANG; LIEBE-LIEBLING/ LIEB). The differentiation or clumbing together therefore always represents one of many possible ways of viewing the basic material in a particular form (for example, a «lemmatized index» in book-form), whereby the aspect of suitability of use and usefulness receives prime consideration.

2. Decomposition of Compound Words
In the German language compound words appear very frequently in texts. It is just a question of particular writing habits, namely that of writing in one word, as the alternative forms of the so-called «instant compounds» show (AMERIKAREISE - REISE NACH AMERIKA, MINISTERBESUCH - BESUCH DES MINISTERS). However, various possibilities of combining words within the context should not be overlooked. On the other hand, expressions comprising several words, such as JURISTISCHE PERSON which can be seen as one unit of meaning, show that writing together alone is not an indication for a change of meaning, and vice versa.

Therefore, it is useful in the interests of the user to decompose the compounds into their constituent elements, and to make use of them in an index or a register in an appropriate form of representation, at least in all those cases in which the elements of a compound are to some extent self-contained; or, perhaps, already in the case of rather formal decomposition, as in several cases, it is difficult to make a strict distinction.

3. Derivations
In many languages there are certain word derivations (mostly morphologically characterized) which make it seem a good idea to place the corresponding derivation variants together in relation to a core element. By the same token one can link the derivation element itself (above all suffixes) with the basis element, in order to facilitate assessments on frequency, category, etc., in a general linguistic corpus or in an author-related text.
The aims mentioned under (2) and (3) led to the fact that, at the initial stages of corpus research, simple wordform lists were prepared in which the words were spelt backwards. These lists were capable of satisfying almost all queries in this regard.

4. Wordclass Labelling
The labelling of word-class entries in a text collection is on the one hand an important instrument for separating the rather functional elements in a language (functional words, particles - such as conjunctions, articles, prepositions) from the elements which convey meaning (nouns, adjectives, verbs, adverbs). This makes it possible, for example, to restrict oneself to these meaning-conveying words when dealing with line or sentence concordances, since these words as a rule will be the centre of further research.
At the same time, a more clearly differentiated labelling of word-classes in several cases allows a (partial) semantic disambiguation of word-forms, as far as different word-classes are concerned (e.g. KREUZE (verb) derived from KREUZEN as against KREUZE (noun) derived from KREUZ; WAGEN (verb) as opposed to WAGEN (noun); LAUTE (adjective) derived from LAUT as opposed to LAUTE (verb) from LAUTEN; LAUTEN or LAUTE (noun) from (die) LAUTE/ (der) LAUT...).
Finally, the labelling of word-classes can form the basis for far-reaching evaluation of text material, such as for the frequency of use of the definite article as opposed to the indefinite, for the structure of noun groups used, etc.

5. Semantic Disambiguation
In the case of heterogeneous texts with different themes and also of larger text collections, a distinction between the meanings of text-words where ambiguities occur is sensible for reasons of practicality.
This leads to two fundamental problems:
•	How far must the meanings be differentiated? In several cases there occurs an additional partial problem, as besides (completely?) different meanings, semantic family and semantic hierarchies can also be established. Thus text collections with diverging semantic differentiation can no longer be comparable. In such cases, therefore, at least a general standard (or simply a point of reference), for example a reference to a particular lexicon, should be given, which permits a user to comprehend the differentiation. Further, a formal access via an undifferentiated word should be made possible, as can be found in traditional lexica.
•	How far is it possible to find out the «correct meaning» in any particular case? Very often the context (for instance of a newspaper article or even of a poem) is not enough, and one inclines in such a situation, even if in rare cases, to interpret (which should actually be left to the user of a product). Therefore, if a semantic disambiguation is undertaken, the doubtful cases must be made recognizable as being such.

In the following it will be undertaken to demonstrate, with the help of the lemmatized index for the complete works of the Austrian poet Georg Trakl, as to how one can achieve appreciable results economically with the most basic use of a computer.
In the preparation of the lemmatized index for the works of Georg Trakl11, the computer was as usual made use of purely as sorting and storing instrument, as, at that time, further developed computer functions were not available12.

1) First of all the complete text was put on 5-channel punched tapes, whereby every page of the critical edition13 was marked accordingly, and the text was structured according to the line numbers as given by the publisher. With the help of this, later on references to page and line numbers were produced in the index.
2) Only in the case of substantives was the word-class explicitly marked by means of a capitalization characteristic. When an ambiguity cropped up in the case of other word-classes, a differentiating mark was likewise used.
3) Thereafter an alphabetically sorted word-form list was produced by the machine, and punched onto punched cards. The references were output onto reference cards which were corrected with the word-form (punched) cards (= word cards) by means of an identification number.
4) By means of a sorting machine, the word cards were separated mechanically from the reference cards, and the substantives from the remaining word-forms for which a corresponding marking was already there.
5) The word-cards were subsequently printed with the help of an automatic inscriber. With this the contents were made legible. These word-form cards, excepting, of course, the substantives, were divided intellectually into 3 groups: Particles («P», i.e. functional words such as DER, EIN, ODER...), inflected and uninflected adjectives («A») and verbs («V», including the participles). Subsequently, these three piles of cards were automatically given a corresponding word-class marking by means of a card-puncher.
6) The morphological lemmatization (basic form assignment) was to be undertaken in relation to word-class (only the particles remained as word-forms). For this purpose, all the word-forms in which the word and the basic form were identical were taken out manually and subsequently put together automatically in the place already left for this purpose. Others were put together with their basic forms by hand (this had to be done in approximately 2,000 different cases).
7) The basic forms (lemmata) thus received were then sorted by the computer in backward spelling order. Components of compounds and derivation elements for the relevant entries which were detected by simple comparison on specific reference element cards were listed intellectually; the same was true of suffixes.
8) The initial data were then mixed by means of a sorting machine in the order: basic form - word-form - reference instances; subsequently the reference cards were intellectually / manually ordered.
9) Thereafter, the print format for the index was established with the help of a controlling and paging program. Simultaneously, the control symbols were produced, which made it possible to transfer the output, which was stored on magnetic tape, onto a film by photosetting, for production in book form (cf. Figure 1).
10) With the help of a further mechanical sorting program, a frequency dictionary for the entire data was produced, on the basis of the basic forms. The consideration of the word-classes in printed form showed that the «world» of Georg Trakl and his epoque, as symbolically expressed in the most frequent adjectives, was: dunkel (dark), blau (blue), schwarz (black), leise (low-voiced), still (quiet) .... (cf. Figure 2).

In order to complete phases (3) to (10), about one man-month was necessary. With this it became apparent that an appropriate working concept can lead to economically acceptable, and at the same time appreciable results, even with very limited resources.
Since the completion of this work more than ten years have passed. The unhandy punched tapes and punched cards with their limited stock of characters have in the meantime been replaced by terminals with the capacity for capital and small letters. In some research institutes, for instance at the University of Saarland, there are electronic procedures applicable at least in model form, which make it possible to use the computer not as a sorting instrument, but rather to use its «intelligence» for solving the above-mentioned problems. Such an «intelligent» system has been developed in Saarbrücken in the form of the «Saarbrücken method for automatic text analysis» which can produce from any kind of German language text morphologically, syntactically and semantically differentiated words (lemmata)14 .
As early as in the sixties several research programs were already being pursued under the supervision of Hans Eggers15. 
In the meantime a practicable version of this system has been developed, and has been used for investigations in several cases.

Figure 1 - Alphabetical Index for Georg Trakl's poetry.
Example page 90.

Figure 2 - Frequency Index for Georg Trakl's poetry.
Example page 169.
However, it was not considered sensible to test the system on the «extreme case» of belles-lettres or poetry. It appeared more feasible to use texts written in everyday prose. The work with these texts can also be used for other kinds of text and can be projected onto a more practical use than would be the case in processing a poetical work.
Such a practical use is apparent in the field of specialized information and documentation. The main objective hereby is to index the relevant documents (for instance newspaper articles, texts of judgements, regulations, minutes, patent descriptions, etc.) to such an extent as to be able to retrieve them with the help of the words which appeared in the text. In other words, this can be termed as a type of «automatic indexing».
As it is apparent that the stored data will be quite voluminous, and in certain cases a data-bank or information bank would also contain documents of a heterogeneous nature, the problem of semantic differentiation does gain in importance. (In processing the works of Georg Trakl, this could safely be left aside).
The automatic indexing system which has been developed in Saarbrücken on the basis of the experience gained from the above project comprises all the above-mentioned aspects, i.e.:
- Morphosyntactic lemmatization
- Decomposition 
- Derivation
- Categorization according to word-class 
- Semantic disambiguation
In certain aspects the system surpasses the above-mentioned framework of questions. For instance, even multiword expressions (such as JURISTISCHE PERSON) can be identified. In addition, between the disambiguated or lexically unambiguous elements a series of semantic relations (synonyms, generic and generated terms, etc.) is established, which is very useful for the process of retrieval. Finally, the syntactic relations which appear in the text (e.g. adjective-substantive, substantive and coordinated substantive) are made available in direct relation (as so-called complex descriptors) for retrieval.
In the following, the basic procedural steps of the system are described briefly16:

1) As the first step, the input text is split into individual words which are then «looked up» in a morphosyntactical dictionary via a standardized input interface. Every word-form is then supplied with various pieces of information necessary for further syntactic analysis, and at the same time the basic form is also determined, wherever possible. In the case of those word compounds and derivations which are not found in the general morphosyntactical lexicon, a decomposition or derivation analysis is carried out. In this way, the orthographic mistakes can also be detected at the same time. As our morphosyntactic lexicon for the German language in the meantime has over 142,000 entries of basic forms, the probability of a non-identified token being misspelt is fairly large (cf. Figures 3a and 3b).

... akademische Grade, die Anschrift sowie auf eine Angabe über die Zugehörigkelt Zugehörigkeit Betroffenen zu dieser Personengruppe beschränkt und kein Grund zur Annahme besteht, daft dadurch schutzwürdige Belange des Betroffenen beeinträchtigt werden.
§ 33
Datenveränderung

Das Verändern personenbezogener Daten ist zulässig, soweit dadurch schutzwürdige Belange des Betroffenen nicht beeinträchtigt werden.
§ 34
Auskunft an den Betroffenen
(1) Werden erstmals zur Person des Betroffenen ...
Figure 3a - Example of reduction of text for federal data protection law (BDSE$33).

2) The dictionary check, as elaborated above, produces the potential word-classes and the basic forms. Thereafter, the various relevant functions are established on the basis of a sentence or a context-oriented mechanical syntactic analysis. This part of the procedure fulfils the function of determining word-classes and also that of lemmatization, in so far as the potential categories from the lexicon are reduced to the actual ones in the context (cf. Figure 4).

3) The next step is to identify multi-word expressions and to undertake semantic disambiguation as far as possible on the basis of the sentence-related context. This is done with the help of  a semantic lexicon, which, inter alia, contains rules for determining multi-word or inflected expressions as well as characteristics and rules for semantic disambiguation (cf. figure 5).

4) In the next step, it is attempted to remove all remaining ambiguities on the basis of statistical data (probability of the use of a word in a particular meaning in a specialized subject area), whereby the semantic relations as given in a technical lexicon are made use of, going beyond the sentence-based context.
5) The last step is a procedure which processes the information in such a way as to make it possible to produce descriptors which can then be stored in a data bank or put in the form of entries which can be put in a register (cf. Figure 6 regarding the generation of descriptors; Figure 7 shows an example of access to a data bank).

The system was developed as a model. In a laboratory application a running text of a little more than 100,000 words from the area of data production was processed, and it could well be established that automatic indexation of text is practicable. During further intensive research at the University of Saarland the system was modified and optimized17. The system has been tested in several pilot applications. On the whole, running texts of more than 3 million words from various specialized subject areas (especially from the German Patent Bureau) have been successfully processed and indexed.
Viewing all this, one tends to realize that simple procedures like the generation of word form indices, corresponding frequency lists of KWIC/KWOC concordances seem to have only historical value. The methodology for processing text corpora has entered a phase which could be termed as the «second generation» of text corpora indexation. Any work on text corpora, therefore, should be seen in light of the fact that the «instrument» computer has to be optimized and made more effective by developing appropriate lexica and rules for data-processing to such an extent, that it could easily make use of modern indexing systems as described above, and barring, of course, the cases in which an immediate ad hoc processing of the texts becomes essential.

« »
*. Revised version of a lecture delivered at the symposium “Computer corpus of the Serbo-Croatian language”, Belgrade, 14-18 December, 1981. 

Anmerkungen:
1. Serie «Indices zur deutschen Literatur», edited by H. SCHWERTE and H. SCHANZE, Athenaeum, Frankfurt.
2. For the concept of corpus, cf. various contributions in the volume Empirische Textwissenschaft. und Auswertung von Text-Corpora.  Edited by H. BERGENHOLTZ, B. SCHAEDER, Konigstein/Ts., 1979, also including the article by B. RIEGER (pp. 52-70).
3. Microfiche edition: MCS-Verlag, Nürnberg, 1979. Series: Regensburger Materialien auf Microfiche (RMM). For the structure of the LIMAS corpus, cf. R. GLAS: Das LIMAS- Korpus, ein Textkorpus für die deutsche Gegenwartssprache. Linguistische Berichte 40 (1975), pp. 63-66.
4. Cf. I. ROSENGREN, Ein Frequenzwörterbuch der modernen Zeitungssprache - wie und wozu? Beiträge zur Linguistik und Informationsverarbeitung 14 (1968), pp. 7-21.
5. For the corpora of the Institute of the German Language, cf. B. SCHAEDER: Das Bonner Zeitungskorpus: Eine maschinelle Dokumentation von Tageszeitungen der BRD und der DDR. Mimeo, Bonn 1978, and U. ENGEL: Das Mannheimer Corpus, Forschungsberichte des Instituts für Deutsche Sprache 2, Mannheim, 1969, pp. 75-84.
6. F.W. KAEDING (ed.), Häufigkeitswörterbuch der Deutschen Sprache. Festgestellt durch einen Arbeitsausschuss der deutschen Stenographiesysteme. Steglitz bei Berlin, 1898. A detailed description can be found in: W.B. ORTMANN (ed.), Hochfrequente deutsche Wortformen I, Munich, 1975, pp. 5-26.
7. H. MEIER, Deutsche Sprachstatistik. Hildesheim 1964. Investigations by this «idealistic lone-wolf» (according to ORTMANN, p. 27) are based on the Kaeding material, whereby the Kaeding material was processed and modified in over 40 years of work done in his free-time.
8. Machine-aided evaluation of the Kaeding material by the section for scientific didactics of the Goethe Institute by W. D. ORTMANN has since led to a series of publications. These are of use above all in phonological studies.
9. W. KLEIN, H. ZIMMERMANN: Index zu Georg Trakl. Dichtungen. Frankfurt, 1971.
10. For a formal definition of lemma» in this extended sense, cf. R. DIETRICH: Automatische Textwörterbücher. Studien zur maschinellen Lemmatisierung verbaler Wortformen des Deutschen. In: H.E. BREKLE et al. (eds.), Linguistische Arbeiten 2, Tübingen 1973, esp. pp. 1f. Dietrich relies on the definition by H. D. MAAS: Homographie und maschinelle Sprachübersetzung, in Linguistische Arbeiten des Germanistischen Instituts und des Instituts für Angewandte Mathematik der Universität des Saarlandes, No. 8, Saarbrücken, 1969.
11. Cf. footnote 9. Technical instruments used were: Computer of the type Philips Electrologica X1; a CDC2200 computer and punched card sorting machines.
12. As this method, in spite of the fact that certain changes have taken place in hardware technology in the meantime, still appears to be economical, inter alia for languages in which no procedures for automatic analysis exist, it is described in short.
13. Georg Trakl. Dichtungen und Briefe. Historical-critical edition. Ed. W. KILLY and H. SZKLENAR. Salzburg, 1969 (2 volumes). The index does not include the letters. This was due to the counting of frequency. Cf. also the preface to the index.
14. For a description of the method, cf.: SALEM Ein Verfahren zur automatischen Lemmatisierung deutscher Texte. Ed.: Sonderforschungsbereich 100 «Elektronische Sprachforschung», Projektbereich A. Tübingen 1980.
15. The most important result of this early research is: H. EGGERS et al., Elektronische Syntaxanalyse der deutschen Gegenwartssprache, Tübingen, 1969.
16. The method is described in: H.H. ZiMMERMANN: Ansätze einer realistischen automatischen Indexierung unter Verwendung linguistischer Verfahren. In: R. KUHLEN (ed.), Datenbasen, Datenbanken, Netzwerk. Vol. 1, München, 1979, pp. 311-338. Also: Bürgernahe Informationsvermittlung am Beispiel des Modellsystems «Juristische Dokumentanalyse im Bereich Datenschutz» (JUDO-DS), edited by the same. In: Österreichische Gesellschaft für Informatik (ed.): Informationssysteme für die 80-er Jahre (Fachtagung 1980), vol. 1, Linz 1980, pp. 143-168.
17. For a detailed description, cf. H.H. ZIMMERMANN, E. KROUPA, G. KEIL et al., CTX - Ein Verfahren zur Computer-gestützten Texterschließung.