Merkkijonoista suomen kielen sanoiksi

Abstract

Suurten tekstiaineistojen tallentaminen ja arkistoiminen elektronisesti on tullut yhä edullisemmaksi. Samalla perinteiset tekstin tallennus- ja hakumenetelmät ovat alkaneet jäädä tehottomiksi.

Tallennus- ja hakutulosten voidaan olettaa paranevan, kun käytetään menetelmiä, jotka ottavat kunkin kielen erityisominaisuudet huomioon. Tutkimuksessa selvitettiin, miten suomen kielen morfologisten tulkintaohjelmien eli sanoja tunnistavien ja muokkaavien tietokoneohjelmien avulla voidaan ratkaista ongelmia, jotka johtuvat suomen kielen erityispiirteistä. Morfologisten tulkintaohjelmien avulla voidaan muun muassa toteuttaa hakujärjestelmä, jossa otetaan huomioon suomen sanojen taipuminen ja jaetaan yhdyssanat osiinsa.

Tutkimus oli luonteeltaan laboratorioympäristössä toteutettu evaluointitutkimus. Siinä rakennettiin erityinen testausympäristö, jossa samasta tekstiaineistosta tuotettiin joukko erilaisia tietokantoja. Aineistona oli otos suomalaisesta sanomalehtiarkistosta.

Kun hakujärjestelmän hakemistoon tallennettavat sanat normalisoitiin eli kaikki taivutusmuotoiset sanat palautettiin perusmuotoonsa, ts. sanakirjamuotoon (esimerkiksi: kaupoissa -> kauppa), hakemisto tarvitsi vähemmän tietokoneen muistitilaa kuin vastaava, normalisoimaton hakemisto.

Kun tekstin sanat morfologisilla tulkintaohjelmilla palautettiin perusmuotoon ja perusmuodot tallennettiin hakemistoon, tiedonhakujen tulokset olivat tarkempia kuin silloin, kun sanoja ei ollut perusmuotoistettu. Toisaalta havaittiin, että on tärkeää hakea myös perusmuotojen rinnakkaisilmaukset, kuten johdokset (vero -> verottaminen, verotus) ja yhdyssanat (vero -> autovero), jotta olennaista tietoa ei jäisi löytymättä.

Tutkimuksen perusteella sanojen perusmuotoistaminen kannattaa, koska perusmuotohakemistosta tehtyjen tiedonhakujen tulokset ovat yhtä hyvät tai paremmatkin kuin silloin, kun tulkintaohjelmia ei ole käytetty. Hakemista helpottaa, ettei hakijan tarvitse välittää sanojen taipumisesta. Toisaalta sanojen perusmuotoistaminen tekee mahdolliseksi hyödyntää uusia tallennus- ja hakumenetelmiä, joita on kehitetty muissa kielissä, joissa sanojen taipuminen ja yhdyssanojen esiintyminen on vähäisempää kuin suomessa.

This thesis deals with linguistic processing and retrieval techniques in Finnish fulltext databases, and especially the special problems due to the characteristics of the Finnish language.

In the research project, natural language analysis modules for Finnish were incorporated into the commercial BASIS information retrieval system, which is based on inverted files and Boolean searching. Several test databases were produced, each using one or two Finnish morphological analysis modules. Thus the work has been carried out within the traditional laboratory environment paradigm, presenting various test parameters and their evaluation. The test databases were as follows: T1) Traditional database: the inflected word forms were stored as such in the index and the searcher truncated search terms manually T2) Automatic stemming: traditional index, but a language module was used to stem the search terms (linguistic stemming using the base form as input) T3) Sifting the results of automatic stemming: As above, but the character strings that matched stemmed strings were further analyzed. Only strings that were variants of the original search term were accepted; false drops having only the beginning of the word identical with the search string were rejected. T4) Base word forms: The inflected word forms were reduced to their base form before storing the words in the index. The searcher used the base form as a search term. When compound words were searched, the basic word form was fed into the automatic stemming module (as in T2 above, but using a different index) T5) Base word forms + split compounds: The inflected word forms were reduced to their basic form and compound words were broken down into their components before storing the words in the index. The basic word form was used as a search term. Compound words were searched by adding a special "compound wild card" symbol. T6) Double index: consisted of both the inflected index and the base word form index. (The purpose was to test how to recover from errors made in base word form generation.)

When word forms were normalized to their base form, the memory size of the base form index file was smaller than the size of the inflected word form index. Even when the compounds were broken down into their components, the base form index was smaller than the inflected form index. When automatic stemming was used in the different databases, the precision figures were higher when the searches were conducted in the base form index compared to searching with the same search stems in the inflected word form index. This indicates that a base form index is in-herently more precise than an inflected word form index. When searching in base form index, it is not practical to use only the base form of the original search term. When a simple query consisting of only the original search term in its base form is conducted, the recall ratio will collapse, compared to the corresponding search with truncated search terms in the inflected word form index. The recall ratio becomes better when such a simple query is augmented with the derivatives and/or compound words related to the search term. Although the precision ratio simultaneously will decrease, the loss in precision will be smaller than the profit in recall. Consequently, derivatives and compounds are of importance when search-ing in base word form index.

The best recall ratio was achieved in the index containing the base word forms (T4) and the one containing the base forms as well as the components of compound words (T5). Differences from other databases were systematic, but usually not statistically significant.

The best precision ratio was achieved when the search terms were sifted (T3) and when using the base word form index (T4). Sifting, however, obtained the poorest recall ratio, and the differences from other databases were also statistically significant.

Problems in using natural language modules were also investigated. One problem is that language modules may provide a false reading of an inflected word form. This may result in information loss (the correct word form will not be in the index) or information overflow (all possible readings of homonymic word forms must be stored to index, which increases the amount of index terms). Some simple methods for transforming improper search terms to appropriate base forms were tested.