Your browser does not allow JavaScript!
JavaScript is necessary for the proper functioning of this website. Please enable JavaScript or use a modern browser.
|
|
SLO
|
ENG
|
Cookies and privacy
DKUM
EPF - Faculty of Business and Economics
FE - Faculty of Energy Technology
FERI - Faculty of Electrical Engineering and Computer Science
FF - Faculty of Arts
FGPA - Faculty of Civil Engineering, Transportation Engineering and Architecture
FKBV - Faculty of Agriculture and Life Sciences
FKKT - Faculty of Chemistry and Chemical Engineering
FL - Faculty of Logistic
FNM - Faculty of Natural Sciences and Mathematics
FOV - Faculty of Organizational Sciences in Kranj
FS - Faculty of Mechanical Engineering
FT - Faculty of Tourism
FVV - Faculty of Criminal Justice and Security
FZV - Faculty of Health Sciences
MF - Faculty of Medicine
PEF - Faculty of Education
PF - Faculty of Law
UKM - University of Maribor Library
UM - University of Maribor
UZUM - University of Maribor Press
COBISS
Faculty of Business and Economic, Maribor
Faculty of Agriculture and Life Sciences, Maribor
Faculty of Logistics, Celje, Krško
Faculty of Organizational Sciences, Kranj
Faculty of Criminal Justice and Security, Ljubljana
Faculty of Health Sciences
Library of Technical Faculties, Maribor
Faculty of Medicine, Maribor
Miklošič Library FPNM, Maribor
Faculty of Law, Maribor
University of Maribor Library
Bigger font
|
Smaller font
Introduction
Search
Browsing
Upload document
Statistics
Login
First page
>
Show document
Show document
Title:
Modelling highly inflected languages
Authors:
ID
Sepesy Maučec, Mirjam
(
Author
)
ID
Kačič, Zdravko
(
Author
)
ID
Horvat, Bogomir
(
Author
)
Files:
http://dx.doi.org/10.1016/j.ins.2003.12.004
Language:
English
Work type:
Unknown
Typology:
1.01 - Original Scientific Article
Organization:
FERI - Faculty of Electrical Engineering and Computer Science
Abstract:
Statistical language models encapsulate varied information, both grammatical and semantic, present in a language. This paper investigates various techniques for overcoming the difficulties in modelling highly inflected languages. The main problem is a large set of different words. We propose to model the grammatical and semantic information of words separately by splitting them into stems and endings. All the information is handled within a data-driven formalism. Grammatical information is well modelled by using short-term dependencies. This article is primarily concerned with the modelling of semantic information diffused through the entire text. It is presumed that the language being modelled is homogeneous in topic. The training corpus, which is very topically heterogeneous, is divided into three semantic levels based on topic similarity with the target environment text. Text on each semantic level is used as training text for one component of a mixture model. A document is defined as a basic unit of a training corpus, which is semantically homogeneous. The similarity of topic between a document and a collection of target environment texts is determined by the cosine vector similarity function and TFIDF weighting heuristic. The crucial question in the case of highly inflected languages is how to define terms. Terms are defined as clusters of words. Clustering is based on approximate string matching. We experimented with Levenshtein distance and Ratcliff/Obershelp similarity measure, both in combination with ending-stripping. Experiments on the Slovenian language were performed on a corpus of VEČER newswire text. The results show a significant reduction in OOV rate and perplexity.
Year of publishing:
2004
PID:
20.500.12556/DKUM-27499
UDC:
004.93
ISSN on article:
0020-0255
COBISS.SI-ID:
8504086
NUK URN:
URN:SI:UM:DK:HOEPIJTE
Publication date in DKUM:
01.06.2012
Views:
1532
Downloads:
53
Metadata:
Categories:
Misc.
Cite this work
Plain text
BibTeX
EndNote XML
EndNote/Refer
RIS
ABNT
ACM Ref
AMA
APA
Chicago 17th Author-Date
Harvard
IEEE
ISO 690
MLA
Vancouver
:
Copy citation
Average score:
(0 votes)
Your score:
Voting is allowed only for
logged in
users.
Share:
Hover the mouse pointer over a document title to show the abstract or click on the title to get all document metadata.
Record is a part of a journal
Title:
Information sciences
Shortened title:
Inf. sci.
Publisher:
North-Holland
ISSN:
0020-0255
COBISS.SI-ID:
25613056
Secondary language
Language:
English
Keywords:
jezikovni modeli
,
slovenski jezik
Comments
Leave comment
You must
log in
to leave a comment.
Comments (0)
0 - 0 / 0
There are no comments!
Back
