20.500.12556/DKUM-50411 O avtomatski evalvaciji strojnega prevajanja On automatic machine translation evaluation Stalen del razvoja strojnega prevajanja je evalvacija prevodov, pri čemer se v glavnem uporabljajo avtomatski postopki. Ti vedno temeljijo na referenčnem prevodu. V tem prispevku pokažemo, kako zelo različni so lahko referenčni prevodi za področje podnaslavljanja ter kako lahko to vpliva na oceno – ista metrika lahko isti prevajalnik oceni kot neuporaben ali kot zelo uspešen samo na podlagi tega, da uporabimo referenčne prevode, ki so pridobljeni po različnih postopkih, vendar vedno jezikovno in pomensko povsem ustrezni. An important task of developing machine translation (MT) is evaluating system performance. Automatic measures are most commonly used for this task, as manual evaluation is time-consuming and costly. However, to perform an objective evaluation is not a trivial task. Automatic measures, such as BLEU, TER, NIST, METEOR etc., have their own weaknesses, while manual evaluations are also problematic since they are always to some extent subjective. In this paper we test the influence of a test set on the results of automatic MT evaluation for the subtitling domain. Translating subtitles is a rather specific task for MT, since subtitles are a sort of summarization of spoken text rather than a direct translation of (written) text. Additional problem when translating language pair that does not include English, in our example Slovene-Serbian, is that commonly the translations are done from English to Serbian and from English to Slovenian, and not directly, since most of the TV production is originally filmed in English. All this poses additional challenges to MT and consequently to MT evaluation. Automatic evaluation is based on a reference translation, which is usually taken from an existing parallel corpus and marked as a test set. In our experiments, we compare the evaluation results for the same MT system output using three types of test set. In the first round, the test set are 4000 subtitles from the parallel corpus of subtitles SUMAT. These subtitles are not direct translations from Serbian to Slovene or vice versa, but are based on an English original. In the second round, the test set are 1000 subtitles randomly extracted from the first test set and translated anew, from Serbian to Slovenian, based solely on the Serbian written subtitles. In the third round, the test set are the same 1000 subtitles, however this time the Slovene translations were obtained by manually correcting the Slovene MT outputs so that they are correct translations of the Serbian subtitles. The results of MT evaluation were calculated for the metrics NIST, BLEU and TER. They were strikingly diverse, even though the system output was always the same: when calculated on the original translations from the parallel corpus, BLEU was 19.47%, TER 65.27% and NIST 5.05; when calculated on directly translated subtitles from Serbian to Slovenian, BLEU was 43.10%, TER 32.91% and NIST 7.78; when calculated on the manually corrected MT output, BLEU (also so-called hBLEU) was 71.6%, (h)TER 14.1% and (h)NIST 10.62. strojno prevajanje vrednotenje evalvacija referenčni prevod BLEU TER machine translating evaluation reference translation BLEU TER true false true Slovenski jezik Angleški jezik Znanstveno delo 2015-07-10 12:39:35 2015-07-10 12:39:35 2022-05-29 03:06:56 2021-05-09 03:27:51 2013 0 0 str. 111-133 št. 1 Letn. 1 2013 0000-00-00 Zaloznikova Objavljeno NiDoloceno 0000-00-00 0000-00-00 0000-00-00 2335-2736 81'322.4 264547328 16892438 2335-2736 URN:SI:UM:DK:PVG9XPPX Slovenscina_2.0_2013_Verdonik,_Sepesy_Maucec_O_avtomatski_evalvaciji_strojnega_prevajanja.pdf Slovenscina_2.0_2013_Verdonik,_Sepesy_Maucec_O_avtomatski_evalvaciji_strojnega_prevajanja.pdf 1 A301A4206287D262F591AC7FC4C5E415 b298a99df14ec3c7b73effcababd6d6a58b11c744587f2c6f845336458a9bd77 7812da7e-7c0e-11eb-bb7a-00155d0001ca https://dk.um.si/Dokument.php?lang=slv&id=114760 http://slovenscina2.0.trojina.si/arhiv/2013-1/2013-1-06/ 1 https://dk.um.si/Dokument.php?lang=slv&id=74970 Fakulteta za elektrotehniko, računalništvo in informatiko 0 0 0