Penalizing unknown words' emissions in HMM POS tagger based on Malay affix morphemes

The challenge in unsupervised Hidden Markov Model (HMM) training for a POS tagger is that the training depends on an untagged corpus; the only supervised data limiting possible tagging of words is a dictionary. Any published dictionary can be used even if it does not include all the words found in a corpus. Therefore, training cannot properly map possible tags. The exact morphemes of prefixes, suffixes and circumfìxes in the agglutinative Malay language was examined to assign unknown words' probable tags based on linguistically meaningful affixes using a morpheme-based POS guessing algorithm for baseline tagging. The tagger was first examined using HMM-Viterbi with unknown words handled by character-based prediction; next, it was examined using HMM-Viterbi with unknown words handled by morpheme-based POS guessing; lastly, it was examined using a combination of the first and second. HMM-Viterbi tagging with morpheme-based POS guessing was found to be better than HMM-Viterbi tagging with characterbased prediction, especially at tagging unknown words not identified in the dictionary. The best combination proved to be using morpheme-based POS guessing with unknown word emissions replaced by a value proportionate to the marginal distribution of tags and words ' ending information, given a maximum predefined length of six characters and ignoring affixed words in smoothing. This method proved to be satisfactory for guessing tags of words not in the dictionary, and it outperformed the baseline.