Julio Montoya
fe5db3a50e
Remove unused code. Cleaning up i18n library.
|
%!s(int64=9) %!d(string=hai) anos | |
|---|---|---|
| .. | ||
| language_profiles | %!s(int64=10) %!d(string=hai) anos | |
| sample_texts | %!s(int64=10) %!d(string=hai) anos | |
| index.html | %!s(int64=10) %!d(string=hai) anos | |
| readme.txt | %!s(int64=14) %!d(string=hai) anos | |
readme.txt
Libbrary of statistical profiles for language recognition
---------------------------------------------------------
The sample texts for dieffernt languages have been taken from
Perl module: Lingua::LanguageGuesser - http://gensen.dl.itc.u-tokyo.ac.jp/LanguageGuesser/LanguageGuesser_demo.html
Statistical Text Analysis - http://boxoffice.ch/pseudo/
Some random sample texts have been taken from Wikiedia - http://wikipedia.org/
All the sample texts should be UTF-8 encoded!
To understand how does language recognition work you need to read the following remarkable work:
W. B. Cavnar and J. M. Trenkle. N-gram-based text categorization. Proceedings of SDAIR-94, 3rd Annual Symposium on Document Analysis and Information Retrieval, 1994.
http://citeseer.ist.psu.edu/cache/papers/cs/810/http:zSzzSzwww.info.unicaen.frzSz~giguetzSzclassifzSzcavnar_trenkle_ngram.pdf/n-gram-based-text.pdf
License: GNU General Public License 3 as published by the Free Software Foundation (http://www.fsf.org/).
Assembled by Ivan Tcholakov,
November, 2009
---------------------------------------------------------
The sample texts for dieffernt languages have been taken from
Perl module: Lingua::LanguageGuesser - http://gensen.dl.itc.u-tokyo.ac.jp/LanguageGuesser/LanguageGuesser_demo.html
Statistical Text Analysis - http://boxoffice.ch/pseudo/
Some random sample texts have been taken from Wikiedia - http://wikipedia.org/
All the sample texts should be UTF-8 encoded!
To understand how does language recognition work you need to read the following remarkable work:
W. B. Cavnar and J. M. Trenkle. N-gram-based text categorization. Proceedings of SDAIR-94, 3rd Annual Symposium on Document Analysis and Information Retrieval, 1994.
http://citeseer.ist.psu.edu/cache/papers/cs/810/http:zSzzSzwww.info.unicaen.frzSz~giguetzSzclassifzSzcavnar_trenkle_ngram.pdf/n-gram-based-text.pdf
License: GNU General Public License 3 as published by the Free Software Foundation (http://www.fsf.org/).
Assembled by Ivan Tcholakov,
November, 2009