ITeye博客

postagger

共 8 篇文章

Lucene 3.0 分词 IKAnalyzer

最近lucene已经更新到lucene 3.0版本了 2.X版本的一些用法已经彻底不在支持了。 下面的例子主要是介绍中文分词器IKAnalyzer的使用和Lucene高亮显示。 lucene 3.x版本中有些2.x方法已经完全被剔除了,这里会捎带一下3.x的用法,当然我这里用的还是2.X的版本。 lucene自带的分词方式对中文分词十分的不友好,基本上可以用惨不忍睹来形容,所以这里推荐使用IKAnalyzer进行中文分词。 IKAnalyzer分词器是一个非常优秀的中文分词器。 下面是官方文档上的介绍 采用了特有的“正向迭代最细粒度切分算法“,具有60万字/秒的高速处理能力。 采用了多子处理…

2010-09-02 12:46 · postagger

university 4/n (45)

chapter? Vocabulary use in classroom teaching and textbooks ?.?Introduction The description of vocabulary use in university contexts is an essential prerequi- site to the development of effective teaching materials and approaches.There are many important research questions about word use in univers…

2010-08-24 07:57 · postagger

university 3/n

Chapter 1.Introduction diagnostic tools for the evaluation of exam prompts,using the corpus analyses as a baseline. In the first stage of the project,we constructed the T2K-SWAL Corpus,which was designed to represent both spoken and written university registers,as well as the major academic discipl…

2010-08-24 07:55 · postagger

lucene analyzer pos

Parsing? Tokenization? Analysis! Lucene, indexing and search library, accepts only plain text input. Parsing Applications that build their search capabilities upon Lucene may support documents in various formats – HTML, XML, PDF, Word – just to name a few. Lucene does not care about the Parsing of …

2010-08-20 07:16 · postagger

penn tree bank 1/n

Building a Large Annotated Corpus of English: The Penn Treebank Mitchell P Marcus* University of Pennsylvania Beatrice Santorinit Northwestern University Mary Ann Marcinkiewiczt University of Pennsylvania Introduction There is a growing consensus that significant, rapid progress can be made in both…

2010-08-15 23:24 · postagger

[ZZ]功能词与实义词

Words are divided into two categories: Function Words and Content Words. Function words are closed class words (only about 300 in English) while content words are open class words (new words are being added in every language). Function Words with Examples Prepositions of, at, in, without, between P…

2010-03-21 22:21 · postagger

hack the stanford postagger demo

stanford postagger 的demo默认情况下输出slashtags。在代码中如何修改,才能让它输出xml? 在MaxentTagger.java文件中,修改public String apply(String o)代码 public String apply(String o) { StringBuilder taggedSentence = new StringBuilder(); int outputStyle; boolean tokenize; if (config != null) { outputStyle = PlainTextDocumentReaderAndW…

2010-03-20 10:18 · postagger

stanford postagger tagsets

stanford postagger stagsets stanford出的postagger用的词性标签集正是pen的词性标签集。 The Penn Treebank Tag SetThe Penn Treebank Tag Set The tagset used in tagging the demo corpus available here is the Penn Treebank Tag set, described for example in Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz: …

2010-03-20 10:12 · postagger