Lucene47IKAnalyzer中文分词入门教程.docx

1.ucene4.7+IKAnaIyZer中文分词入门教程作者：大明日期:2019-4-1环境：1. IUCene版本：4.7下载地址：下载页面中的2. IKAnaIyZer版本:IKAnalyer2019-FFHotfix1下载地址：选择最上面的源码包。目前只有这个源码包才能支持Iucene的4.7版本。3. 参考文章：此篇文章作者用的是IUCCne3.5版本，我做了肯定幅度的修改，使其在4.7中可用。下面进入正式部署阶段：1 .在eelipse中新建一个java项目：Iucenetest2 .解压3 .进入到解压后的书目，搜寻*.jar4 .搜寻出来的结果是lucene4.7供应的全部的jar包，依据须要拷贝不同的jar包到自己的工程。本例子须要如下几个jar包：5 .将以上几个Jar包拷贝到Iucenetest下的Iib文件夹下，假如没有lib,则新建。6 .在eclipse中，打开lib,选中这儿个jar包，点右键一buildparhaddtobuildpath7,解压下载的IKAnalyzer2019FF_hfl_source.rar8 .进入到IKAnalyzer2019FFIf1.SOUrCesrc将如下全部的文件拷贝到IUCCneteslsrc下9 .刷新eclipse工程。假如源码报错，可能是编码格式的问题。解决方法：在Iuccnetest上点右键，选择最终一项Properties0如下图所示选择Utf-8,点确定10 .至此，环境已经搭建完毕。假如嫌ik的源码烦，可以自行打成jar包。目前eclipse项目的结构图如下:11 .加入测试的例子。在src新建ik包，新建下面3个Java文件(I)DataFactorypackageik:importjava.util.Array1.ist;importjava.util.1.ist;publicclassDataFactoryprivatestaticDataFactorydataFactory=newDataFactoryO;privateDataFactoryOpublic1.ist<Medicine>getDataO1.ist<Medicine>list=newArray1.ist<Medicine>（）;list.add（newMedicine。，”银花感冒颗粒"，"功能主治：银花感冒颗粒，头痛,清热，解表，利咽.”）；list,add（newMedicine（2,"感冒止咳糖浆"功能主治：感冒止咳糖浆,解表清热，止咳化痰。”）；list,add（newMedicine（3,"感冒灵颗粒功能主治：解热镇痛。头痛,清热。”）；list.add（newMedicine（4,"感冒灵胶奏功能主治:银花感冒颗粒，头痛,清热，解表，利咽。”）；list.add（ne睛MediCine（5,"仁和感冒颗粒功能主治：疏风清热，宣肺止咳，解表清热，止咳化痰°"）;returnlist;）publicstaticDataFactorygetInstanceOreturndataEactory;(2)1.uceneIKUtilpackageik：importjava.io.FiIe;importjava.io.IOException;importjava,uti1.Array1.ist;importjava.util.1.ist：importorg.apache.Iucene.analysis.Analyzer:importorg.apache.Iucene.document.Document;importorg.apache.Iucene.document.Field;importorg.apache.Iucene.document.FieldType;importorg.apache.Iucene.document.TextField;importorg.apache.Iucene.index.DirectoryReader;importorg.apache.Iucene.index.IndexReader:importorg.apache.Iucene.index.IndexWriter;importorg.apache,lcene.index.IndexWriterConfig;importorg.apache.Iucene.index.Term;import0rg.apache.Iucene.queryparser,classic.MultiFieldQueryParser;importorg.apache.Iucene.search.IndexSearcher;importorg.apache.Iucene.search.Query;importorg.apache.Iucene.search.ScoreDoc;importorg.apache.Iucene.searchDocs：importorg.apache,ucene.search,highlight.Formatter;importorg.apache.Iucene.search,highlight.Fragmenter:importorg.apache,ucene.search,highlight.Highlighter;importorg.apache.Iucene.search,highlight.QueryScorer;importorg.apache,uce11e.search,highlight.Scorer;importorg.apache.Iucene.search,highlight.SimpleFragmcnler;importorg.apache.Iucene.search,highlight.SimpleHTU1.Formatter;importorg.apache.Iucene.store.Directory;importorg.apache,ucene.store.FSDirectory;importorg.apache.Iucene.util.Version;importorg.wltea.analyzer.Iuccnc.TKAnalyzer;publicclass1.uceneIKUtilprivateDirectorydirectory：privateAnalyzeranalyzer;*带参数构造,参数用来指定索引文件书目pubic1.uceneIKUtil(StringindexFiIcPath)try(directory=FSDirectory.open(newFile(indexFiIePath);analyzer=newIKAnalyzerO：catch(IOExceptione)e.PrintStackTraceO;)* 创建索引* Description：* ©authordennisitQ163Apr3,2019* throwsException* /publicvoidCreateIndex()throwsExceptionIndexWriterConfigindexWr1.terConfig=newIndexWriterConfig(Version.1.UCENE_47,analyzer);IndexWriterindexWriter=newIndexWriter(directory,indexWriterConfig);indexWriter.deletellO;1.ist<Medicine>listDataFactory.getInstanceO.getDataO;for(inti=0;i<list.size();i+)Medicinemedicine=list.get(i);DocumentdocumentaddDocument(medicine,getIclO,medicine.getName(),medicine.gctFunction();/indexWriter.addDocument(document);假如用添加，每次都是新的。所以须要用UPdaIeindexWriter.UpdateDocument(newTenn("id",medicine,getld().IoString(),document);indexWriter.closeO;I* Description：* 触IUIhordennisit0163Apr3,2019* ©paramid* ©paramcontent* ©return* /publicDocumentaddDocument(Integerid,Stringname,Stringfunction)(Documentdoc=newDocumentO;/Field.Index.NO表示不索引/Field.Index.ANA1.YZED表示分词且索引/Field.Index.NOT-N1.YZED表示不分词但索引/doc.add(newField(*id*,String,valuef(id),Field.Store.YES,Field.Index.NOT_ANA1.YZED);/doc.add(newField("name”,name,Field.Store.YES,Field.Index.ANA1.YZED)/doc.add(newField("function”,function,Field.Store.YES,Field.Index.AN1.YZED);FieldTypeft=newFieldTypeO：ft.SetStored(true);ft.SetIndexed(true)：ft.setTokenized(false);doc.add(newField("id”,String.VaIUeOf(id),ft);doc.add(newField("name”,name,TextField.TYPE_STORED);doc.add(newField(*function*,function,TextField.TYPE_STORED)：returndoc:)* Description：更新索引* ©authordennisit163Apr3,2019* Wparamid* ©paramtitle* Paramcontent* /publicvoidupdate(Integerid,Stringtitle,Stringcontent)try(IndexWriterConfigindexWriterConfig=new1ndexWriterConfig(Version.1.UCENE_47,analyzer);IndexWriter(directory,indexWriterConfig);Documentdocument=addDocumcnt(id,title,content);Termterm=newTerm("id”,String,valuef(id);indexWriter.UpdateDocument(term,document);indexWriter.closeO;catch(Exceptione)e.PrintStackTraceO;)* Description：依据ID进行索引* ©authordennisit0163APr3,2019* ©paramid* /publicvoiddelete(Integerid)try(IndexWriterConfigindexWriterConfig=new1ndexWriterConfig(Version.1.UCENE_47,analyzer);IndexWriter(directory,IndexWriterConfig);Termterm=newTerm("id”,String.vaIueOf(id);indexWriter.deleteDocuments(term);indcxWriter.cIoseO;catch(Exceptione)e.PrintStackTraceO;)* Description：查询* ©authordennisit0163Apr3,2019* ©paramwhere查询条件* ParamscoreDoc分页时用* /public1.ist<Medicine>search(Stringfields,Stringkeyword)IndexSearcherindexSearcher=null;1.ist<Medicine>result=newArray1.isKMedicineX);try创建索引搜寻器,且只读/DirectoryRcaderireaderDirectoryReader.open(directory);/IndexSearcherisearcher=newIndexSearcher(ireader);IndexReaderIndexKeaderDirectoryReader.open(directory);indexSearcher=newIndexSearcheNindexReader);MultiFielClQueryParserqueryParser=newMulIiFieldQueryParser(Version.1.UCENE_47,fields,analyzer);Queryquery=queryParser.parse(keyword);返回前number条记录TopDocstopDocs三IndexSearcher.search(query,10);信息展示inttotalCount=topDocs,totalllits;System,out.Println("共检索出*+totalCount+*条记录")；/高亮显示/* 创建高亮器，便搜寻的结果高亮显示SimpleHTM1.Eorniatter:用来限制你要加亮的关键字的高亮方式此类有2个构造方法* 1：SimPleHTM1.FOrmatter()默认的构造方法.加亮方式：邻关键字<B>* 2：SimplellTM1.l'ormatter(StringpreTag,String* PoStTag).加亮方式：PreTag关健字PoStTag*/Formatterformatter=newSinlPIeHTM1.Formatter("fontcolor=*red'>","<font>");* QueryScorerQueryScorer* 是内置的计分器。计分器的工作首先是将片段排序。QueryScorer运用的项是从用户输入的查询中得到的：* 它会从原始输入的单词、词组和布尔查询中提取项，并且基于相应的加权因子(boostfactor)给它们加权。* 为了便于QUehSCOere运用，还必需对查询的原始形式进行重写。比如，带通配符查询、模糊杳询、前缀查询以及范围查询* 等，都被重写为B。IenaQUei、y中所运用的项。* 在将Query实例传递到QueryScorer之前,可以调用Query,rewrite* (IndeXReadCr)方法来重写QUery对象* /Scorerfragme11tScorer=newQueryScorer(query);Highlighterhighlighter=newHighlighter(formatter,fragmentScorer);Fragmenterfragmenter=newSimpleFragmenter(100)：* Highlighter利用Fragmenter将原始文本分割成多个片段。* 内置的SimpleFragmenter将原始文本分割成相同大小的片段，片段默认的大小为100个字符。这个大小是可限制的。* /highlighter.SetTextFragmenter(fragmenter);ScoreDocScoreDocs=topDocs.scoreDocs;for(ScoreDocscDoc:scoreDocs)Documentdocument=indexSearcher.doc(scDoc.doc);Integerid=Integer,parselnt(document,get("id");Stringname=document,get("name");Stringfunctiondocument,get("function"):/floatscore=scDoc.score;相像度StringIighterName=highlighter.getBestEragment(analyzer,"name”,name);if(nulI=IighterName)(IighterName=name：)StringIighterFunciton=highighter.getBestFragmcnt(analyzer,“function*,function);if(nulI=IighterFunciton)IighterFunciton=function;)Medicinemedicine=newMedicineO；medicine,setld(icl);medicine.SetName(IighterNamc);medicine.SetFunction(IighterFunciton);result,add(medicine);)catch(Exceptione)e.printStackTrace()；returnresu1t;)* 推断是否已经存在索引文件* ParamindexPath* ©return* /privatebooleanIsExistIndexFile(StringindexPath)throwsException(Filefile=newFiIe(indexPath);if(!file.exists()file,mkdirs()：StringindexSufix=*/segments,gen”;依据索引文件segments,gen是否存在推断是否是第一次创建索引FileindexFile=newFile(indexPath+indexSufix);returnindexFile.exists();)publicstaticvoidmain(Stringargs)1.uceneIKUti1IuceneProcess=new1.uceneIKUtil(wD:/索引存放的位置”)；tryIuccncProccss.CreateIndex():catch(Exceptione)e.printStackTrace();修改测试IuceneProcess.update(2,“测试内容"，Z修改测JR。")；查询测试Stringfields="name","function"IuceneProcess.SearCh(fields,“感冒")；for(inti=0;i<list.size();i+)Medicinemedicine=list.get(i);System,out.PrintIn("("+medicine,getIdO+*)w+medicine,getNameO+"t"+medicine.getFunction():删除测试/IuenceProcess.deleted)；)(3)Medicinepackageik;publicclassMedicineprivateIntegerid;privateStringname;privateStringfunction;publicMedicineOpublicMcdicinedntegerid,Stringname,Stringfunction)superO;this,id=id;this,name=name;this,function=function;)/getterandsetter()publicStringtoString()returnthis,id+"+this,name+"+this,function;)publicIntegergetld()returnid;publicvoidsetld(lntegerid)this.id=id;publicStringgetName()returnname;publicvoidsetNamc(Stringname)this,name=name;publicStringgetFunction()returnfunction;)publicvoidsetFunction(Stringfunction)this,function=function:)12.运行1.UCeneIKUtil的main方法,即可。Iucene我也是刚刚入门，好多东西还不明白，假如本文档有问题或者志同道合者，希望加我QQ：1315485916