est / cx-extractor Goto Github PK
View Code? Open in Web Editor NEWAutomatically exported from code.google.com/p/cx-extractor
Automatically exported from code.google.com/p/cx-extractor
建议: 1. 如果要提取娱乐类的网页,尤其是在图片把正文分割的比较支离破碎时, 建议用Java版代码。Java版实现时对多个正文片段进行合并,可以很好 的处理这一问题。但缺点是正文结尾可能会有少许噪声。 2. Perl和PHP的实现版本,一遍扫描只求最大行块,不进行拼接。如果出 现特别支离破碎的正文时,可能会有丢失。但优点是边缘的噪声去除的 很好。 有任何问题,欢迎随时联系我:) **************************************** 陈 鑫 Email: [email protected] Blog: http://hi.baidu.com/爱心同盟_陈鑫 ****************************************
Perl版本中,生成的txt文件名都是乱码,文件内容正常。
Original issue reported on code.google.com by [email protected]
on 21 May 2013 at 7:55
source = links.matcher(source).replaceAll("");
样例:http://news.itxinwen.com/2013/0802/515691.shtml
单是这一步 将耗时90s+
建议:可以直接通过source = source.replaceAll("<[^>]+>", "");
移除所有Tag?
Original issue reported on code.google.com by [email protected]
on 2 Aug 2013 at 8:01
如果网页中含有英文,单词间的空格不应该被删去。
Original issue reported on code.google.com by [email protected]
on 21 May 2013 at 7:56
cx-extractor
算法不错,提供了一种新的思路,以前我做过的是分析提取��
�页面中所有的TABLE和DIV区块,按区块字段的大小多少来判断��
�
我按cx-extractor算法做了一下,碰到以下几个问题:我是用C#来
做的
1、preProcess不能过滤标签中有脚本的情况,如其中的IMG
http://developer.51cto.com/art/201012/236066.htm
2、是否考虑以下2个方面的进一步改进;即在第一次匹配失败
后进行下面2中再次过滤
1、正文一般是DIV或者TABLE(TR/TD) 进行包围的,将这些标签换成特殊标签;在行和块合并时把这些特殊标签作为一种参考界定
2、类似下文中,正文中<p>应用较多,P中间的标签可以替换掉,计算连续的P标签
http://hi.baidu.com/jrckkyy/blog/item/a0c70a995e3579196f068c4e.html
3、博客方面还不是很理想
http://www.cnblogs.com/zhoujg/archive/2010/12/04/1895887.html
http://sarin.javaeye.com/blog/830831
http://blog.sina.com.cn/s/blog_4c4fd3070100nbvt.html?tj=1
4、这篇新闻好像也出了点问题
http://news.sina.com.cn/c/2010-12-04/100718432475s.shtml
Original issue reported on code.google.com by [email protected]
on 4 Dec 2010 at 3:07
如题。
Original issue reported on code.google.com by [email protected]
on 16 Oct 2013 at 3:12
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.