Missing Maven dependency about boilerpipe HOT 11 OPEN

Hi Benjamin,

thanks for your bug report.

Technically, boilerpipe does not need NekoHTML to work (you could use TagSoup 
instead, for example).
Especially since boilerpipe is a dependency of Apache Tika (>= 0.8), which uses 
TagSoup, it would not make sense to enforce that dependency to NekoHTML in 
boilerpipe.

Still, right now, the boilerpipe jar contains some patched classes from 
NekoHTML (HTMLElements and HTMLTagBalancer), which somehow feels wrong.

In the next release of boilerpipe, I will remove these classes and provide a 
new jar file called something like "nekohtml-patch-2909310-1.9.14.jar" (2909310 
stands for the NekoHTML bug I reported almost a year ago, see 
http://sourceforge.net/tracker/?func=detail&aid=2909310&group_id=195122&atid=952
178 ).

Since that NekoHTML has not been fixed upstream for a long time, please also 
consider helping me to get this bug fixed in NekoHTML, by commenting at the 
link above.

Cheers,
Christian

Makes sense.  Feel free to close this issue then.  I commented on the NekoHTML 
bug and will try to follow up there.

i am wanting to use hotsax as the parser and just drop in the 
BoilerpipeHTMLContentHandler. making me depend on anything but the core engine 
and letting me pick the parser would be nice. so really the engine should be 
separated into a core.jar and the demos should be in a demo.jar which depends 
on the core jar and whatever parser you are running on boilerpipe-web. the best 
way to do this is a multimodule maven project. it would also be nice to then 
push the jars out onto maven central. i have experience in all of that (and how 
to have it play nicely with eclipse indigo). let me know if you want me to do 
this and i will find a free hour (post here as i don't check this account 
often.....)

I don't think this makes sense - a project's Maven dependency file should 
contain all of its dependencies.  If two dependencies reference the same 
dependency, as you fear with Apache Tika, then Maven takes care of it.

Boilerpipe would be perfect for me but trying to get it working has been 
painful due to missing dependencies (including nekohtml and xerces) in the 
pom.xml.

Boilerpipe should "just work" when you add it as a Maven dependency to your 
pom.xml - but currently it doesn't, and I'm going to have to find another 
solution :-(

Ian,

thanks for your comment.
Please provide a patch that takes care of that. I will be more than happy to 
add this feature.

Best,
Christian

Will this issue be ever resolved? Getting started with Boilerpipe is a real 
pain thanks to this undocumented missing dependencies jazz. Either specify them 
or clearly document how people can get started.

This is seriously broken. One one side you include some classes from the 
nekohtml but running a simple test with ArticleExtractor breaks with NoClassDef 
from nehkohtml.

On the other when I include nekohtml in dependencies, I get two versions of 
classes one from boilerpipe and other from nekohtml.

Better solution would be to make a different package name for you own nekohtml 
classes so it won't break other libs that depend on it.

This is the class not found exception:
Caused by: java.lang.ClassNotFoundException: 
org.cyberneko.html.HTMLConfiguration

Adding the following to the pom of boilerpipe should fix the problem:

               <dependency>
                       <groupId>xerces</groupId>
                       <artifactId>xerces</artifactId>
                       <version>2.4.0</version>
               </dependency>
               <dependency>
                       <groupId>net.sourceforge.nekohtml</groupId>
                       <artifactId>nekohtml</artifactId>
                       <version>1.9.17</version>
               </dependency>

The nekohtml "patched" classes change the nekohtml functionality. If you have 
Nekohtml on your classpath and use it at another part of a larger system, it 
can break previously working features... 

Guys ..

Its very easy to use boiler pipe in your maven project, just include dependency 
like below in your project's pom.xml.

     <repository>
            <id>boilerpipe-m2-repo</id>
            <url>http://boilerpipe.googlecode.com/svn/repo/</url>
    </repository>


...
              <dependency>  
            <groupId>de.l3s.boilerpipe</groupId>
            <artifactId>boilerpipe</artifactId>
            <version>1.2.0</version>
        </dependency>
        <dependency>
            <groupId>xerces</groupId>
            <artifactId>xercesImpl</artifactId>
            <version>2.9.1</version>
        </dependency>
        <dependency>
            <groupId>net.sourceforge.nekohtml</groupId>
            <artifactId>nekohtml</artifactId>
            <version>1.9.13</version>
        </dependency>