Comments (3)
@kegsay if you curl that URL, you can see that response is:
<html>
<head>
<META NAME="robots" CONTENT="noindex,nofollow">
<script src="/_Incapsula_Resource?SWJIYLWA=2977d8d74f63d7f8fedbea018b7a1d05">
</script>
<script>
(function() {
var z="";var b="7472797B766172207868723B76617220743D6E6577204461746528292E67657454696D6528293B766172207374617475733D227374617274223B7661722074696D696E673D6E65772041727261792833293B77696E646F772E6F6E756E6C6F61643D66756E6374696F6E28297B74696D696E675B325D3D22723A222B286E6577204461746528292E67657454696D6528292D74293B646F63756D656E742E637265617465456C656D656E742822696D6722292E7372633D222F5F496E63617073756C615F5265736F757263653F4553324C555243543D363726743D373826643D222B656E636F6465555249436F6D706F6E656E74287374617475732B222028222B74696D696E672E6A6F696E28292B222922297D3B69662877696E646F772E584D4C4874747052657175657374297B7868723D6E657720584D4C48747470526571756573747D656C73657B7868723D6E657720416374697665584F626A65637428224D6963726F736F66742E584D4C4854545022297D7868722E6F6E726561647973746174656368616E67653D66756E6374696F6E28297B737769746368287868722E72656164795374617465297B6361736520303A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2072657175657374206E6F7420696E697469616C697A656420223B627265616B3B6361736520313A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2073657276657220636F6E6E656374696F6E2065737461626C6973686564223B627265616B3B6361736520323A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2072657175657374207265636569766564223B627265616B3B6361736520333A7374617475733D6E6577204461746528292E67657454696D6528292D742B223A2070726F63657373696E672072657175657374223B627265616B3B6361736520343A7374617475733D22636F6D706C657465223B74696D696E675B315D3D22633A222B286E6577204461746528292E67657454696D6528292D74293B6966287868722E7374617475733D3D323030297B706172656E742E6C6F636174696F6E2E72656C6F616428297D627265616B7D7D3B74696D696E675B305D3D22733A222B286E6577204461746528292E67657454696D6528292D74293B7868722E6F70656E2822474554222C222F5F496E63617073756C615F5265736F757263653F535748414E45444C3D333330343638303230353238363636333535392C363039373834373236313734363832383438312C31303830363430353238313234373131343133322C343532323432222C66616C7365293B7868722E73656E64286E756C6C297D63617463682863297B7374617475732B3D6E6577204461746528292E67657454696D6528292D742B2220696E6361705F6578633A20222B633B646F63756D656E742E637265617465456C656D656E742822696D6722292E7372633D222F5F496E63617073756C615F5265736F757263653F4553324C555243543D363726743D373826643D222B656E636F6465555249436F6D706F6E656E74287374617475732B222028222B74696D696E672E6A6F696E28292B222922297D3B";for (var i=0;i<b.length;i+=2){z=z+parseInt(b.substring(i, i+2), 16)+",";}z = z.substring(0,z.length-1); eval(eval('String.fromCharCode('+z+')'));})();
</script></head>
<body>
<iframe style="display:none;visibility:hidden;" src="//content.incapsula.com/jsTest.html" id="gaIframe"></iframe>
</body></html>
It appears that the HTTP request that gofeed
makes is being flagged by their server's Incapsula WAF, while the validator website is receiving the valid feed response.
I'm honestly not sure how to modify our requests such that an overly aggressive WAF won't block them. The gofeed.Parser
uses golang's default http.client and default User Agent.
I'm open to suggestions on changing the signature of our requests, as long as it doesn't involve spoofing browsers or something similar.
If someone is interested in attempting their own work-around, they can skip using gofeed
's http.client and instead, create their own (with their own settings) and pipe the IO into the gofeed.Parse
method, rather than gofeed.ParseURL
.
from gofeed.
Looking into this.
from gofeed.
Thanks for looking into this. Interesting to see how broken that site is. I don't think you can set a default user agent header on http.Client
which is annoying, otherwise I'd just suggest making this the caller's problem. I guess I could intercept the RoundTripper
and set the User-Agent there.
Either way, this isn't really your problem. Thanks!
from gofeed.
Related Issues (20)
- Undetect Json array HOT 2
- Multiple 'enclosure' elements not being picked up... HOT 2
- build failed on go 1.18
- GoFeed parses the CoinDesk feed but the Link fields are empty HOT 4
- no parse content HOT 2
- Thanks for the hat tip! HOT 1
- Support for comments tag in feed Item HOT 3
- Limit the maximum number of feed items when parsing HOT 5
- Support URI field for Author in Atom Feed HOT 3
- corrupted / mangled nested custom XML HOT 1
- Enhance custom element handling HOT 5
- Make illegal character sanitization more robust HOT 1
- These links cannot be resolved HOT 3
- ITunes Summary should be store in the Description field of the feed if Description is empty
- Unit tests are failing HOT 8
- The RSS 2.0 comments fields appears to be missing HOT 1
- Ignore RSS "enclosure" child tags
- feeds.mashable.com: unexpected EOF in CDATA section HOT 1
- infoworld feeds: failed to detect feed type HOT 1
- Atom <logo> is cropped into a squared icon even if it should be 2:1 landscape according to specs HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gofeed.