Turn unstructured HTML pages into structured data. The OpenScraping library can extract information from HTML pages using a JSON config file with xPath rules. It can scrape even multi-level complex objects such as tables and forum posts. This is the C# version.
License: Other
In the documentation for Transformations, the RemoveExtraWhitespaceTransformation example doesn't display the extra spaces in the first occurrence of "hello world". I think you'll need to use non-breaking spaces there, or if that doesn't work something else that has the same effect as non-breaking spaces.
It looks like this:
Replaces consecutive spaces with a single space. For the string "hello world" it would return "hello world".
But it should look like this:
Replaces consecutive spaces with a single space. For the string "hello world" it would return "hello world".
I have a need to get the content of meta data - how is this possible?
<meta content="Tim Fischer" name="author">
Thanks
The _xpath feature seems to behave differently if there is only one result from the _xpath. It will generate:
"rows": { "col1": "val1", "col2": "val2" }
Instead of:
"rows": [{ "col1": "val1", "col2": "val2" }]
This makes it difficult to iterate over the array of rows, as you end up iterating over the columns. Thoughts? Is there a way to force it to be an array regardless of there being a single row? Thank you.
There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.
Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.
While using openscraping faced quite common scenario when a specific subset(word) has to be selected from element with plain text.
Sample: <div class="info">Contact information. Phone: 111-111-111, Address: str.Street 1/1, City. 2017</div>
Would be really useful to have built-in RegexpTransformation which can take custom regexp expressions as an input param '_regexp'. Something like '_separator' in SplitTransformation.
On this page:
https://github.com/Microsoft/openscraping-lib-csharp
in the README.md Transformations Example column, all of the "Here" links return a 404 error.
When trying to add this package to a project targeting .NET standard 2.0, an error is thrown because OpenScraping v1.0.1 only supports netcoreapp2.0.
Here is the output when running dotnet add OpenScraping:
info : GET https://api.nuget.org/v3-flatcontainer/openscraping/index.json
info : OK https://api.nuget.org/v3-flatcontainer/openscraping/index.json 380ms
error: Package OpenScraping 1.0.1 is not compatible with netstandard2.0 (.NETStandard,Version=v2.0). Package OpenScraping 1.0.1 supports: netcoreapp2.0 (.NETCoreApp,Version=v2.0)
error: Package 'OpenScraping' is incompatible with 'all' frameworks in project '<REDACTED>/Project.csproj'.
<a href="test">dfsdfsdf</a>
I have tried systax like this //a/@href
It just returns the contents of the anchor tag, but I'm looking for "test" in the href attribute.
Is this possible?
Hello, I am using this config
{
"id": {
"_xpath": "//link[@rel='canonical']/@href",
"_transformations": [
{
"_type": "RegexTransformation",
"_regex": "[0-9]{7}$"
}
]
}
}
to extract from this document
<!DOCTYPE html>
<html lang="en" xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta charset="utf-8" />
<link rel="canonical" href="http://example.com/test-xyz-1234567" />
<title>Test</title>
</head>
<body>
Test
</body>
</html>
My goal is to extract "1234567" from the link-tag.
The resulting output is empty.
Reason: In RegexTransformation.cs the first match is ignored (line 77).
I am not sure why the first match is ignored and how to work around this in my example.
Could someone please provide some advice? Thank you very much.
My use case:
Given this URL: https://dev.test/index.php?PHPSESSID=a&action=profile;u=99 i wanted to extract the 99 user ID from the end of the string. My solution was to use a simple Regex and convert it to integer:
"_transformations": [
{
"_type": "RegexTransformation",
"_regex": "u=(\\d+)",
},
"CastToIntegerTransformation",
],But after i got
Transformation chain broken at transformation type CastToIntegerTransformation
started to debug the library and recognized that the CastToIntegerTransformation not inherits from ITransformationFromObject so i cannot use at the end of the parsing pipeline.
Yes, this problem can easily fixed with inheritance but i thought mention here.
/// <summary>
/// Class to cast selected XPath value to <see cref="int"/>.
/// </summary>
public class CastToIntegerTransformation : ITransformationFromHtml, ITransformationFromObject
{
public object Transform(Dictionary<string, object> settings, HtmlNodeNavigator nodeNavigator, List<HtmlAgilityPack.HtmlNode> logicalParents)
{
var text = nodeNavigator?.Value ?? nodeNavigator?.CurrentNode?.InnerText;
if (text != null)
{
int intVal;
if (int.TryParse(text, out intVal))
{
return intVal;
}
}
return null;
}
/// <summary>
/// Transforms the input to a valid <see cref="int"/>.
/// </summary>
/// <param name="settings"><seealso cref="Config.TransformationConfig.ConfigAttributes"/>.</param>
/// <param name="input">Parsed XPath value.</param>
/// <returns><see cref="int"/>.</returns>
/// <exception cref="FormatException">Occurs when the <paramref name="input" /> parameter
/// is not a valid integer.</exception>
public object Transform(Dictionary<string, object> settings, object input)
{
if (int.TryParse(input.ToString(), out int number))
{
return number;
}
throw new FormatException($"Input parameter {input} is not a valid integer!");
}
}Thank You for this great library!
I'm trying to use OpenScraping in a CSharp Script .csx using dotnet-script (https://github.com/filipw/dotnet-script). I posted an issue over there, but wanted to also submit here on the off chance that you'd be interested and/or possibly have some advice.
I have tried "link":"//a[selector]/@href".
The parsed result is the text of the link instead of the href url.
What is the correct way to retrieve the link?
O Gosh, it is a duplicate issue with the previous attribute selection problem.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.