Git Product home page Git Product logo

allorigins's Introduction

All Origins

FOSSA Status Build Status Donate!

Pull contents from any page via API (as JSON/P or raw) and avoid Same-origin policy problems.


A free and open source javascript clone of AnyOrigin, inspired by Whatever Origin, but with support to gzipped pages.

Examples

To fetch data from http://wikipedia.org:

fetch(`https://api.allorigins.win/get?url=${encodeURIComponent('https://wikipedia.org')}`)
  .then(response => {
    if (response.ok) return response.json()
    throw new Error('Network response was not ok.')
  })
  .then(data => console.log(data.contents));

Or with jQuery

$.getJSON('https://api.allorigins.win/get?url=' + encodeURIComponent('https://wikipedia.org'), function (data) {
    alert(data.contents);
});

Options

charset

Description: Set the response character encoding (charset)
Example: https://api.allorigins.win/get?charset=ISO-8859-1&url=https://pt.wikipedia.org/

raw

Description: Get the raw contents (CORS)
Example: https://api.allorigins.win/raw?url=https://example.org/

callback

Description: Get a JSONP response
Example: https://api.allorigins.win/get?callback=myFunc&url=https://example.org/

On your own server

# Clone the repo
git clone https://github.com/gnuns/AllOrigins

# Install dependencies
cd AllOrigins
npm install

# Fire it up!
npm start # the default port is 1458

License

FOSSA Status

allorigins's People

Contributors

dependabot[bot] avatar fossabot avatar frontstall avatar gnuns avatar jimmywarting avatar lui22 avatar mrhacker avatar pandawhocodes avatar tobihagemann avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

allorigins's Issues

Params from requests are dropped

Hi, I am sending this request
curl -X GET "https://api.allorigins.win/raw?url=https://api.nhle.com/stats/rest/en/skater/summary?isAggregate=false&isGame=false&sort=%5B%7B%22property%22:%22points%22,%22direction%22:%22DESC%22%7D,%7B%22property%22:%22goals%22,%22direction%22:%22DESC%22%7D,%7B%22property%22:%22assists%22,%22direction%22:%22DESC%22%7D%5D&start=0&limit=50&factCayenneExp=gamesPlayed%3E=1&cayenneExp=gameTypeId=2%20and%20nationalityCode=%22CZE%22%20and%20seasonId%3C=20202021%20and%20seasonId%3E=20202021"

but the response is different from expected, probably some parameters of the request are dropped by proxy. I am not sure, if I am using it well, or if it is a bug. Can you give me some advice on preventing dropping those params, if I am doing it badly? Thank you.

Decode URL

Since you're suggesting to use encodeURIComponent() to encode the URL, it seems that a decodeURIComponent() is missing in your code.

https://github.com/gnuns/AllOrigins/blob/0eea4d6e60486376deaf3a812828174490899978/lib/get-page.js#L18

https://github.com/gnuns/AllOrigins/blob/0eea4d6e60486376deaf3a812828174490899978/lib/get-page.js#L41

Instead of unescaping, I'd suggest to replace it with decodeURIComponent().

Btw, I think it totally makes sense to encode the URL in order for URLs like https://www.google.com/search?q=foo+bar to work. Otherwise the + gets lost. Escaping/unescaping doesn't work for this case.

If you want, I can open a pull request.

Edit: Another suggestion would be to change this line of code:

https://github.com/gnuns/AllOrigins/blob/0eea4d6e60486376deaf3a812828174490899978/lib/index.js#L10

And replace it with:

const page = await getPage(decodeURIComponent(params.url), params.method) 

issues with re-writing hyperlinks

maybe it needs an additional option, but:

  1. using <iframe is="x-frame-bypass" src="https://api.allorigins.win/raw?url=${url}"></iframe>
  2. where url = "https://example.com/demo.html"
  3. and contains a relative hyperlink to e.g. /example/bar.html
  4. it is re-written to "https://api.allorigins.win/example/bar.html"
  5. but should probably be re-written to:
    • https://api.allorigins.win/raw?url=https://example.com/example/bar.html

Is that something that could be added?

Caching

Can a response ever be cached? I'm noticing it always downloads the resource I request even if it hasn't changed.

CORS support

It would be nice to be able to request data with CORS instead of JSONP for better security.

Cross-Origin Request Blocked: CORS header 'Access-Control-Allow-Origin' missing

As of Firefox 56, the responses from this (and all the other anyorigin-like proxy services) are caught and blocked by the browser for missing the Access-Control-Allow-Origin http response header.

The error message on the console goes like the following:
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://www.whateverorigin.org/get?url=<INSERT URL>. (Reason: CORS header 'Access-Control-Allow-Origin' missing).

A relatively quick and easy fix is to add said headers to the response like so:
Access-Control-Allow-Origin: *

SSL

Hi love your solution. It appears the SSL has expired for https://allorigins.me . Can you please renew.

Cheers
Neil

Error 521

We have a trouble to render any url using the api. The error is a 521 telling us the server is down. Can you check what happened with the server? Thanks in advance.

https://allorigins.me gives me Nothing here yet :/ ?

I've been integrating AllOrigins with my app today, everything worked as it should, but just recently all my fetch requests returned GET error. From what I can see, this problem is not on my part, it might be the server you're hosting this API on.

image

SyntaxError: Unexpected token function

I get a strange error message when running this on a private host.

Config:
Node V6.9.0
Ubuntu server

Error message:

/workspaces/AllOrigins$ node server.js
/home/lancent/workspaces/AllOrigins/lib/index.js:7
async function processRequest (req, res) {
      ^^^^^^^^
SyntaxError: Unexpected token function
    at Object.exports.runInThisContext (vm.js:76:16)
    at Module._compile (module.js:542:28)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/home/lancent/workspaces/AllOrigins/server.js:8:21)
    at Module._compile (module.js:570:32)

Domain change breaks tool

A tool I've made that uses allorigins has broken on account of the domain change. The fix is trivial, but is there some way that I can prevent this from happening in the future? Perhaps auto detect the new domain somehow?

dns failure in JSONP mode produces broken unexecutable JSON data

While experimenting with allorigins, I noticed for a DNS failure, non-existent domain, the server returns HTTP code 400, which causes the JS callback to never execute since proper browsers do not execute JS code from a script element unless its a 2XX code https://www.theguardian.com/info/developer-blog/2012/jul/16/http-status-codes-jsonp .

example:

http://allorigins.me/get?url=https%3A//nx.baddomain.org/gis%3Floc%3DNYK%26_%3D1511810056058&callback=X

GET /get?url=https%3A//nx.baddomain.org/gis%3Floc%3DNYK%26_%3D1511810056058&callback=X HTTP/1.1
Host: allorigins.me
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2600.0 Iron Safari/537.36
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8

HTTP/1.1 400 Bad Request
Server: nginx/1.10.3 (Ubuntu)
Date: Mon, 27 Nov 2017 19:23:19 GMT
Content-Type: application/json
Content-Length: 141
Connection: keep-alive
X-Powered-By: Express
etag: "206367122"

{"error":{"code":"ENOTFOUND","errno":"ENOTFOUND","syscall":"getaddrinfo","hostname":"nx.baddomain.org","host":"nx.baddomain.org","port":443}}

Since the server DID return JSON, it is probably intended for the client to parse the DNS failure. But there is one more bug, I asked for JSONP, and my callback wrapper "X" function isn't in the output, so that is a 2nd bug. If my browser client was doing CORS the no callback wrapper and non-200 status code is fine, the client JS code will still be able to read and parse the DNS failure JSON block, but for JSONP mode, I need my callback wrapper and a 2XX status code.

How do you enable tls(https)

Setting up allOrigins was simple and works great but how do I set up tls(https) with it? For reference I'm running it on an ec2 was instance.

How to pass headers

I have to provide a special header in order to fetch. I am trying the following:

let response = await fetch( `https://api.allorigins.win/get?url=${encodeURIComponent('https://platform.tier-services.io/v1/vehicle?zoneId=BERLIN')}`, {
    headers: {
      "X-Api-Key": "MY_KEY"
    }
  });

but I always receive:
Request header field x-api-key is not allowed by Access-Control-Allow-Headers in preflight response.

Any ideas how to correctly provide the headers?

data contents: null, error: name: "RequestError" code: "ENOTFOUND"

Though the target API has xml content, the error displayed on returning response.text() is:

{"contents":null,"status":{"error":{"name":"RequestError","code":"ENOTFOUND","timings":{"start":1614172484183,"socket":1614172484183,"lookup":1614172484186,"error":1614172484186,"phases":{"wait":0,"dns":3,"total":3}}},"response_time":5}}

Issue setting up allOrigins on server

So I was able to install allOrigins on my server successfully however I'm having issues with CORS on subdomain I have it installed on. I have allOrigins setup on my domain as cors.mydomain.com. When I try to make a request from www.mydomain.com I get the following errors:

  1. Access to XMLHttpRequest at 'https://cors.mydomain.com/raw?url=https://youtube.com' from origin 'https://www.mydomain.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
  2. GET https://www.mydomain.com/raw?url=https://youtube.com 403

I created a .htaccess file set it in the cors.mydomain.com root folder to allow requests from mydomain.com but that didn't seem to do anything. Is there some kind of config file I need to modify in order to allow requests from certain domains?

Add option to not follow redirects?

Is it possible to add an option to not follow redirects?

Here is a possible use case:

The url freemusic.com/great_song/download redirects to files.freemusic.com/other_things/great_song.mp3. I just want to get the file url and not the actual file through the proxy.

It would be nice if I could fetch https://api.allorigins.win/get?noredirect=true&url=https://freemusic.com/great_song/download and see where it would have redirected.

Responding with the actual status code

Consider the following setup: Deploying AllOrigins on a server that has caching activated. For example, using this feature in NGINX: https://www.nginx.com/blog/nginx-caching-guide/#stale

The thing is now that caching doesn't work for errors because AllOrigins always responds with status code 200. Therefore, NGINX caches the error page even though it should handle it as a stale response.

I understand that it could make sense because AllOrigins's service indeed responds successfully but I think that AllOrigins should be considered a transparent proxy. At least that's what it's probably used for in most cases.

My proposal is to add res.status(page.status.http_code) somewhere in this method: https://github.com/gnuns/AllOrigins/blob/e170b6adbb8f538d71356c1e540e5c67f1f5dece/lib/index.js#L14

Please let me know what you think.

ISO-8859-1

I have a big problem, i need to get data from a page that uses iso-8859-1 charset, and when i use getjson, this give me � on all the special characters, can you give at least an example how to use $.ajax so i can try to decode the charset? thank you

Allow other request Methods

like post, put, delete, patch, or whatever
would also be nice if all response headers keys where exposed by Access-Control-Expose-Headers

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.