Hi @ustayready,

Thank you very much for sharing this tool!

Was wondering if it would be possible to keep a log of the rotating IPs. The rational behind that is that blue teams may want a list of IP addresses used during the activity to filter out the noise and potentially identify any non-expected activity occurred around the same time.

Keep up the good work!

The "Spoof X-Forwarded-For source IP header by requesting with an X-My-X-Forwarded-For header" is not working:

Request:
GET /fireprox/ HTTP/1.1
Host: 6dge1sxsw0.execute-api.us-west-2.amazonaws.com
Accept-Encoding: gzip, deflate
Accept: /
Accept-Language: en
X-Forwarded-For: 1.1.1.1
X-Forwarded-Port: 4431
X-Forwarded-Proto: http
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36
Connection: close

request headers on the server-side, arrives with randomized IP, but with passed real IP:
X-Forwarded-For: 1.1.1.1, real.ip.real.ip
X-Forwarded-Port: 443
X-Forwarded-Proto: https
x-amzn-apigateway-api-id: 6dge1sxsw0

~/fireprox$ virtualenv -p python3 .
bash: /home/xzj8b3/fireprox$: Non-existent file or directory
[xzj8b3@fedora fireprox]$

Can you explain that better?

Hi,

I tested on a Cloudflare enabled website and it is getting "Forbidden"

HTTP/1.1 403 Forbidden
Date: Wed, 12 Aug 2020 16:18:31 GMT
Content-Type: application/json
Content-Length: 23
Connection: close
x-amzn-RequestId: XXXX-XXXX-XXX-XXX-XXXX
x-amzn-ErrorType: ForbiddenException
x-amz-apigw-id: XXXXXXX

{"message":"Forbidden"}

Is there a way to bypass this?

I guess Cloudflare is blocking because we are coming with an IP in the AWS IP pool

Thanks

Thank you for the awesome project.

When a percentage symbol is sent anywhere in the URL, the request is not proxied and 400 response is received.

Deleting sroo41spvd => Failed!
how to delete this

i tried this tool with google search. it works fine without using concurrency,
it gets rate limited after several requests, when using concurrent
i ve set x forwarded for header correctly.
i have no clue why does that happen

When I run the command, I get the error below. I installed all the requirements successfully. I'm running wsl.

python fire.py --access_key aaaaa --secret_access_key aaaaaa --region us-east-1 --url https://login.microsoft.com --command create

File "fire.py", line 18
def init(self, arguments: argparse.Namespace, help_text: str):
^

                            ^

We tried to use it for google but when we are doing continuous requests for web crawling we end und up with 429 - Too Many Requests. How often is the IP address rotating?

I created fireprox api

Creating => http://httpbin.org/...
[2020-05-10 20:40:33+07:00] (2h5e9igxr4) fireprox_httpbin => https://2h5e9igxr4.execute-api.us-east-2.amazonaws.com/fireprox/ (http://httpbin.org/)

But when I request to https://2h5e9igxr4.execute-api.us-east-2.amazonaws.com/fireprox/ it return {'message': 'Internal server error'}. What is the prolem? Any help would be appreciated!

Hi

Thanks for this tool, really amazing.

I do have an issue with indeed.com where the scraper gets blocked (403 responses) even when adding the X-My-X-Forwarded-For : 127.0.0.1 to the headers.

Indeed.com is protected by cloudflare.

any suggestions?

Hi there,

Great concept. When I ping my the webserver I setup to return my IP addresses fireprox passes both my local IP as well as the AWS IP. Would you know why this is happening and if there is a way to only send the AWS IP?

Example return from the webserver "X-Forwarded-For":"local_ip,aws_ip","Cf-Ipcountry":"AU","Accept-Encoding":"gzip","Connection":"Keep-Alive","Host":"xxxx","Content-Length":"","Content-Type":"","ip":"local_ip,aws_ip"

Thanks

When I send request to Amazon API gateway url which is created by fireprox, I get 500 Internal Server Error
Anyone know what might be causing this issue?

If I create my fireprox proxy directly to www.example.com/subdirectory the request goes through. If I set the proxy to www.example.com and try to access .../fireprox/subdirectory I get 403: Forbidden.
Any Idea how to debug or why is it?

The google and bing examples are not working! Can you please fix it?

python3 fire.py --access_key ***************** --secret_access_key "******************************************" --region "us-east-1" --profile_name **** --command create --url "https://www.whatsmyip.org/"

The above command shows help output and then "Unable to load AWS credentials"

Hi,

I'm trying to use this this with the following command:

python fire.py --access_key XXX --secret_access_key XXX --region us-east-1 --command create --url XX

But I'm getting this error:

botocore.errorfactory.BadRequestException: An error occurred (BadRequestException) when calling the ImportRestApi operation: Errors found during import: Unable to put integration on 'GET' for resource at path '/': Invalid HTTP endpoint specified for URI Unable to put integration on 'ANY' for resource at path '/{proxy+}': Invalid HTTP endpoint specified for URI

Anyone know why?

This may be a simple, or stupid issue, but when I try to run:
┌──(fireprox)(kali㉿kali)-[~/MSOLSpray/fireprox]
└─$ python fire.py
exit

I get:

File "/home/kali/MSOLSpray/fireprox/fire.py", line 4, in
import tldextract
ModuleNotFoundError: No module named 'tldextract'

I looked back at the output of:

sudo pip install -r requirements.txt

which shows:

Requirement already satisfied: tldextract in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 2)) (3.1.2)

I have run all applicable with sudo as well and I am not understanding why it does not see that module.

additional:
┌──(kali㉿kali)-[~]
└─$ uname -a 127 ⨯
Linux kali 5.10.0-kali9-amd64 #1 SMP Debian 5.10.46-4kali1 (2021-08-09) x86_64 GNU/Linux

Maybe something changed in AWS infrastructure, but while creating an entry like this:

fire.py --command create --url https://ifconfig.me/ --region eu-west-1 --access_key ABC --secret_access_key "XYZ"

I do see my IP in the X-Forwarded-For response on the ifconfig.me page.

bing and google python script not working
please look into this

If you try to view a video, the response has status code 500 with message `{"message": "Internal server error"}``
Example:

python3.9 fire.py --access-key KEY --secret_access_key SECRETKEY --region eu-west-1 --command create --url "https://video.xx.fbcdn.net/"

and you visit https://newlygeneratedurl-execute-api.eu-west-1.amazonaws.com/fireprox/v/t66.36240-2/10000000_5162062970516874_3952172667405771162_n.mp4?_nc_cat=110&_nc_ht=video.fmnl8-2.fna&_nc_ohc=RuGOgev5TvEAX8FI4Hk&_nc_sid=985c63&ccb=1-7&efg=eyJybHIiOjIzNzQsInJsYSI6MTc5MSwidmVuY29kZV90YWciOiJvZXBfaGQifQ%3D%3D&oe=628E6A4A&oh=00_AT__3AG9wy4Vz9QqimNjwzLTspBsfs0E5-VyVRaLCqGAUw&rl=2374&vabr=1583

Also tried with:

• https://previews.customer.envatousercontent.com/h264-video-previews/34bc5a70-9db3-4ee9-821d-b4304d0c5dc5/33244341.mp4

This appears to be a configuration issue at aws API Gateway. Any idea what is the problem?

Using a --profile_name parameter returns the error Unable to load AWS credentials:

fireprox (master) $ ./fire.py --profile_name jmerckle --region us-east-1 --command list
usage: fire.py [-h] [--profile_name PROFILE_NAME] [--access_key ACCESS_KEY] [--secret_access_key SECRET_ACCESS_KEY] [--session_token SESSION_TOKEN]
               [--region REGION] [--command COMMAND] [--api_id API_ID] [--url URL]

FireProx API Gateway Manager

optional arguments:
  -h, --help            show this help message and exit
  --profile_name PROFILE_NAME
                        AWS Profile Name to store/retrieve credentials
  --access_key ACCESS_KEY
                        AWS Access Key
  --secret_access_key SECRET_ACCESS_KEY
                        AWS Secret Access Key
  --session_token SESSION_TOKEN
                        AWS Session Token
  --region REGION       AWS Region
  --command COMMAND     Commands: list, create, delete, update
  --api_id API_ID       API ID
  --url URL             URL end-point

Unable to load AWS credentials

Debugging this indicates this is the problem:

self.client = boto3.session.Session(profile_name=self.profile_name).client('apigateway')

The region is required otherwise Boto3 will raise botocore.exceptions.NoRegionError. This is a broad try/except block with no specific exception handling, so it's easy to miss this. This could be a change in Boto3 (I'm using 1.17.59) since the initial release too.

Cloudflare is offering an API gateway as well now.

Would it be possible to use their API gateway instead of the AWS API gateway?

Cloudflare is cheaper per 1million requests than AWS I think.

While working through VPS and using custom scripts, adding the fireprox urls everytime at the end of those scripts can be challenging. I currently use a rotating proxy service and configured redsocks to rotate my IP with each request.

I am thinking of ways to configure aws gateway in redsocks. Or maybe make changes in fireprox to do so.

Shall we work on this together?

I really apologize, because I feel like this is probably really obvious, but how do I really use this? I was looking over the examples, but was still kind of confused.

Is the --url argument supposed to be the domain of the site I'm going to be hitting? I'm using it for scraping. Do I run this once for each target site I want to scrape?

After I've run it, do I replace the domain of my target site in my spider code with the execute-api.us-east-2.amazonaws.com/fireprox/ URL? Or do I use that as my HTTP proxy? Or something else?

I'm using Scrapy. If I have something simple like:

    def start_requests(self):
        yield Request('http://thesiteiamcrawling.org/category/parameter', callback=self.parse)

Would I simply pass thesiteiamcrawling.org as the url argument when I run Fireprox and otherwise carry on as usual?

How does the spider actually make use of the proxy when it tries to hit thesiteiamcrawling.org ? In the examples, it almost seemed like you just hit the same url you always would, and so I assumed Fireprox was rerouting requests that matched the domain passed to it in the url argument. Didn’t seem to work for me when I tried it in my spider, though.

Does the code that’s going to use the proxy also have to be Python 3.6? My spiders still need to be converted from 2.7.15.

I ran http.server the same as in the screenshots, but no traffic appeared.

Thank you so much!

Hello,

I have tested the tool and, when making GET requests, it works correctly. The requests are made from different AWS IPs and arrive correctly at the web server. But when I try to make POST requests (for a password spraying or user enumeration attack, for example) the requests do not reach the web server and I get a "Missing Authentication Token" error. I have searched about this error and I think I should not specify any token when making cURL requests using the fireprox url as indicated in the tool. Does anyone know why this is happening? Maybe I am doing something wrong?

I ran the tool with the following command to create the AWS proxy:

python fire.py --access_key 'XXXX' --secret_access_key 'XXXX' --command create --url https://login.microsoftonline.com/common/GetCredentialType --region us-east-2

Thanks a lot for the tool, it's great!

It appears that all of the detectable headers are still included in the request.

Repro:

python fire.py --command=create --url=https://headers.cloxy.net/request.php

Then:

curl https://xyz.execute-api.ap-northeast-1.amazonaws.com/fireprox/ | grep '<li>'

<li>Connection: close</li>
<li>Host: headers.cloxy.net</li>
<li>X-Amzn-Apigateway-Api-Id: xyz</li>
<li>X-Forwarded-Proto: https</li>
<li>X-Forwarded-Port: 443</li>
<li>X-Forwarded-For: 161.230.x.x</li>
<li>X-Amzn-Trace-Id: Root=1-62726bb1-7d733e213b3e1c624qwerty</li>
<li>User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Firefox/31.0</li>
<li>Accept: */*</li>

Is there a way to use the root URL https://{api_id}.execute-api.{self.region}.amazonaws.com/ instead of the path https://{api_id}.execute-api.{self.region}.amazonaws.com/fireprox/? I tried modifying the code to support this, but I wasn't able to figure it out, as I'm not sure what the paths in the template are doing. (I believe that's what needs to be changed)

I have a PHP crawler, is there a way i can use this to generate proxy details for SOCK5 and use with the existing app?

it seems x-fowarded it still being sent.
I used the following site to check if headers were being sent.

here is the command to run.
python fire.py --access_key ACCESSKEY --secret_access_key SECRET --region us-west-2 --command create --api_id 1 --url http://www.whatismyproxy.com/

result after visiting endpoint:

X-Forwarded-For: my actual ip
X-Forwarded-Port: 443
X-Forwarded-Proto: https

did the patch stop working?

I would like to simulate user sessions as close as possible. Hence, I need to switch IPs on defined number of page loads.
Is this possible with the API gateway method?

Hi, sorry to pollute this with a dumb question, but: How many IP addresses does this rotate, I am unable to find anything in the AWS documentation. I imagine it's different per region, anything helps. Thanks!

It looks like Api Gateway is automatically turning the "%2f" to a "/". This causes request to fail. In this particular case the error message is 403, URL signature mismatch. Problematic url example:

https://scontent.xx.fbcdn.net/v/t39.25447-2/315991144_6377206065657734_5025935354679506890_n.mp4?_nc_cat=106&vs=e0ece699e5d69073&_nc_vs=HBksFQAYJEdHaWsxUktHOXpQX0NLZ1dBTXEzdjVlVHRiOUZibWRqQUFBRhUAAsgBABUAGCRHTlI1MVJKU3ZyWjZlYmtFQUM5WExHZ25HS0F6YnJGcUFBQUYVAgLIAQBLB4gScHJvZ3Jlc3NpdmVfcmVjaXBlATENc3Vic2FtcGxlX2ZwcwAQdm1hZl9lbmFibGVfbnN1YgAgbWVhc3VyZV9vcmlnaW5hbF9yZXNvbHV0aW9uX3NzaW0AKGNvbXB1dGVfc3NpbV9vbmx5X2F0X29yaWdpbmFsX3Jlc29sdXRpb24AHXVzZV9sYW5jem9zX2Zvcl92cW1fdXBzY2FsaW5nABFkaXNhYmxlX3Bvc3RfcHZxcwAVACUAHAAAJtLalcytxbYCFQIoAkMzGAt2dHNfcHJldmlldxwXQDQaXjU%2FfO4YIGRhc2hfdjRfNXNlY2dvcF9ocTFfZnJhZ18yX3ZpZGVvEgAYGHZpZGVvcy52dHMuY2FsbGJhY2sucHJvZDgSVklERU9fVklFV19SRVFVRVNUGwqIFW9lbV90YXJnZXRfZW5jb2RlX3RhZwZvZXBfaGQTb2VtX3JlcXVlc3RfdGltZV9tcwEwDG9lbV9jZmdfcnVsZQd1bm11dGVkE29lbV9yb2lfcmVhY2hfY291bnQGMTA5NTIxEW9lbV9pc19leHBlcmltZW50AAxvZW1fdmlkZW9faWQPNDcwODM4MDA1MTIwMzYxEm9lbV92aWRlb19hc3NldF9pZBAxMDk3NTM2MDE3NjIxNjU2FW9lbV92aWRlb19yZXNvdXJjZV9pZA82ODI4ODg3Mzk4NjIxODUcb2VtX3NvdXJjZV92aWRlb19lbmNvZGluZ19pZA81MTk4NzQyODMzNjkyNTEOdnRzX3JlcXVlc3RfaWQAJQIcACWOAhsIiAFzAzYzOAJjZAoyMDIyLTExLTE4A3JjYgYxMDk1MDADYXBwDFBvd2VyIEVkaXRvcgJjdBlDT05UQUlORURfUE9TVF9BVFRBQ0hNRU5UE29yaWdpbmFsX2R1cmF0aW9uX3MJMjAuMTM4NjY3AWYCYWQCdHMVcHJvZ3Jlc3NpdmVfZW5jb2RpbmdzAA%3D%3D&ccb=1-7&_nc_sid=4ea082&efg=eyJ2ZW5jb2RlX3RhZyI6Im9lcF9oZCJ9&_nc_ohc=Cfpbu-XsZBsAX87uM0c&_nc_ht=scontent.fltn3-2.fna&oh=00_AfAJY6nikmbiUBPnm8G-_aKrghM7WeryZX3b0Sk4zrarZw&oe=63A51240&_nc_rid=089394793026750

Is there a workaround?

The execution of my code is successful, but the ip is not changed. Do I need to change any other network configuration files? I use the kali system in the cloud server.

This is a sort of noob question .
Most ip rotators have addtional costs such as:
Additional EC2 instances
Elastic IP remapping
Elastic IP holding more than 1hour

so what costs does this have?
if none how does it bypass the elastic ip remapping cost.

I do not understand how to use it. Can anyone help me?