automatedtester / browsermob-proxy-py Goto Github PK

When I include this as an egg in my buildout.cfg, a bin script is not created. What would I reference in the Server initialization?

We saw 5 options after we run browsermob-proxy --help, but currently, there only implement one port option.

I think it would be better to implement all the options.

$ browsermob-proxy --help
Option                                  Description                            
------                                  -----------                            
-?, --help                              This help text                         
--address                               The address to bind to (default:       
                                          0.0.0.0)                             
--port [Integer]                        The port to listen on (default: 8080)  
--proxyPortRange [Integer]              The range of ports to use for proxies  
                                          (default: [8081, 8581])              
--ttl [Integer]                         Time in seconds until an unused proxy  
                                          is deleted (default: 0)              
--use-littleproxy [Boolean]             Use the littleproxy backend instead of 
                                          the legacy Jetty 5-based             
                                          implementation (default: true) 

Hi all,
I am working on downloading HAR from Chrome for YouTube through Selenium Python Script.
Code Snippet:

            chrome_options = webdriver.ChromeOptions()
        chrome_options.add_argument("--proxy-server={0}".format(url))
    chrome_options.add_argument("--enable-quic")

        self.driver = webdriver.Chrome(chromedriver,chrome_options = chrome_options)

           self.proxy.new_har(args['url'], options={'captureHeaders': True})   // BrowserMobProxy
       self.driver.get(args['url'])
           result = json.dumps(self.proxy.har, ensure_ascii=False)

I want QUIC to be used whenever I download HAR but when I look at the packets through Wireshark Selenium driver is using TCP only . Is there a way to force BrowserMobProxy and Chrome Driver to use QUIC?

proxy.har
File "/Users/Dropbox/Work/adops/qaserver/lib/python3.5/site-packages/browsermob_proxy-0.8.0-py3.5.egg/browsermobproxy/client.py", line 104, in har
File "/Users/Dropbox/Work/adops/qaserver/lib/python3.5/site-packages/requests/models.py", line 886, in json
return complexjson.loads(self.text, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/init.py", line 319, in loads
return _default_decoder.decode(s)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Hi

I tried to load an HTTPS url (any one like youtube), and I get this:

Page Could Not Be Loaded
Debug Info:

Status: 407
Response: Proxy Authentication Required
XID: 2131774295

I have this connection:

Selenium + FirefoxWebdriver --> BrowserMobProxy --> External Proxy --> Internet

I only get this error with HTTPS, in HTTP there's no problem.

This is a bug or I forget something?

PD. A little code to test

    from browsermobproxy.server import Server
    server = Server("/path/to/browsermob-proxy")
    server.start()
    browsermob_proxy = server.create_proxy({
    'httpProxy' : "{0}:{1}".format(proxy_host,proxy_port),
    'httpsProxy': "{0}:{1}".format(proxy_host,proxy_port_ssl),
    'sslProxy'  : "{0}:{1}".format(proxy_host,proxy_port_ssl)
    })

   browsermob_proxy.headers({'Proxy-Authorization':'Basic <proxy-authorization-code>'})
   profile.set_proxy(browsermob_proxy.selenium_proxy())
   browsermob_proxy.new_har("MY_HAR",{
       'captureHeaders' : True,
       'captureContent' : True
    })

    profile.update_preferences()
    browser = webdriver.Firefox(firefox_profile = profile,proxy = browsermob_proxy)

    browser.get('https://www.youtube.com/watch?v=a_v9oqXAAfg')

On Ubuntu 16, while running server.start(),

---------------------------------------------------------------------------
PermissionError                           Traceback (most recent call last)
<ipython-input-3-508ab784f514> in <module>()
----> 1 server.start()

/home/raja/browsermob-proxy-py/browsermobproxy/server.py in start(self, options)
    111         self.process = subprocess.Popen(self.command,
    112                                         stdout=self.log_file,
--> 113                                         stderr=subprocess.STDOUT)
    114         count = 0
    115         while not self._is_listening():

/usr/lib/python3.5/subprocess.py in __init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds)
    945                                 c2pread, c2pwrite,
    946                                 errread, errwrite,
--> 947                                 restore_signals, start_new_session)
    948         except:
    949             # Cleanup if the child failed starting.

/usr/lib/python3.5/subprocess.py in _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, restore_signals, start_new_session)
   1549                             else:
   1550                                 err_msg += ': ' + repr(orig_executable)
-> 1551                     raise child_exception_type(errno_num, err_msg)
   1552                 raise child_exception_type(err_msg)
   1553

PermissionError: [Errno 13] Permission denied

Hi. I am trying to connet to twitter.com with selenium and browsermobproxy but I keep getting the untrusted connection error and I can't get over it. I simply copy your code from the readme. It works fine but it looks like there are problems with ssl certificate with some sites not only twitter.

from browsermobproxy import Server
server = Server("path/to/browsermob-proxy")
server.start()
proxy = server.create_proxy()

from selenium import webdriver
profile  = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)


proxy.new_har("twitter")
driver.get("https://twitter.com")
proxy.har # returns a HAR JSON blob

server.stop()
driver.quit()

Hi even after doing proxy_server.stop() and proxy.close() browsermob proxy is leaving java processes running the background in Mac.
I am using the latest version.

When I try to get har file with it,some exception occur:
chrome : Message: Reached error page: about:neterror?e=nssFailure2&u=https%3A//current.sina.com.cn/theone/IO.WebPush4.localConn.html&c=UTF-8&f=regular&d=%E8%BF%9E%E6%8E%A5%20current.sina.com.cn%20%E6%97%B6%E5%8F%91%E7%94%9F%E9%94%99%E8%AF%AF%E3%80%82%0A%0A%E5%AF%B9%E7%AD%89%E7%AB%AF%E6%8A%A5%E5%91%8A%E5%85%B6%E9%81%87%E5%88%B0%E4%B8%80%E4%B8%AA%E5%86%85%E9%83%A8%E9%94%99%E8%AF%AF%E3%80%82%0A%0A%E9%94%99%E8%AF%AF%E4%BB%A3%E7%A0%81%3A%20%3Ca%20id%3D%22errorCode%22%20title%3D%22SSL_ERROR_INTERNAL_ERROR_ALERT%22%3ESSL_ERROR_INTERNAL_ERROR_ALERT%3C/a%3E%0A
firefox:ERR_SSL_PROTOCOL_ERROR

how can I deal with it? Help me !

server = Server('bin/browsermob-proxy', {'port':1337})
server.start()
proxy = server.create_proxy()

create_proxy is failing, reporting that "ValueError: No JSON object could be decoded".
If I go to localhost:1337/proxy in my browser, I can download the file; however, when Client goes in it's initialization, it gets a 500

<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
        <title>
        Error 500 Exception [MultiException - "Multiple exceptions"] thrown by event method [public com.google.sitebricks.headless.Reply net.lightbody.bmp.proxy.bricks.ProxyResource.newProxy(com.google.sitebricks.headless.Request) throws java.lang.Exception]
        at net.lightbody.bmp.proxy.jetty.http.HttpServer.doStart(HttpServer.java:669)
        (See below for entire trace.)
        </title>
    </head>
    <body>
        <h2>HTTP ERROR: 500</h2>
        <p>Problem accessing /proxy. Reason:
<pre>    Exception [MultiException - "Multiple exceptions"]
    thrown by event method [
        public com.google.sitebricks.headless.Reply
        net.lightbody.bmp.proxy.bricks.ProxyResource.newProxy(com.google.sitebricks.headless.Request)
    throws java.lang.Exception]
    at net.lightbody.bmp.proxy.jetty.http.HttpServer.doStart(HttpServer.java:669)
(See below for entire trace.)
</pre>
        </p>
        <hr /><i><small>Powered by Jetty://</small></i>
    </body>
</html>

What does this mean? Am I doing something wrong? I am not using the default port because I want to run this on jenkins, which is using port 8080. Everything works locally using port 8080.

Traceback (most recent call last):
File "getTrending.py", line 7, in
api = TikTokapi()
File "C:\Users\Sagar\Downloads\TikTok-Api-master\tiktok.py", line 18, in ini
t
server.start()
File "C:\Users\Sagar\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\browsermobproxy\server.py", line 122, in start
raise ProxyServerError(message)
browsermobproxy.exceptions.ProxyServerError: The Browsermob-Proxy server process
failed to start. Check <_io.TextIOWrapper name='C:\Users\Sagar\Downloads\Ti
kTok-Api-master\server.log' mode='w' encoding='cp1252'>for a helpful error mess
age.

using python 3 on windows7...

i am passing in a dict with the keys log_file and log_path but it still sets the log file at the root of the project dir

In your example, I think proxy = s.create_proxy should be proxy = server.create_proxy instead, no?

Regardless, thanks for this!

-tk

I passed this as params to the client constructor.

client = Client(PROXY_SERVER_ENDPOINT, params={'httpProxy': '172.17.0.12:5566', 'httpsProxy': '172.17.0.12:5566' })
Here is what I see in the logs:

[WARN 2017-08-24T15:11:27,676 net.lightbody.bmp.BrowserMobProxyServerLegacyAdapter] (qtp2053628870-18) Chained proxy support through setOptions is deprecated. Use setUpstreamProxy() to enable chained proxy support.
Connect timeouts from the configured browsermobproxy proxy.

Hi All,
I am using proxy to connect to net so when I am trying to start browsermob proxy server it is unable to get started.How to run the server when I am behind the proxy?

server=Server('path to browsermob-proxy')
server.start()
proxy=server.create_proxy()

It would be great if we could customize the SSL cert and CA during runtime.

Similar to the java examples provided: https://github.com/lightbody/browsermob-proxy/tree/master/mitm#using-a-custom-certification-authority

Example usage could be:

from browsermobproxy import Server
server = Server("path/to/browsermob-proxy")
server.start(sslCert = "/dir/sslKey.p12", sslCA = "/dir/sslCA.crt")

The python's version is 2.7. I wanted to install browsermob-proxy by the utility of pip,but there were exceptions when ''pip2.7 install browsermobproxy" executed. I don't where goes wrong?The whole process is below

root@nokia:~# pip2.7 install browsermobproxy
Collecting browsermobproxy
Using cached https://files.pythonhosted.org/packages/d1/35/5dd6dabf153874debbe13d7d10ce5a72e738ce45170c8fced55761a4480c/browsermobproxy-0.1.1.tar.gz
Building wheels for collected packages: browsermobproxy
Running setup.py bdist_wheel for browsermobproxy ... error
Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-install-hUREti/browsermobproxy/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d /tmp/pip-wheel-2GVyy7 --python-tag cp27:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/browsermobproxy
copying browsermobproxy/init.py -> build/lib.linux-x86_64-2.7/browsermobproxy
copying browsermobproxy/cli.py -> build/lib.linux-x86_64-2.7/browsermobproxy
installing to build/bdist.linux-x86_64/wheel
running install
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-hUREti/browsermobproxy/setup.py", line 47, in
'browsermobproxy = browsermobproxy.cli:cli',
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib/python2.7/dist-packages/wheel/bdist_wheel.py", line 215, in run
self.run_command('install')
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/tmp/pip-install-hUREti/browsermobproxy/setup.py", line 20, in run
raise Exception("You probably meant to install and run browsermob-proxy")
Exception: You probably meant to install and run browsermob-proxy

Failed building wheel for browsermobproxy
Running setup.py clean for browsermobproxy
Failed to build browsermobproxy
Installing collected packages: browsermobproxy
Running setup.py install for browsermobproxy ... error
Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-install-hUREti/browsermobproxy/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-_UMtZY/install-record.txt --single-version-externally-managed --compile:
running install
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-hUREti/browsermobproxy/setup.py", line 47, in
'browsermobproxy = browsermobproxy.cli:cli',
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/tmp/pip-install-hUREti/browsermobproxy/setup.py", line 20, in run
raise Exception("You probably meant to install and run browsermob-proxy")
Exception: You probably meant to install and run browsermob-proxy

----------------------------------------

Command "/usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-install-hUREti/browsermobproxy/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-_UMtZY/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-hUREti/browsermobproxy/
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

So the upper information is that I've encountered,do anyone have suggestions on this, thank you !

when i deploy to win server i see error
"Another server running on this port?\n%s..." % content[:512]) Exception: Could not read Browsermob-Proxy json Another server running on this port?"
or "An existing connection was forcibly closed by the remote host"

please help me how to resolve this issues.

thanks

When the server is started it does so using subprocess.Popen('browsermob-proxy.bat')

The result process's pid on windows is the cmd.exe process that controls the bat file, but I don't think Windows is linking the java.exe which the bat file spawns.

Calling server.stop() kills the cmd.exe, but a java.exe is left hanging around at about 35MB.

This may be a problem with windows not killing child process correctly and it is apparently not possible to get the java pid in the bat file without some actual windows-style grep for java.exe.

There is a method to delete all blacklisted regex:
Clears all URL patterns from the blacklist DELETE /proxy/[port]/blacklist
Should add new method for this.

empty_dns_cache and clear_dns_cache in client.py do the same thing.

I'm not sure if there's a reason for this, but I thought I'd point it out anyway.

I tried response interceptor as per your test method. But it only returns 200 status code. and doesn't change response content.
Also, I saw proxy server change at https://github.com/lightbody/browsermob-proxy project.
are you planning to update your project with littleproxy instead or Jetty?

def test_response_interceptor_with_parsing_js(self):
"""
/proxy/:port/interceptor/response
"""
js = 'alert("foo")'
status_code = self.client.response_interceptor(js)
assert(status_code == 200)

When I use profile.set_proxy(proxy.selenium_proxy()), it rasies warning that

This method has been deprecated. Please pass in the proxy object to the Driver Object",
DeprecationWarning

Especially, after I upgrade my Firefox to 63.0, it cannot run totally.
Do you guys know how to use another way to deploy this proxy?
Should I set proxy when I call selenium.webdriver.Firefox(proxy=?)
What parameter should I input?

It would be great if some of the latest bug fixes could make their way to PyPi. #71 would be especially nice to have available via pip.

Cheers!

I run selenium hub with firefox and proxy browsermob by bwowk/browsermob-proxy

1. all in docker-compose.yml file

hub:
  image: selenium/hub
  ports:
    - 4444:4444
firefox:
  image: selenium/node-firefox
  links:
    - hub
    - proxy
proxy:
  image: bwowk/browsermob-proxy
  ports:
    - 58080:8080

2. Create a Dockerfile with pip install selenium browsermob-proxy and link hub and proxy

FROM python:3-alpine
RUN pip install selenium browsermob-proxy

3. in terminal write

from selenium import webdriver
from browsermobproxy import Client

proxyClient = Client("proxy:8080")

proxy = proxyClient.selenium_proxy()

caps = webdriver.DesiredCapabilities.FIREFOX.copy()
proxy.add_to_capabilities(caps)

driver = webdriver.Remote(command_executor='http://hub:4444/wd/hub', desired_capabilities=caps)


proxyClient.new_har()

driver.get('http://google.com')

for ent in proxyClient.har['log']['entries']:
    print(ent['request']['url'])

proxyClient.close()
driver.quit()

in debug i see that var caps have are proxy key and in proxy server i see that new proxy is registered

Result

But firefox does not use this proxy address. That is why i can not get logs.

proxyClient.har['log']['entries'] is null []

in docker image

It's great that I can get the har information, however, what do I do with it? It isn't going to a file I can upload anywhere?

I try to:
import browsermobproxy

but have:
Traceback (most recent call last):
File "C:/Users/Myalee/Desktop/Prog/JetBrains PyCharm 3.4.1/Json/selenium_second.py", line 1, in
import browsermobproxy
File "C:\Python34\lib\site-packages\browsermobproxy__init__.py", line 3, in
from server import Server
ImportError: No module named 'server'

It's work fine in python 2.7, but not in 3.4

Hello,

I would like to propose to support starting a new proxy configured to use a http proxy. At https://github.com/lightbody/browsermob-proxy we can see , that we can :
[~]$ curl -X POST http://localhost:9090/proxy?httpProxy=yourproxyserver.com:8080
{"port":9091}

I propose the next simple patch:

diff --git a/browsermobproxy/client.py b/browsermobproxy/client.py
index 20827f3..f9302e4 100644
--- a/browsermobproxy/client.py
+++ b/browsermobproxy/client.py
@@ -4,15 +4,17 @@ import json


 class Client(object):
-    def __init__(self, url):
+    def __init__(self, url,params={}):
         """
         Initialises a new Client object


         :param url: This is where the BrowserMob Proxy lives
+        :param params: URL query (for example httpProxy and httpsProxy vars)
         """
         self.host = "http://" + url
-        resp = requests.post('%s/proxy' % self.host, urlencode(''))
+        urlparams="?"+urlencode(params) if params else ""
+        resp = requests.post('%s/proxy' % self.host + urlparams)
         jcontent = json.loads(resp.content)
         self.port = jcontent['port']
         url_parts = self.host.split(":")
diff --git a/browsermobproxy/server.py b/browsermobproxy/server.py
index 7857a06..e8e8e37 100644
--- a/browsermobproxy/server.py
+++ b/browsermobproxy/server.py
@@ -85,12 +85,14 @@ class Server(object):
         """
         return "http://localhost:%d" % self.port

-    def create_proxy(self):
+    def create_proxy(self,params={}):
         """
         Gets a client class that allow to set all the proxy details that you
         may need to.
+        :param params: Dictionary where you can specify params \
+                  like httpProxy and httpsProxy
         """
-        client = Client(self.url[7:])
+        client = Client(self.url[7:],params)
         return client

     def _is_listening(self):

If you prefer I fork and make a pull request, please tell me ( I think it is better that smaller changes like this is ok with a patch inline a comment)

Do you think it is appropiate to include in your next release?

Thanks

Exception in thread "main" org.openqa.selenium.WebDriverException: Unable to bind to locking port 7054 within 45000 ms
Build info: version: '2.42.2', revision: '6a6995d', time: '2014-06-03 17:42:30'
System info: host: 'DOGIPARTHY', ip: '192.168.1.124', os.name: 'Windows 8.1', os.arch: 'amd64', os.version: '6.3', java.version: '1.8.0_121'
Driver info: driver.version: FirefoxDriver
at org.openqa.selenium.internal.SocketLock.lock(SocketLock.java:98)
at org.openqa.selenium.firefox.internal.NewProfileExtensionConnection.start(NewProfileExtensionConnection.java:84)
at org.openqa.selenium.firefox.FirefoxDriver.startClient(FirefoxDriver.java:246)
at org.openqa.selenium.remote.RemoteWebDriver.(RemoteWebDriver.java:114)
at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:193)
at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:186)
at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:182)
at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:95)
at selprgs.Gmaillogin.main(Gmaillogin.java:17)

        Server server = new Server(@"D:\NDrive\GR\MiscTest\BrowserMob-Tests\browsermob-proxy-master\browsermob-dist\src\main\scripts\browsermob-proxy.bat");
        server.Start();

        Client client = server.CreateProxy();
        client.RemapHost("host", "ip address");
        client.NewHar("google");

        var seleniumProxy = new Proxy { HttpProxy = client.SeleniumProxy };
        var profile = new FirefoxProfile();
        profile.SetProxyPreferences(seleniumProxy);
        // Navigate to the page to retrieve performance stats for
        var driver = new FirefoxDriver(profile);
        driver.Navigate().GoToUrl("http://google.com");

        // Get the performance stats
        HarResult harData = client.GetHar();

        AutomatedTester.BrowserMob.HAR.Log log = harData.Log;
        AutomatedTester.BrowserMob.HAR.Entry[] entries = log.Entries;
        foreach (var entry in entries)
        {
            AutomatedTester.BrowserMob.HAR.Request request = entry.Request;
            var url = request.Url;
            var time = entry.Time;
            Console.WriteLine("Url: " + url + " - Time: " + time);
        }
        driver.Quit();
        client.Close();
        server.Stop();

Message : {"Can not connect to BrowserMob Proxy"}
at AutomatedTester.BrowserMob.Server.Start() in D:\NDrive\GR\MiscTest\BrowserMob-Tests\AutomatedTester.BrowserMob-master\AutomatedTester.BrowserMob\Server.cs:line 54
at MobTester.Program.Main(String[] args) in D:\NDrive\GR\MiscTest\BrowserMob-Tests\MobTester\MobTester\Program.cs:line 14
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
WHAT IS WRONG IN THIS? PLEASE GET BACK TO ME ASAP

Several fixes and new features are on master. Please tag a new version and release on pypi. Ideally it would be great to see versions released more often. Thanks!

When I do a plain-vanilla test, using this code:

from browsermobproxy import Server
server = Server("venv/bin/browsermob-proxy-2.1.4/bin/browsermob-proxy", options={'port':8008})
server.start()
proxy = server.create_proxy()

from selenium import webdriver
profile  = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)

proxy.new_har("google", options={"captureContent":True, "captureBinaryContent":True})
driver.get("http://www.google.co.uk")

server.stop()
driver.quit()

Some responses have response.content.text base64 encoded but doesn't decode to what it should. In this example, what should be an HTML page decodes to gibberish: https://gist.github.com/n-kb/8b8818230c54be998007ee855e037404#file-google-har-L191

Because the problem only arises with text and not with images, I'm hypothesizing that the issue comes from this (from RFC1341): "A CRLF sequence in base64 data should be converted to a quoted-printable line break, but ONLY when converting text data" but I'm not familiar at all with these things.

Any idea how this could be solved?

Hi all,
Is there a way to capture HAR for a youtube video at the end of its duration?Through the following code it is capturing the HAR at the start of the video itself.What modifications need to be done if I don't know the duration in advance.

server=Server('path to browsermob-proxy')
server.start()
proxy=server.create_proxy()
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--proxy-server={0}".format(proxy.proxy))
browser = webdriver.Chrome(executable_path="/home/sivakesava/chromedriver",chrome_options = chrome_options)
proxy.new_har("CS")
browser.get("https://www.youtube.com/watch?v=NFQeItqTfZk")
har=proxy.har
print har
browser.quit()
server.stop()

My test environment is under a corporate proxy ("proxy.ptbc.std.com:2538").I want to open a particular video on YoTube for a period of time (eg 200 seconds) and capture the har file for each visit, the process is repeated several times for a massive test. I have tried different examples found here but the firefox / chrome browsers do not connect to the internet because they are behind the proxy.

How can run "python-selenium + browsermobproxy" behind a corporate proxy and capture the har file for each instance.

Example code:

from browsermobproxy import Server
server = Server("C:\\Utility\\browsermob-proxy-2.1.4\\bin\\browsermob-proxy")
server.start()
proxy = server.create_proxy()

from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--proxy-server={}".format(proxy.proxy))
driver = webdriver.Chrome(options=chrome_options)

proxy.new_har("google")
driver.get("http://www.google.co.in")
proxy.har # returns a HAR JSON blob

server.stop()
driver.quit()

Any help would be appreciated

import psutil
import time
from browsermobproxy import Server

server = Server('../resources/browsermob-proxy-2.1.4/bin/browsermob-proxy')
server.start()
proxy = server.create_proxy()
print(proxy.proxy)
proxy.headers({'test': 'close'})
#proxy.headers({'Connection': 'close'})

I'm using code above to start a proxy server for my browser to add/modify headers. Unfortunately, I can't do this with "Connection" header. Is there any way to do this?

I am using Windows + Python + Selenium (latest) + Browsermob-proxy-py + Chrome, FF, IE.

After following the installation instructions and the example provided by the documentation, my code looks like this:

@pytest.fixture(scope="session", autouse=True)
def selenium_proxy():
    proxy = Client("localhost:9090")
    proxy.trustAllServers(True)
    proxy.basic_authentication("", HTACCESS_USERNAME, HTACCESS_PASSWORD)

    yield proxy.selenium_proxy()

    proxy.close()

# I defined this method in browsermob-proxy-py/client.py 
def trustAllServers(self, value):
    r = requests.post(url='%s/proxy' % (self.host),
                      data=json.dumps({'trustAllServers': value}),
                      headers={'content-type': 'application/json'})
    return r.status_code

I am running the browsermob proxy standalone version like this:
java -jar browsermob-dist-2.1.4.jar --port 9090

I did this by following the official browsermob-proxy documentation, which can be found here

Printing the response code for both methods: trustAllServers (Returns 200) and basic_authentication (Returns 404) <- Why?

The main issue is: when I try to load an URL with selenium the browser will say that the page cannot be loaded. If I run my tests with internet explorer, some weird unwanted windows configurations will take place and I will have to manually deactivate the LAN Proxy configuration in the Internet Options in order to recover my internet connection.

The console logs informs two main Problems:

...
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
...

and

...
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching <host> found.
...

I've been reading for hours and the solution I always end up finding is trusting all servers, which "I am already doing" with the trustAllServers method I defined. Maybe I am doing something wrong?

Edit: I am trying to do something like this
https://www.softensity.com/browsermob-proxy-qa/

Is there a way to do this?

I'm on mac, sever.start() raise error: Permission denied: '/server.log'.
It's because
log_path_name = os.path.join(log_path, os.path.sep, log_file) in start method.
log_path_name return '/server.log'.

I change code, log_path_name = os.path.join(log_path, log_file).
then it's ok

Hi everyone,

I'm trying to run a PhantomJS webdriver with BrowserMob-proxy using Selenium in Python. It's the first time for me and the basic "getting started" code provided here is already returning error.

from browsermobproxy import Server
server = Server('browsermob-proxy-2.0.0\\bin\\browsermob-proxy') #I also tried browsermob-proxy-2.1.0-beta-6 but the error stays the same
server.start()
proxy = server.create_proxy()

I get this error:

>>> server.start()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "...\Python27\lib\site-packages\browsermobproxy\server.py", line 104, in start
    raise Exception("Can't connect to Browsermob-Proxy")
Exception: Can't connect to Browsermob-Proxy

I downloaded the Python library simply with pip install browsermob-proxy. (version 0.7.0)

I downloaded both versions of BrowserMob proxy from https://github.com/lightbody/browsermob-proxy/releases

I'm using Python 2.7.11 on Windows 10.

Any idea what I'm doing wrong? It seems like I'm stuck at the very start and I'm clueless as I don't have virtually any experience with this.

Any help would be much appreciated!

I am using a MacBook Air with osX 10.13.5, prowsermob-proxy 2.1.4 (python part 0.8.0), selenium 3.13.0, firefox 61.0.1 and python 2.7.10, and trying to run essentially the example given in the README:

from browsermobproxy import Server
server = Server(path="./browsermob-proxy-2.1.4/bin/browsermob-proxy")

server.start()
proxy = server.create_proxy()

from selenium import webdriver
profile  = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)


proxy.new_har("google")
driver.get("http://www.google.co.uk")
print(proxy.har) # returns a HAR JSON blob

server.stop()
driver.quit()

However, printing the proxy.har does not see to work. I get the following error

Traceback (most recent call last):
  File "example.py", line 16, in <module>
    print(proxy.har) # returns a HAR JSON blob
  File "/Users/adietz/Work/20_BSP/Jenkins/Invest/Browsermobproxy/venv_mac/lib/python2.7/site-packages/browsermobproxy/client.py", line 104, in har
    return r.json()
  File "/Users/adietz/Work/20_BSP/Jenkins/Invest/Browsermobproxy/venv_mac/lib/python2.7/site-packages/requests/models.py", line 896, in json
    return complexjson.loads(self.text, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Maybe this is a bug on the python implementation?

In my environment of Window8.1, browsermob-proxy-py will be dead after running a time. Is there others met this codition?

browsermob-proxy-2.1.4, python 3.7
Tips: DeprecationWarning: This method has been deprecated. Please pass in the proxy object to the Driver Object
profile.set_proxy(proxy.selenium_proxy())
What should I do
Thank‘s!!!

I try to test GA events in my app, using browsermob. I run my webdriver with options (code sample from my setUp() method)
self.client = Client("localhost:2000") self.proxy = Proxy({ 'proxyType': ProxyType.MANUAL, 'httpProxy': self.client.proxy, 'ftpProxy': self.client.proxy, 'sslProxy': self.client.proxy, 'noProxy': '' # set this value as desired }) self.driver = webdriver.Firefox(proxy=self.proxy)
Every event on http works fine and browsermob writes it to HAR right, but when I try to do the same on https, some events seem not to happen at all. If I start browser without a proxy, they work fine.

I am using MacOS 10.12.6. with the preinstalled Python 2.7.10. and browsermobproxy 2.1.4.

browsermobproxy-py is version 0.8.0 installed via pip.

I trying to create har files with the following code (which mostly works):

from browsermobproxy import Server
server = Server("browsermob-proxy/bin/browsermob-proxy")
server.start()
proxy = server.create_proxy()

from selenium import webdriver

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--allow-file-access-from-files");
chrome_options.add_argument("disable-infobars");
chrome_options.add_argument("--proxy-server={0}".format(proxy.proxy))
driver = webdriver.Chrome(chrome_options = chrome_options)

proxy.new_har('someString')
driver.get("https://www.google.de/")
result = proxy.har # returns a HAR JSON blob

server.stop()
driver.quit()

If I understand the documentation correctly then the first argument to new_har() is an arbitrary string to disambiguate multiple har files if I want to record har files from multiple urls in one go. However whenever I supply that argument I get the following output:

Traceback (most recent call last):
File "test.py", line 23, in
result = proxy.har # returns a HAR JSON blob
File "build/bdist.macosx-10.12-intel/egg/browsermobproxy/client.py", line 104, in har
File "/Library/Python/2.7/site-packages/requests-2.18.4-py2.7.egg/requests/models.py", line 892, in json
return complexjson.loads(self.text, **kwargs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/init.py", line 338, in loads
return _default_decoder.decode(s)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

The return value from new_har contains a http status code, and I can see that when I provide the argument I get a 500, which i guess is why I don't have any JSON to be decoded.

If I do not supply any arguments to new_har() I get a valid har file.

From looking at the function definition for new_har() I do not understand why this is happening.

I also see a ton of warnings and error messages in server.log ("illegal reflective access operation","Failed to start Jetty server"), but that happens in both cases, so I am not sure if this is related to my problem.

At the moment I am not sure if I there is a problem with my code or my setup, or if there is a bug in browsermobproxy-py.

I would appreciate any guidance on this.

Hey,
I'm getting the following error on server.start()

Exception: Can't connect to Browsermob-Proxy

I have the following:

from browsermobproxy import Server
server = Server("/Users/username/Desktop/browsermob-proxy/bin/browsermob-proxy")
server.start()
proxy = server.create_proxy()

I am on a Mac with 10.10.1. I downloaded the browsermob-proxy files from http://bmp.lightbody.net/

I am able to successfully start the proxy server manually with /Users/username/Desktop/browsermob-proxy/bin/browsermob-proxy -port 9090

My apologies if I'm missing something obvious

Hi,

I'm getting this on Macosx:
File "/Users/guandalf/PycharmProjects/qatools/ENV/lib/python2.7/site-packages/browsermobproxy/server.py", line 48, in start
stderr=subprocess.STDOUT)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 679, in init
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1228, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

The missing file is the 'sh' in the init on line 35.

Putting "/bin/sh" or "/bin/bash" fixes the "No such file or directory" but triggers:
self.server.start()
File "/Users/guandalf/PycharmProjects/qatools/ENV/lib/python2.7/site-packages/browsermobproxy/server.py", line 55, in start
raise Exception("Can't connect to Browsermob-Proxy")
Exception: Can't connect to Browsermob-Proxy

instead.

I've not investigated further.

Is it possible to use this with webdriver.Ie()

I'm trying to load webpages, wait until they've finished loading, scroll to their bottoms, and wait for until they've finished loading. Because the load times are indeterminate, I am using "browsermob-proxy.Client.wait_for_traffic_to_stop" to wait for traffic to stop before scrolling. I am using Python via Anaconda and Jupyter-Notebook. Given a quiet period of 0.1 seconds and a timeout of 20 seconds, my code consistently prints below my Jupyter-Notebook cell a duration of about 19.9 seconds, which is the difference between the wait time limit and the length of the quiet period, which I think is basically the amount of time wait_for_traffic_to_stop is allowed to wait.

That all being said, in Anaconda Prompt, I executed the command "pip install git+https://github.com/AutomatedTester/browsermob-proxy-py.git" to try to install browsermob-proxy 2.0. Despite browsermob-proxy-py's documentation seeming to imply that my command would install browsermob-proxy 2.0, my command installed browsermob-proxy 0.8.0. Given a post in the thread at "lightbody/browsermob-proxy#212", it seems that the above time-out issue was fixed in browsermob-proxy 2. How do I install browsermob-proxy 2 or later via Anaconda Prompt?

From https://github.com/webmetrics/browsermob-proxy:

POST /proxy/[port]/headers - Set and override HTTP Request headers. For example setting a custom User-Agent.
POST data should be json encoded set of headers (not url-encoded)

This could allow us to set a header for basic authentication.