HTTP Error 403: Forbidden about finam-export HOT 16 OPEN

Workaround with selenium

`
def fetcher (url, lines=False):
print ('>>>', url)
driver.get(url)
res = WebDriverWait(driver, 100).until( lambda driver: driver.find_element(By.XPATH, "//*").get_attribute('outerHTML'))
if lines:
res = res.split('\n')
return res

...

exporter = Exporter(fetcher = fetcher)

`

from finam-export.

@Celeevo I used code above and created new version of the lib. You can install it with: pip install git+https://github.com/am1ter/finam-export
I also created pull request, but before you can use it, you have to wait until @ffeast review it and publish.

from finam-export.

@am1ter Thanks a lot, it works now!

from finam-export.

Yes it seems so. Would be happy to see a pull request because I don't have spare time to figure it out

from finam-export.

I tryed to replace user-agent but error stil repduces. So problen is in securuty policy of finam api.

from finam-export.

Yes it seems so. Would be happy to see a pull request because I don't have spare time to figure it out

I tryed to replace user-agent but error stil repduces. So problen is in securuty policy of finam api.

yes, i did the same and got same result... it seems that security protection distinguishes somehow differently (not by user-agent). i stopped attemts to figure it out and starting to find another way to get stocks quotes

from finam-export.

Interesting. Pull request?

from finam-export.

No nesessary, just copy and use. Also this way contains issue with encoding for cyrilyc symbols.

update

def fetcher (url, lines=False):
print ('>>>', url)
driver.get(url)
res = WebDriverWait(driver, 100).until( lambda driver: driver.find_element(By.XPATH, "//*").get_attribute('outerHTML'))
if lines:
res = res.encode('cp1252').decode('cp1251') # <
res = res.split('\n') # <
return res

UPD 28.01.23: Working solution

from finam-export.

Добрый день,
если у кого-то получилось исправить доступ с помощью описанной выше заплатки, поделитесь пжлст для тех кто в танке - куда копировать новый fetcher и что дополнительно нужно импортировать (driver, WebDriver..., etc)

from finam-export.

Complete workaround, contains downloading data.

def fetcher (url, lines=False):
print ('>>>', url)
driver.get(url)
res = WebDriverWait(driver, 15).until( lambda driver: driver.find_element(By.XPATH, "//*").get_attribute('outerHTML'))
if lines:
res = res.encode('cp1252').decode('cp1251')
res = res.split('\n')
return res

def fetcher_download(url, readlines = False):
print ('>>>', url)
r = requests.get(url, headers = {'User-Agent':'Mozilla 8'}, stream=True)
return r.content.decode('utf8')
if lines:
return r.content.decode('utf8').split('\n\r')
else :
return r.content.decode('utf8')

exporter = Exporter(fetcher = fetcher)
exporter._fetcher = fetcher_download

I think this solution is temorary, until Finam changes an autorisation of the web downloading page.
Different way to reanimate project is research workarount to extract token and pass it to request (to awoid webdriver...)
So its looks like strange, because they are produces acess to stock data throw web page(free), and connector with .net lib (login:password), but dont provides api...

from finam-export.

Thanks @ffeast fo the extensible and clear code.

from finam-export.

Hi
@am1ter commented on the PR, please check. The overall takeaway is that now it looks too complex with 2 different fetchers and all classes being aware of that

from finam-export.

Hi @am1ter commented on the PR, please check. The overall takeaway is that now it looks too complex with 2 different fetchers and all classes being aware of that

Hello @ffeast ! Thank you for a quick response. I don't see any comment to the PR at the moment. I am not sure why.

On one hand, I agree with you that it is not the best decision to have two independant fetchers in Exporter class. By the other hand, it gives a user an option to declare using standart urllib fetcher manually in the case if token-protection will be disabled one day.
Also, as I see there is an alternative way to use second fetcher. For example, try to use urllib-based fetcher and in the case of the error use webdriver-based fetcher. However, I see some downsides in this soltions as well.

What do you think? Or maybe you can even suggest more effective solution?

from finam-export.

Hi
Could you please check if any of the comments are seen on this page or here?

from finam-export.

Hi Could you please check if any of the comments are seen on this page or here?

Hello @ffeast! Unfortunately, not. And it is quite strange. Look at the screenshot 1 and screenshot 2.

from finam-export.

@ffeast Hello again! So, since I don't see any comment on the PR page maybe you can post it here?

from finam-export.