Comments (4)
Hey, thanks for your effort!
I just added FLATHUNTER_IS24_COOKIE
as an environment variable and tested whether it works using the urls from the config.yaml
, thus a combination of both config.yaml
and environment variables. Everything now works as expected.
I close the issue now.
from flathunter.
From a quick glance, having the IS24-cookie or other crawler-specific settings read from an ENV variable is currently not possible without some code overhaul or refactoring.
Your configuration seems fine. Can you try changing the message format in the config.yaml to check whether the file gets read?
from flathunter.
Hey @pomeloy, I did some trial & error and the solution seems to be the following: Once you define the urls inside the config.yaml
instead of the docker-compose.yaml
, the script seems to look for the information in the config.yaml
, and then also used the cookie from there.
As the issue technically still persists, I would leave it open for now. However, with my fix, it is solvable.
from flathunter.
I've added the option to override the IS24 cookie with an environment variable in #587 - you can now set FLATHUNTER_IS24_COOKIE
. In general it should be fine to combine the config file with the environment variables. If the environment variables are present, they will override the settings in the file. And if no config file is present, everything should be read from the environment.
I've added some tests for this. If you find that the functionality isn't working as expected with the latest code on main
, please let me know. If not, please close the ticket as resolved.
Thanks for the report!
from flathunter.
Related Issues (20)
- Should we move the idMaintainer files? HOT 1
- not receiving any listings from Immoscout24 HOT 3
- "NoneType" in title_el.get("href") when scraping kleinanzeigen HOT 3
- Time restrictions HOT 1
- Replace Google Chrome with Chromium in Dockerfile HOT 6
- WebDriverException: Message: unknown error: session deleted because of page crash [docker]
- urllib.error.HTTPError: HTTP Error 404: Not Found HOT 7
- Running on Unraid/Docker HOT 2
- chromium cannot determine loading status running in docker
- urllib.error.HTTPError: HTTP Error 404: Not Found
- Trying to install on Mac OS: Mapping values are not allowed here HOT 3
- yaml.parser.ParserError: expected '<document start>', but found '<block mapping start>' HOT 3
- No flats coming through on Immoscout HOT 30
- vrimmo crawler not working
- Bot domain invalid HOT 4
- `Config file loaction must be specified` after having pulled commits made after 29.04.24 HOT 3
- Current state of solving `IS24 bot detection has identified our script as a bot - we've been blocked` HOT 1
- Problem with `undetected_chromedriver` HOT 1
- ImmoScout24 - Imperva (resee84) can be solved
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flathunter.