kishanbagaria / dabot Goto Github PK
View Code? Open in Web Editor NEW๐ค CLI tool to automate stuff on DeviantArt.com
License: MIT License
๐ค CLI tool to automate stuff on DeviantArt.com
License: MIT License
I have written a patch for dAbot in #22 that allows you to bypass the bot detection.
What you see with the message Please confirm you are human or similar errors, is actually part of a security measure by PerimeterX that internally uses reCAPTCHA to combat bots on DeviantArt.
This is a recent security measure introduced by Wix -- the parent company of DeviantArt -- that triggers on a score-based basis, which means you could have either made more realistic requests, or slowed down the bot enough that you wouldn't trigger the automated detection.
However, this detection seems to get more aggressive each day, and you can't do anything but to bypass it using a reCAPTCHA token.
While trying to debug my own bot, dADroid, I ran into the similar frustrating obstacle, so using a bit of reverse-engineering and looking into the PerimeterX code, I devised a simple method to bypass this piece of crap by providing the token directly to the bot.
I've ported my solution to python for dAbot, so users could also bypass the message.
You need to get a response token by solving a reCAPTCHA that has been originated on a DeviantArt page.
I have written a userscript to make this process easier, but in the meantime, you can use the manual method to gain the token.
โ NOTE: You need the edits in #22 in order to make this work.
These instructions are written for Chrome on Windows. You need to adapt it for your own browser.
https://www.deviantart.com/users/wrong-password?url=%2fusers%2flogin&uuid=perimeterx-token&vid=
window.handleCaptcha = function(token) { document.write("<pre>" + token + "</pre>"); }
WORK-IN-PROGRESS I am developing a userscript that allows you to acquire the token with just one click.
Please post comments to this issue if you need additional help, or encounter any errors.
I'd be happy to hear your feedback if this method proves to be successful and solves your issue!
Over the last months running dAbot on linux i ran into an issue regarding usernames that have hyphens at the beginning or end of their usernames. The script halted with the console saying this:
Retrying... ConnectionError(MaxRetryError("HTTPSConnectionPool(host='shaft-.deviantart.com', port=443): Max retries exceeded with url: /badges/llama/ (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f9fba1693d0>: Failed to establish a new connection: [Errno -2] Name or service not known',))",),)
After investigating the issue i found out that deviantArt's subdomain system for usernames is actually producing invalid subdomains according to RFC1035 ("They must start with a letter, end with a letter or digit, and have as interior characters only letters, digits, and hyphen.").
While Windows doesn't seem to have an issue with resolving domains like these, linux does.
As a simple solution to this issue while preserving the ability to theoretically give users with invalid subdomains llamas i'd like to propose a try-catch in the loops (https://github.com/KishanBagaria/dAbot/blob/master/dAbot/dAbot.py#L697) which would ignore the "invalid" users and continue with the next one without halting.
Hi!
Today i have this problem:
I reinstaling python, dabot and trying on another machine. Thanks for help.
Traceback (most recent call last):
File "d:\python27\lib\runpy.py", line 174, in run_module_as_main
"main", fname, loader, pkg_name)
File "d:\python27\lib\runpy.py", line 72, in run_code
exec code in run_globals
File "D:\Python27\Scripts\dAbot.exe_main.py", line 9, in
File "d:\python27\lib\site-packages\dAbot\dAbot.py", line 811, in main
init()
File "d:\python27\lib\site-packages\dAbot\dAbot.py", line 671, in init
load_data()
File "d:\python27\lib\site-packages\dAbot\dAbot.py", line 594, in load_data
LlamaTransactions = read_llama_transactions()
File "d:\python27\lib\site-packages\dAbot\dAbot.py", line 567, in read_llama_transactions
return set(json.load(f))
File "d:\python27\lib\json_init.py", line 287, in load
return loads(fp.read(),
EOFError: compressed file ended before the logical end-of-stream was detected
When running 'llama give random deviants' it throws this error:
Traceback (most recent call last):
File "C:\Users\User\AppData\Local\Programs\Python\Python38-32\Scripts\dAbot-script.py", line 11, in
load_entry_point('dAbot==2019.9.24.2', 'console_scripts', 'dAbot')()
File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\dAbot\dAbot.py", line 812, in main
run()
File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\retrying.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\retrying.py", line 206, in call
return attempt.get(self._wrap_exception)
File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\six.py", line 703, in reraise
raise value
File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\dAbot\dAbot.py", line 680, in run
while True: give_llama_to_deviant(get_random('deviant'))
File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\dAbot\dAbot.py", line 197, in get_random
parsed = urlparse(get_redirected_url(url['random'] + what))
File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\dAbot\dAbot.py", line 194, in get_redirected_url
return dA.head(url).headers['Location']
File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\requests\structures.py", line 52, in getitem
return self._store[key.lower()][1]
KeyError: 'location'
All features of dAbot have stopped working for me. Now, when attempting to login after "downloading login page" it literally outputs all html, css, js data of the dA page into the terminal and quits with the following:
Traceback (most recent call last):
File "dAbot.py", line 803, in <module>
main()
File "C:\Python27\lib\site-packages\retrying.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "C:\Python27\lib\site-packages\retrying.py", line 206, in call
return attempt.get(self._wrap_exception)
File "C:\Python27\lib\site-packages\retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "C:\Python27\lib\site-packages\retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "dAbot.py", line 667, in main
if not login(username, password):
File "dAbot.py", line 205, in login
'validate_token' : get_validate_token(login_html),
File "dAbot.py", line 178, in get_validate_token
return r.group(1)
AttributeError: 'NoneType' object has no attribute 'group'
Press any key to exit...
Hey there i've been a faithful user of the OCLB for a while now and just found out about this bot.
Been running it for a few hours now to give llamas to random deviants and so far it looks like it is able to throw out about 19 llamas every half an hour.
Before i knew about this nifty CLI tool i've developed a GreaseMonkey script which would run through a memberlist of a group and automatically click every .oclb-give and continue to the next page when done.
However my script clicked a single button per minute and waited for 5 minutes if it reached the limit.
Unfortunately i didn't really calculate my script's 'llamas per hour' for comparison.
I was wondering if you did a bit of research wether you are getting 'punished' if you hit the spam limit.
Maybe there's a chance that putting in delays between the giving of llamas results in a higher llamas/hour rate.
Access to this page has been denied because we believe you are using automated tools to log into and browse DeviantArt.
This may happen as a result of the following:
Please make sure Javascript and cookies are enabled on your browser in order to browse
Since the last 2 months, the speed that the code runs on seemed to... slow down by about 4 times?
It's not a wifi or computer issue, as it's the same on other devices and other networks
I developed a simmillar bot for scraping answers and used 2capcha to bypass recapcha , there was a callback function call "handlecapcha" which i used to callback and bypass capcha. This doesnt seem to work now. any idea Why?
@KishanBagaria
How would I
a) edit watch settings for a user and
b) fetch the last date a user was online
through code?
C) Also I think the server is backed up
I understand that dabot can not find username in urls,
Because deviantart change username.deviantart.com to deviantart.com/username
This is also visible in the source of the page, on the list with deviants
I find this in code:
'dev_names' : r'([A-Za-z0-9-]+).deviantart.com',
Its not correct i think... could you help me change this? (i know i sucks)
'dev_names' : r'\deviantart.com([A-Za-z0-9-]+)',
I've been using this for giving llamas for a while and it was working fine, but now it throws me the following error:
Traceback (most recent call last):
File "c:\python27\lib\runpy.py", line 174, in run_module_as_main
"main", fname, loader, pkg_name)
File "c:\python27\lib\runpy.py", line 72, in run_code
exec code in run_globals
File "C:\Python27\Scripts\dAbot.exe_main.py", line 9, in
File "c:\python27\lib\site-packages\dAbot\dAbot.py", line 817, in main
init()
File "c:\python27\lib\site-packages\dAbot\dAbot.py", line 677, in init
load_data()
File "c:\python27\lib\site-packages\dAbot\dAbot.py", line 598, in load_data
LlamaTransactions = read_llama_transactions()
File "c:\python27\lib\site-packages\dAbot\dAbot.py", line 571, in read_llama_transactions
return set(json.load(f))
File "c:\python27\lib\json_init.py", line 291, in load
**kw)
File "c:\python27\lib\json_init_.py", line 339, in loads
return _default_decoder.decode(s)
File "c:\python27\lib\json\decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "c:\python27\lib\json\decoder.py", line 380, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting object: line 1 column 221152 (char 221151)
Press any key to exit...
The error happens only when I enter my username. I tried typing different usernames and in those cases the script works fine again, it just throws the "wrong password" error which is to be expected. It isn't a problem after logging in either because if I type my username with a wrong password, it crashes again before even trying to log in. I can run the script properly in another computer too (same internet IP), so I'd guess it's not something from deviantart but an issue with the script.
I've tried reinstalling Python and the script again but that doesn't fix it either. Any idea what could the issue be?
So i've been running that command for a bit now and i came up with an idea to improve that process.
Currently i'm running it in 2 windows simultanously because it's kinda slow to find exchangers (which didn't receive a llama from me yet) and that way i'm actually sometimes able to hit the spam limit.
I guess this would be a rather drastic change to the codebase but the whole thing could be more effective if there'd be multiple processes running: One (or more) finding suitable exchangers and putting them into a queue and the other one actually giving llamas to the people in the queue.
This came to my mind when i saw that dAbot stopped doing anything for half an hour even though it could theoretically search for exchangers in the meantime.
Can dAbot collect dev username and save to file?
Thanks !!
When i try send lama to group_members:
Traceback (most recent call last):
File "d:\python27\lib\runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "d:\python27\lib\runpy.py", line 72, in run_code
exec code in run_globals
File "D:\Python27\Scripts\dAbot.exe_main.py", line 9, in
File "d:\python27\lib\site-packages\dAbot\dAbot.py", line 812, in main
run()
File "d:\python27\lib\site-packages\retrying.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "d:\python27\lib\site-packages\retrying.py", line 206, in call
return attempt.get(self._wrap_exception)
File "d:\python27\lib\site-packages\retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "d:\python27\lib\site-packages\retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "d:\python27\lib\site-packages\dAbot\dAbot.py", line 710, in run
for m in get_group_members(args[''], args['--reversed']):
File "d:\python27\lib\site-packages\dAbot\dAbot.py", line 459, in get_group_members
end_offset = int(re.search(regex['group_last_offset'], html).group(1))
AttributeError: 'NoneType' object has no attribute 'group'
Press any key to exit...
Ok. I'm bad a programming.
How can I run this through python?
os.system("say hi") #Works
os.system("dAbot <MyUser> <MyPass> -v llama give <Username>") #Nope. <>s have actual info
Just tried running the bot and dA managed to find out that it is not a browser doing the requests again.
Tried changing the UserAgent strings in the code to see if it would fix the issue like last time but it seems to be something different now.
Interestingly i was able to login in using an incognito tab with disabled JS...
The HTML of the login form didn't show any sign of google recaptcha apart from an attribute data-sigil="recaptcha-form"
on the form tag and references in js files.
Google's ReCaptcha can't be used without Javascript enabled though (at least the Invisible one) so i don't have a single clue what's going on here...
Maybe the HTML that the bot gets upon GETting the login form is already different from what browsers users will get?
1.I have a question if I can send a badge to my / other profile watchers on deviantart?
If yes - tell me how to do it ^^
2.What is the "hof"command ?
12 Jul 2017
07:37:37.076 PM [Llama Transactions] 0
07:37:37.076 PM Checking if logged in as kay-ra
07:37:37.435 PM Downloading login page
07:37:38.826 PM Logging in as kay-ra
07:37:39.390 PM Retrying... DAThrottlingError()
What does it mean and how can it be solved?
Thank you !
In dAbot.py, there's a line that says "import urlparse"
After the new Python update, urlparse has been moved to urllib.parse
So..... now what?
Also, I have python 2 and 3, and it chooses to run with 3. How do I make it run with 2?
DeviantArt's "Please confirm you are human." message prevents dAbot from working. It appears when you try to login. The full message is:
"Access to this page has been denied because we believe you are using automated tools to log into and browse DeviantArt.
This may happen as a result of the following:
Please make sure Javascript and cookies are enabled on your browser in order to browse DeviantArt."
When i logged into my computer this morning i saw several errors in my terminals stating that they couldn't request https:///someuser.deviantart.com
I've looked a bit through the code and saw that the username variables had a leading slash due to incorrect substringing here.
Replacing the 7 (http://
) with an 8 (https://
) fixed it for me.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.