Comments (87)
just following.
Then please just please subscribe, nobody wants to get spammed with "just following" updates!
from icloud_photos_downloader.
1.17.0 released and has the fix
from icloud_photos_downloader.
Update: All test are now passing for the auth_fix
branch with 99% code coverage (matching the master branch). Hopefully this will allow for the branch to be pulled into the master.
The 2SA/2FA cases noted above will still need to be looked into but I'm hopeful these can be treated as separate issues and allow the core fix to be merged for those that are waiting on this due to various package distribution requirements.
Happy Holidays!
from icloud_photos_downloader.
I can replicate the issue on docker with 1.16.3 without keyring, so it is general issue and most likely related to some changes on Apple side as my 1.16.3 was running for a number of days before the issue started.
from icloud_photos_downloader.
@scaraebeus' fix worked for me. Built a local docker image and successfully grabbed my photos from two accounts! 🎉
from icloud_photos_downloader.
Does anyone know how to apply this fix for docker running in synology nas?
from icloud_photos_downloader.
The docker solution does not seem to work for me: I'm still getting invalid email/password combo when attempting this via a build docker container:
The master branch has yet to be updated. One proposed solution is available in PR #734.
from icloud_photos_downloader.
@amdydesign as above, git checkout auth_fix
. Or wait until branch is merged into master.
from icloud_photos_downloader.
Is there a update for the PIP (windows) Version? I don't use docker.
Or do I have to wait for next friday?
Not sure on timing for the official update. Unfortunately for my fix to be pulled in, there are a few things that need to be in place on my branch for it to be accepted. I've been working through updating the various tests and code coverage and in doing so I am finding some other parts of the pyicloud_ipd/base.py
auth flow that are no longer working correctly due to (possibly) recent changes in how the responses are coming through.
I'm working through it but it will take some time - especially with the holidays coming up.
The only option I'm aware of at this time is to consume the auth_fix branch directly if your environment and setup allows you - with the risk that it has yet to be fully tested and vetted.
from icloud_photos_downloader.
@scaraebeus thanks a lot for your hard work and dedication. It is well appreciated by many.
from icloud_photos_downloader.
@scaraebeus I join previous comment to thank you for your great contribution here, I patched the main branch with #734, built and installed it with pip and got it working on both raspbian/debian bullseye (11) and bookworm (12) . I am also using edits as described in issue #249 so that I can download both edited media (the ones we see on the phone and in iCloud) and original media in a regular cron backup script to my pi-based NAS, which makes this script critical to me. thanks again and congrats for keeping it working !
from icloud_photos_downloader.
Previous behaviour was that saving to the keychain required SMS MFA, it did not have an option for Apple's built in MFA.
Performing a download would then trigger a second MFA prompt, which had two options for MFA; SMS and Apple.
I noticed this behaviour when attempting to relocate the MFA cookie in my container from $HOME/.local to /config. Took me a lot of deletes/recreates to get it working reliably.
Edit: I think it's also worth mentioning that I actually had two SMS numbers I could use for three of my containers. I have added my phone number as a trusted number to the three accounts of my family members. This allows me to re-authenticate their containers without me needing access to their devices. Very useful feature.
from icloud_photos_downloader.
I've built a new container with iCloud 1.17.0, but the behaviour is not the same. I am no longer presented with the option to perform SMS based multifactor authentication. I also receive a warning about it not being able to parse JSON, which I've not seen before but I'm not too worried about, as the application downloads the photos regardless:
2023-12-20 12:08:01 DEBUG Configure password
2023-12-20 12:08:01 DEBUG Adding password to keyring file: /config/python_keyring/keyring_pass.cfg
2023-12-20 12:08:01 DEBUG Switched to icloudpd: 1.17.0
Enter iCloud password for [email protected]:
Save password in keyring? [y/N]: y
Two-step authentication required.
Please enter validation code
(string) --> 137446
2023-12-20 12:08:46 INFO Starting container initialisation
2023-12-20 12:08:46 DEBUG Generate MFA cookie using password stored in keyring file
2023-12-20 12:08:47 DEBUG Switched to icloudpd: 1.17.0
2023-12-20 12:08:47 ERROR Authentication required for Account. (421)
Please enter two-factor authentication code: 119033
2023-12-20 12:09:20 WARNING Failed to parse response with JSON mimetype
Is SMS based multifactor authentication no longer possible?
from icloud_photos_downloader.
I'm having the same issue. Hoping it's a temporary problem with iCloud itself. I was able to log in just yesterday with iCloudPD, and my configuration has not changed since then.
from icloud_photos_downloader.
Same issue as well, failure to login, even after changing password to something simple.
from icloud_photos_downloader.
Same here
from icloud_photos_downloader.
Same here
from icloud_photos_downloader.
also experiencing the same issue with 1.16.2 as well as after upgrading to 1.16.3
from icloud_photos_downloader.
Same here
from icloud_photos_downloader.
Same issue - container has stopped downloading today
from icloud_photos_downloader.
Same issue here, stopped working yesterday Dec 6
from icloud_photos_downloader.
Same. Following this thread for updates.
from icloud_photos_downloader.
Following
from icloud_photos_downloader.
Same here, following
from icloud_photos_downloader.
Same, following;
from icloud_photos_downloader.
I also get the same error on two different iCloud accounts. Suddenly stopped working 2 days ago. Both of my boredazfcuk/docker-icloudpd containers stopped and also manual execution of the script
~/.local/bin/icloudpd -u [email protected] -p ****** -d . --folder-structure={:%Y/%m/%d} --delete-after-download
fails with:
_2023-12-07 23:24:36 DEBUG Authenticating... 2023-12-07 23:24:36 ERROR Unknown reason Traceback (most recent call last): File "pyicloud_ipd/base.py", line 220, in authenticate File "requests/sessions.py", line 637, in post File "pyicloud_ipd/base.py", line 105, in request File "pyicloud_ipd/base.py", line 127, in _raise_error pyicloud_ipd.exceptions.PyiCloudAPIResponseError: Unknown reason
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "starters/icloudpd.py", line 5, in File "click/core.py", line 1157, in call File "click/core.py", line 1078, in main File "click/core.py", line 1434, in invoke File "click/core.py", line 783, in invoke File "icloudpd/base.py", line 317, in main File "icloudpd/base.py", line 744, in core File "icloudpd/authentication.py", line 31, in authenticate_ File "pyicloud_ipd/base.py", line 204, in init File "pyicloud_ipd/base.py", line 228, in authenticate pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError('Unknown reason')) [1357605] Failed to execute script 'icloudpd' due to unhandled exception!_
Logging in on the icloud.com website works as it should.
Does this mean there is no remedy for this problem yet? Has Apple changed something?
from icloud_photos_downloader.
Same, following;
from icloud_photos_downloader.
just following.
from icloud_photos_downloader.
Hi, maybe it will help to solve the problem. I have three icloud Accounts running and the issue is the same in every account.
icloudpd --directory /xxx/xxx/xxx/icloudpd_photo_backup --username [email protected] --password xxxxxxxx --log-level debug
2023-12-08 16:16:11 DEBUG Authenticating...
2023-12-08 16:16:12 ERROR Unknown reason
Traceback (most recent call last):
File "pyicloud_ipd/base.py", line 220, in authenticate
File "requests/sessions.py", line 637, in post
File "pyicloud_ipd/base.py", line 105, in request
File "pyicloud_ipd/base.py", line 127, in _raise_error
pyicloud_ipd.exceptions.PyiCloudAPIResponseError: Unknown reason
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "starters/icloudpd.py", line 5, in
File "click/core.py", line 1157, in call
File "click/core.py", line 1078, in main
File "click/core.py", line 1434, in invoke
File "click/core.py", line 783, in invoke
File "icloudpd/base.py", line 317, in main
File "icloudpd/base.py", line 744, in core
File "icloudpd/authentication.py", line 31, in authenticate_
File "pyicloud_ipd/base.py", line 204, in init
File "pyicloud_ipd/base.py", line 228, in authenticate
pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError('Unknown reason'))
[201833] Failed to execute script 'icloudpd' due to unhandled exception!
from icloud_photos_downloader.
I'm experiencing this too. Started on the 6th at around 5pm EST.
from icloud_photos_downloader.
I see the following paths:
- dig into current code/protocol to see if minor tweak will make it working
- rationale: fastest and least invasive
- concerns: low chances
- chances: low; suspect Apple turned off old protocol that
icloudpd
uses (they support OAUTH on web for some time) - investments: low
- use original pyicloud lib
- rationale: IIUC pyicloud was updated to support OAUTH some time ago
- concerns: may break some of our code
- chances: mid-high; need to confirm that pyicloud works with current Apple services first
- investments: mid; need to update tests and that has been a challenge in the prev attempts to bring latest pyicloud
- use own code to support latest Apple auth
- rationale: guaranteed success, opens door for other long-requested improvements
- concerns: may force larger updates, leading to breaking changes, long alpha-beta cycle
- chances: high
- investments: high
Is anybody interested in 1) and/or 2)? I plan to allocate some time in the next number of days to look into this issue and will invest in 1 & 3. IMO it is okay for multiple ppl to look at the same path as there are always edge cases and different approaches that may give different or complementing outcomes.
from icloud_photos_downloader.
@AndreyNikiforov Thanks for the summary / plans of attack.
To help guide others that may want to debug, a possible good test case to try is within the tests/test_authentication.py
tests, test_2sa_required
. Per the instructions in the code, one can delete the pre-recorded VCR response file tests/vcr_cassettes/auth_requires_2sa.yml
, and fill in their actual username/password in the test, and then run scripts/test
(SEE https://github.com/icloud-photos-downloader/icloud_photos_downloader/blob/master/CONTRIBUTING.md#setting-up-the-development-environment on how to setup your local environment to test this code)
The tests will fail, and the VCR recording (aka "cassette") will be updated with the actual response received from icloud.
BE SURE TO NOT SHARE YOUR USERNAME/PASSWORD WHICH ARE PRESENT IN THE VCR RECORDING!
For instance, after doing the above, this is the diff of the VCR cassette. I can't glean much useful information from the response yet:
- My new yml file has lots of whitespace/formatting differences vs the original
- The new response is less verbose than the expected "good" response
- I'm not as familiar with Python or these icloud related Python libraries.
"git diff" of the VCR yaml file
diff --git a/tests/vcr_cassettes/auth_requires_2sa.yml b/tests/vcr_cassettes/auth_requires_2sa.yml
index d0bbf78..ad75ec9 100644
--- a/tests/vcr_cassettes/auth_requires_2sa.yml
+++ b/tests/vcr_cassettes/auth_requires_2sa.yml
@@ -1,37 +1,59 @@
interactions:
- request:
- body: !!python/unicode '{"apple_id": "[email protected]", "password": "password1",
- "extended_login": false}'
+ body: '{"apple_id": "[email protected]", "password": "password1", "extended_login":
+ false}'
headers:
- Accept: ['*/*']
- Accept-Encoding: ['gzip, deflate']
- Connection: [keep-alive]
- Content-Length: ['88']
- Origin: ['https://www.icloud.com']
- Referer: ['https://www.icloud.com/']
- User-Agent: [Opera/9.52 (X11; Linux i686; U; en)]
+ Accept:
+ - '*/*'
+ Accept-Encoding:
+ - gzip, deflate
+ Connection:
+ - keep-alive
+ Content-Length:
+ - '90'
+ Origin:
+ - https://www.icloud.com
+ Referer:
+ - https://www.icloud.com/
+ User-Agent:
+ - Opera/9.52 (X11; Linux i686; U; en)
method: POST
- uri: https://setup.icloud.com/setup/ws/1/login?clientBuildNumber=17DHotfix5&clientMasteringNumber=17DHotfix5&clientId=EC5646DE-9423-11E8-BF21-14109FE0B321&ckjsVersion=2.0.5&ckjsBuildVersion=17DProjectDev77
+ uri: https://setup.icloud.com/setup/ws/1/login?clientBuildNumber=17DHotfix5&clientMasteringNumber=17DHotfix5&ckjsBuildVersion=17DProjectDev77&ckjsVersion=2.0.5&clientId=EC5646DE-9423-11E8-BF21-14109FE0B321
response:
- body: {string: !!python/unicode '{"dsInfo":{"lastName":"Doe","iCDPEnabled":false,"dsid":"123456789","hsaEnabled":true,"ironcadeMigrated":true,"locale":"en-us_US","brZoneConsolidated":false,"isManagedAppleID":false,"gilligan-invited":"true","appleIdAliases":["[email protected]"],"hsaVersion":2,"isPaidDeveloper":true,"countryCode":"USA","notificationId":"12341234-1234-1234-1234-143241234123","primaryEmailVerified":true,"aDsID":"12341234123412341234","locked":false,"hasICloudQualifyingDevice":true,"primaryEmail":"[email protected]","appleIdEntries":[{"isPrimary":true,"type":"EMAIL","value":"[email protected]"}],"gilligan-enabled":"true","fullName":"John
- Doe","languageCode":"en-us","appleId":"[email protected]","firstName":"John","iCloudAppleIdAlias":"[email protected]","notesMigrated":true,"hasPaymentInfo":true,"pcsDeleted":false,"appleIdAlias":"","brMigrated":true,"statusCode":2},"hasMinimumDeviceForPhotosWeb":true,"iCDPEnabled":false,"webservices":{"reminders":{"url":"https://p10-remindersws.icloud.com:443","status":"active"},"notes":{"url":"https://p10-notesws.icloud.com:443","status":"active"},"mail":{"url":"https://p10-mailws.icloud.com:443","status":"active"},"ckdatabasews":{"pcsRequired":true,"url":"https://p10-ckdatabasews.icloud.com:443","status":"active"},"photosupload":{"pcsRequired":true,"url":"https://p10-uploadphotosws.icloud.com:443","status":"active"},"photos":{"pcsRequired":true,"uploadUrl":"https://p10-uploadphotosws.icloud.com:443","url":"https://p10-photosws.icloud.com:443","status":"active"},"drivews":{"pcsRequired":true,"url":"https://p10-drivews.icloud.com:443","status":"active"},"uploadimagews":{"url":"https://p10-uploadimagews.icloud.com:443","status":"active"},"schoolwork":{},"cksharews":{"url":"https://p10-ckshare.icloud.com:443","status":"active"},"findme":{"url":"https://p10-fmipweb.icloud.com:443","status":"active"},"ckdeviceservice":{"url":"https://p10-ckdevice.icloud.com:443"},"iworkthumbnailws":{"url":"https://p10-iworkthumbnailws.icloud.com:443","status":"active"},"calendar":{"url":"https://p10-calendarws.icloud.com:443","status":"active"},"docws":{"pcsRequired":true,"url":"https://p10-docws.icloud.com:443","status":"active"},"settings":{"url":"https://p10-settingsws.icloud.com:443","status":"active"},"ubiquity":{"url":"https://p10-ubiquityws.icloud.com:443","status":"active"},"streams":{"url":"https://p10-streams.icloud.com:443","status":"active"},"keyvalue":{"url":"https://p10-keyvalueservice.icloud.com:443","status":"active"},"archivews":{"url":"https://p10-archivews.icloud.com:443","status":"active"},"push":{"url":"https://p10-pushws.icloud.com:443","status":"active"},"iwmb":{"url":"https://p10-iwmb.icloud.com:443","status":"active"},"iworkexportws":{"url":"https://p10-iworkexportws.icloud.com:443","status":"active"},"geows":{"url":"https://p10-geows.icloud.com:443","status":"active"},"account":{"iCloudEnv":{"shortId":"p","vipSuffix":"p"},"url":"https://p10-setup.icloud.com:443","status":"active"},"fmf":{"url":"https://p10-fmfweb.icloud.com:443","status":"active"},"contacts":{"url":"https://p10-contactsws.icloud.com:443","status":"active"}},"pcsEnabled":true,"configBag":{"urls":{"accountCreateUI":"https://appleid.apple.com/widget/account/?widgetKey=12312412412341234123412341234123412341234#!create","accountLoginUI":"https://idmsa.apple.com/appleauth/auth/signin?widgetKey=83545bf919730e51dbfba24e7e8a78d2","accountLogin":"https://setup.icloud.com/setup/ws/1/accountLogin","accountRepairUI":"https://appleid.apple.com/widget/account/?widgetKey=12312412412341234123412341234123412341234#!repair","downloadICloudTerms":"https://setup.icloud.com/setup/ws/1/downloadLiteTerms","repairDone":"https://setup.icloud.com/setup/ws/1/repairDone","vettingUrlForEmail":"https://id.apple.com/IDMSEmailVetting/vetShareEmail","accountCreate":"https://setup.icloud.com/setup/ws/1/createLiteAccount","getICloudTerms":"https://setup.icloud.com/setup/ws/1/getTerms","vettingUrlForPhone":"https://id.apple.com/IDMSEmailVetting/vetSharePhone"},"accountCreateEnabled":"true"},"hsaTrustedBrowser":false,"appsOrder":["mail","contacts","calendar","photos","iclouddrive","notes2","reminders","pages","numbers","keynote","newspublisher","fmf","find","settings"],"version":2,"isExtendedLogin":false,"pcsServiceIdentitiesIncluded":false,"hsaChallengeRequired":true,"requestInfo":{"country":"TH","timeZone":"GMT+7","isAppleInternal":true},"pcsDeleted":false,"iCloudInfo":{"SafariBookmarksHasMigratedToCloudKit":false},"apps":{"calendar":{},"reminders":{},"keynote":{"isQualifiedForBeta":true},"settings":{"canLaunchWithOneFactor":true},"mail":{},"numbers":{"isQualifiedForBeta":true},"photos":{},"pages":{"isQualifiedForBeta":true},"find":{"canLaunchWithOneFactor":true},"notes2":{},"iclouddrive":{},"newspublisher":{"isHidden":true},"fmf":{},"contacts":{}}}'}
+ body:
+ string: '{"success":false,"error":1}'
headers:
- access-control-allow-credentials: ['true']
- access-control-allow-origin: ['https://www.icloud.com']
- access-control-expose-headers: [X-Apple-Request-UUID, Via]
- apple-originating-system: [UnknownOriginatingSystem]
- apple-seq: ['0']
- apple-tk: ['false']
- cache-control: ['no-cache, no-store, private']
- connection: [keep-alive]
- content-length: ['4895']
- content-type: [application/json; charset=UTF-8]
- date: ['Mon, 30 Jul 2018 19:00:39 GMT']
- server: [AppleHttpServer/2f080fc0]
- strict-transport-security: [max-age=31536000; includeSubDomains]
- via: ['icloudedge:si03p00ic-ztde010417:7401:18RC341:Singapore']
- x-apple-jingle-correlation-key: [SJHIUN7879234KJHH8JBH]
- x-apple-request-uuid: [NISUHFIOSUHFOSIDUHFOSIDF]
- x-responding-instance: ['setupservice:328457238759328579234875']
- status: {code: 200, message: OK}
+ Access-Control-Allow-Credentials:
+ - 'true'
+ Access-Control-Allow-Origin:
+ - https://www.icloud.com
+ Cache-Control:
+ - no-cache, no-store, private
+ Connection:
+ - keep-alive
+ Content-Type:
+ - application/json; charset=UTF-8
+ Date:
+ - Fri, 08 Dec 2023 16:37:26 GMT
+ Server:
+ - AppleHttpServer/78689afb4479
+ Strict-Transport-Security:
+ - max-age=31536000; includeSubDomains
+ X-Apple-Edge-Response-Time:
+ - '45'
+ X-Apple-Request-UUID:
+ - e7626cfc-bb0c-4ccd-986d-b9327f8bcef3
+ X-Responding-Instance:
+ - setupservice:44700503:pv51p47ic-qukt21032501:8003:2404B363:5934c9004de5
+ access-control-expose-headers:
+ - X-Apple-Request-UUID,Via
+ content-length:
+ - '27'
+ via:
+ - 631194250daa17e24277dea86cf30319:7ecc2ff259156058e890ba5cf00be461:uschi7
+ x-apple-user-partition:
+ - '47'
+ status:
+ code: 421
+ message: Misdirected Request
version: 1
However, the HTTP response code of "421: Misdirected Request" may be the key, and indicate that icloud has updated its API and therefore pyicloud needs updating to reflect that.
Back in Oct 2023, someone submitted an issue to pyicloud regarding the 421 response: picklepete/pyicloud#441, and 4 days ago there is a possible fix: picklepete/pyicloud#406 (it's not merged yet)
My system:
- macOS Sonoma 14.1.2 (Apple M3 Macbook Pro)
- Python 3.12.0
from icloud_photos_downloader.
Total speculation but I wonder if this has anything to do with Apple's attempts to block iMessage on Android;
https://www.theverge.com/2023/12/8/23994089/apple-beeper-mini-android-blocked-imessage-app
from icloud_photos_downloader.
My guess is that it has something to do with the recent support for passkey logins.
At least for the iCloud website it asks for passkey and logging in via password takes one more click.
Such a thing would be unusual for an api, nevertheless a change that requires specification of the desired login method sounds very likely.
from icloud_photos_downloader.
My guess is that it has something to do with the recent support for passkey logins. At least for the iCloud website it asks for passkey and logging in via password takes one more click.
Such a thing would be unusual for an api, nevertheless a change that requires specification of the desired login method sounds very likely.
This sounds like the issue for me...
from icloud_photos_downloader.
Same error here running on Windows x64 with 1.16.3 version
from icloud_photos_downloader.
Same issue for me as well, docker that was previously working fine has now failed authorization today as well, and I haven't been able to authorize one for 3 days.
from icloud_photos_downloader.
This looks related to beeper mini being blocked. 😔
from icloud_photos_downloader.
I see the following paths:
- dig into current code/protocol to see if minor tweak will make it working
- rationale: fastest and least invasive
- concerns: low chances
- chances: low; suspect Apple turned off old protocol that
icloudpd
uses (they support OAUTH on web for some time)- investments: low
- use original pyicloud lib
- rationale: IIUC pyicloud was updated to support OAUTH some time ago
- concerns: may break some of our code
- chances: mid-high; need to confirm that pyicloud works with current Apple services first
- investments: mid; need to update tests and that has been a challenge in the prev attempts to bring latest pyicloud
[ . . . ]
Is anybody interested in 1) and/or 2)? I plan to allocate some time in the next number of days to look into this issue and will invest in 1 & 3. IMO it is okay for multiple ppl to look at the same path as there are always edge cases and different approaches that may give different or complementing outcomes.
If it helps, I was able to verify the current 1.0.0 version of pyicloud works as is. I was able to instantiate a PyiCloudService and list albums (pyicloud,photos.albums).
Potentially doing a combination of path 1) and 2) may be an option - possibly just focusing on merging over the key authentication pieces from the 1.0.0 pyicloud.
from icloud_photos_downloader.
If it helps, I was able to verify the current 1.0.0 version of pyicloud works as is. I was able to instantiate a PyiCloudService and list albums (pyicloud,photos.albums).
I was noticing that as well. icloud_photos_downloader
uses some customized versions of pyicloud
code to be compatible with China vs US icloud domains. It's possible those customizations are now causing some issues, and may not be updated with the latest code from pyicloud
.
I think changes would have to be made to the following files:
src/pyicloud_ipd/base.py
- Needs to be updated to reflect latest version from pyicloud repo
- Note: this update will remove the domain-specific code changes made by
icloud_photos_downloader
to switch between US and China icloud domain names. That will have to be addressed.
src/icloudpd/authentication.py
- Need to be updated to call
pyicloud_ipd.PyiCloudService(...)
with correct arguments, removing "domain" from the arg list, etc.
- Need to be updated to call
src/pyicloud_ipd/exceptions.py
- Needs to be updated to reflect exception classes from pyicloud repo
"git diff" example that gets icloudpd_ipd updated with latest changes from PyiCloud
diff --git a/src/pyicloud_ipd/base.py b/src/pyicloud_ipd/base.py
index a387a65..e53f6f5 100644
--- a/src/pyicloud_ipd/base.py
+++ b/src/pyicloud_ipd/base.py
@@ -1,15 +1,13 @@
-import six
-import uuid
-import hashlib
+from uuid import uuid1
import inspect
import json
import logging
-import requests
-import sys
-import tempfile
-import os
+from requests import Session
+from tempfile import gettempdir
+from os import path, mkdir
from re import match
-import urllib3
+import http.cookiejar as cookielib
+import getpass
from pyicloud_ipd.exceptions import (
PyiCloudConnectionException,
@@ -29,13 +27,15 @@ from pyicloud_ipd.services import (
)
from pyicloud_ipd.utils import get_password_from_keyring
-if six.PY3:
- import http.cookiejar as cookielib
-else:
- import cookielib
+LOGGER = logging.getLogger(__name__)
-
-logger = logging.getLogger(__name__)
+HEADER_DATA = {
+ "X-Apple-ID-Account-Country": "account_country",
+ "X-Apple-ID-Session-Id": "session_id",
+ "X-Apple-Session-Token": "session_token",
+ "X-Apple-TwoSV-Trust-Token": "trust_token",
+ "scnt": "scnt",
+}
class PyiCloudPasswordFilter(logging.Filter):
@@ -51,329 +51,554 @@ class PyiCloudPasswordFilter(logging.Filter):
return True
-class PyiCloudSession(requests.Session):
+class PyiCloudSession(Session):
+ """iCloud session."""
+
def __init__(self, service):
self.service = service
- super(PyiCloudSession, self).__init__()
+ super().__init__()
- def request(self, *args, **kwargs):
+ def request(self, method, url, **kwargs): # pylint: disable=arguments-differ
# Charge logging to the right service endpoint
callee = inspect.stack()[2]
module = inspect.getmodule(callee[0])
- logger = logging.getLogger(module.__name__).getChild('http')
- if self.service._password_filter not in logger.filters:
- logger.addFilter(self.service._password_filter)
-
- logger.debug("%s %s %s", args[0], args[1], kwargs.get('data', ''))
-
- try:
- response = super(PyiCloudSession, self).request(*args, **kwargs)
- except requests.exceptions.SSLError:
- raise PyiCloudConnectionException("Error establishing secure connection. Try --domain parameter")
-
- content_type = response.headers.get('Content-Type', '').split(';')[0]
- json_mimetypes = ['application/json', 'text/json']
+ request_logger = logging.getLogger(module.__name__).getChild("http")
+ if self.service.password_filter not in request_logger.filters:
+ request_logger.addFilter(self.service.password_filter)
+
+ request_logger.debug("%s %s %s", method, url, kwargs.get("data", ""))
+
+ has_retried = kwargs.get("retried")
+ kwargs.pop("retried", None)
+ response = super().request(method, url, **kwargs)
+
+ content_type = response.headers.get("Content-Type", "").split(";")[0]
+ json_mimetypes = ["application/json", "text/json"]
+
+ for header, value in HEADER_DATA.items():
+ if response.headers.get(header):
+ session_arg = value
+ self.service.session_data.update(
+ {session_arg: response.headers.get(header)}
+ )
+
+ # Save session_data to file
+ with open(self.service.session_path, "w", encoding="utf-8") as outfile:
+ json.dump(self.service.session_data, outfile)
+ LOGGER.debug("Saved session data to file")
+
+ # Save cookies to file
+ self.cookies.save(ignore_discard=True, ignore_expires=True)
+ LOGGER.debug("Cookies saved to %s", self.service.cookiejar_path)
+
+ if not response.ok and (
+ content_type not in json_mimetypes
+ or response.status_code in [421, 450, 500]
+ ):
+ try:
+ # pylint: disable=protected-access
+ fmip_url = self.service._get_webservice_url("findme")
+ if (
+ has_retried is None
+ and response.status_code in [421, 450, 500]
+ and fmip_url in url
+ ):
+ # Handle re-authentication for Find My iPhone
+ LOGGER.debug("Re-authenticating Find My iPhone service")
+ try:
+ # If 450, authentication requires a full sign in to the account
+ service = None if response.status_code == 450 else "find"
+ self.service.authenticate(True, service)
+
+ except PyiCloudAPIResponseException:
+ LOGGER.debug("Re-authentication failed")
+ kwargs["retried"] = True
+ return self.request(method, url, **kwargs)
+ except Exception:
+ pass
+
+ if has_retried is None and response.status_code in [421, 450, 500]:
+ api_error = PyiCloudAPIResponseException(
+ response.reason, response.status_code, retry=True
+ )
+ request_logger.debug(api_error)
+ kwargs["retried"] = True
+ return self.request(method, url, **kwargs)
- if not response.ok and content_type not in json_mimetypes:
self._raise_error(response.status_code, response.reason)
if content_type not in json_mimetypes:
return response
try:
- json = response.json()
- except:
- logger.warning('Failed to parse response with JSON mimetype')
+ data = response.json()
+ except: # pylint: disable=bare-except
+ request_logger.warning("Failed to parse response with JSON mimetype")
return response
- logger.debug(json)
+ request_logger.debug(data)
- reason = json.get('errorMessage')
- reason = reason or json.get('reason')
- reason = reason or json.get('errorReason')
- if not reason and isinstance(json.get('error'), six.string_types):
- reason = json.get('error')
- if not reason and json.get('error'):
- reason = "Unknown reason"
+ if isinstance(data, dict):
+ reason = data.get("errorMessage")
+ reason = reason or data.get("reason")
+ reason = reason or data.get("errorReason")
+ if not reason and isinstance(data.get("error"), str):
+ reason = data.get("error")
+ if not reason and data.get("error"):
+ reason = "Unknown reason"
- code = json.get('errorCode')
- if not code and json.get('serverErrorCode'):
- code = json.get('serverErrorCode')
+ code = data.get("errorCode")
+ if not code and data.get("serverErrorCode"):
+ code = data.get("serverErrorCode")
- if reason:
- self._raise_error(code, reason)
+ if reason:
+ self._raise_error(code, reason)
return response
def _raise_error(self, code, reason):
- if self.service.requires_2sa and \
- reason == 'Missing X-APPLE-WEBAUTH-TOKEN cookie':
- raise PyiCloud2SARequiredError(response.url)
- if code == 'ZONE_NOT_FOUND' or code == 'AUTHENTICATION_FAILED':
- reason = 'Please log into https://icloud.com/ to manually ' \
- 'finish setting up your iCloud service'
- api_error = PyiCloudServiceNotActivatedErrror(reason, code)
- logger.error(api_error)
-
- raise(api_error)
- if code == 'ACCESS_DENIED':
- reason = reason + '. Please wait a few minutes then try ' \
- 'again. The remote servers might be trying to ' \
- 'throttle requests.'
-
- api_error = PyiCloudAPIResponseError(reason, code)
- logger.error(api_error)
+ if (
+ self.service.requires_2sa
+ and reason == "Missing X-APPLE-WEBAUTH-TOKEN cookie"
+ ):
+ raise PyiCloud2SARequiredException(self.service.user["apple_id"])
+ if code in ("ZONE_NOT_FOUND", "AUTHENTICATION_FAILED"):
+ reason = (
+ "Please log into https://icloud.com/ to manually "
+ "finish setting up your iCloud service"
+ )
+ api_error = PyiCloudServiceNotActivatedException(reason, code)
+ LOGGER.error(api_error)
+
+ raise (api_error)
+ if code == "ACCESS_DENIED":
+ reason = (
+ reason + ". Please wait a few minutes then try again."
+ "The remote servers might be trying to throttle requests."
+ )
+ if code in [421, 450, 500]:
+ reason = "Authentication required for Account."
+
+ api_error = PyiCloudAPIResponseException(reason, code)
+ LOGGER.error(api_error)
raise api_error
-class PyiCloudService(object):
+class PyiCloudService:
"""
A base authentication class for the iCloud service. Handles the
authentication required to access iCloud services.
Usage:
- from pyicloud_ipd import PyiCloudService
+ from pyicloud import PyiCloudService
pyicloud = PyiCloudService('[email protected]', 'password')
- pyicloud_ipd.iphone.location()
+ pyicloud.iphone.location()
"""
+ AUTH_ENDPOINT = "https://idmsa.apple.com/appleauth/auth"
+ HOME_ENDPOINT = "https://www.icloud.com"
+ SETUP_ENDPOINT = "https://setup.icloud.com/setup/ws/1"
+
def __init__(
- self, domain, apple_id, password=None, cookie_directory=None, verify=True,
- client_id=None
+ self,
+ apple_id,
+ password=None,
+ cookie_directory=None,
+ verify=True,
+ client_id=None,
+ with_family=True,
):
if password is None:
password = get_password_from_keyring(apple_id)
+ self.user = {"accountName": apple_id, "password": password}
self.data = {}
- self.client_id = client_id or str(uuid.uuid1()).upper()
- self.user = {'apple_id': apple_id, 'password': password}
-
- self._password_filter = PyiCloudPasswordFilter(password)
- logger.addFilter(self._password_filter)
-
- if (domain == 'com'):
- self._home_endpoint = 'https://www.icloud.com'
- self._setup_endpoint = 'https://setup.icloud.com/setup/ws/1'
- elif (domain == 'cn'):
- self._home_endpoint = 'https://www.icloud.com.cn'
- self._setup_endpoint = 'https://setup.icloud.com.cn/setup/ws/1'
- else:
- raise NotImplementedError(f"Domain '{domain}' is not supported yet")
+ self.params = {}
+ self.client_id = client_id or ("auth-%s" % str(uuid1()).lower())
+ self.with_family = with_family
- self._base_login_url = '%s/login' % self._setup_endpoint
+ self.password_filter = PyiCloudPasswordFilter(password)
+ LOGGER.addFilter(self.password_filter)
if cookie_directory:
- self._cookie_directory = os.path.expanduser(
- os.path.normpath(cookie_directory)
- )
+ self._cookie_directory = path.expanduser(path.normpath(cookie_directory))
+ if not path.exists(self._cookie_directory):
+ mkdir(self._cookie_directory, 0o700)
else:
- self._cookie_directory = os.path.join(
- tempfile.gettempdir(),
- 'pyicloud',
- )
+ topdir = path.join(gettempdir(), "pyicloud")
+ self._cookie_directory = path.join(topdir, getpass.getuser())
+ if not path.exists(topdir):
+ mkdir(topdir, 0o777)
+ if not path.exists(self._cookie_directory):
+ mkdir(self._cookie_directory, 0o700)
+
+ LOGGER.debug("Using session file %s", self.session_path)
+
+ self.session_data = {}
+ try:
+ with open(self.session_path, encoding="utf-8") as session_f:
+ self.session_data = json.load(session_f)
+ except: # pylint: disable=bare-except
+ LOGGER.info("Session file does not exist")
+ if self.session_data.get("client_id"):
+ self.client_id = self.session_data.get("client_id")
+ else:
+ self.session_data.update({"client_id": self.client_id})
self.session = PyiCloudSession(self)
self.session.verify = verify
- self.session.headers.update({
- 'Origin': self._home_endpoint,
- 'Referer': '%s/' % self._home_endpoint,
- 'User-Agent': 'Opera/9.52 (X11; Linux i686; U; en)'
- })
+ self.session.headers.update(
+ {"Origin": self.HOME_ENDPOINT, "Referer": "%s/" % self.HOME_ENDPOINT}
+ )
- cookiejar_path = self._get_cookiejar_path()
+ cookiejar_path = self.cookiejar_path
self.session.cookies = cookielib.LWPCookieJar(filename=cookiejar_path)
- if os.path.exists(cookiejar_path):
+ if path.exists(cookiejar_path):
try:
- self.session.cookies.load()
- logger.debug("Read cookies from %s", cookiejar_path)
- except:
+ self.session.cookies.load(ignore_discard=True, ignore_expires=True)
+ LOGGER.debug("Read cookies from %s", cookiejar_path)
+ except: # pylint: disable=bare-except
# Most likely a pickled cookiejar from earlier versions.
# The cookiejar will get replaced with a valid one after
# successful authentication.
- logger.warning("Failed to read cookiejar %s", cookiejar_path)
-
- self.params = {
- 'clientBuildNumber': '17DHotfix5',
- 'clientMasteringNumber': '17DHotfix5',
- 'ckjsBuildVersion': '17DProjectDev77',
- 'ckjsVersion': '2.0.5',
- 'clientId': self.client_id,
- }
+ LOGGER.warning("Failed to read cookiejar %s", cookiejar_path)
self.authenticate()
- def authenticate(self):
+ self._drive = None
+ self._files = None
+ self._photos = None
+
+ def authenticate(self, force_refresh=False, service=None):
"""
- Handles authentication, and persists the X-APPLE-WEB-KB cookie so that
+ Handles authentication, and persists cookies so that
subsequent logins will not cause additional e-mails from Apple.
"""
- logger.info("Authenticating as %s", self.user['apple_id'])
-
- data = dict(self.user)
+ login_successful = False
+ if self.session_data.get("session_token") and not force_refresh:
+ LOGGER.debug("Checking session token validity")
+ try:
+ self.data = self._validate_token()
+ login_successful = True
+ except PyiCloudAPIResponseException:
+ LOGGER.debug("Invalid authentication token, will log in from scratch.")
+
+ if not login_successful and service is not None:
+ app = self.data["apps"][service]
+ if "canLaunchWithOneFactor" in app and app["canLaunchWithOneFactor"]:
+ LOGGER.debug(
+ "Authenticating as %s for %s", self.user["accountName"], service
+ )
+ try:
+ self._authenticate_with_credentials_service(service)
+ login_successful = True
+ except Exception:
+ LOGGER.debug(
+ "Could not log into service. Attempting brand new login."
+ )
+
+ if not login_successful:
+ LOGGER.debug("Authenticating as %s", self.user["accountName"])
+
+ data = dict(self.user)
+
+ data["rememberMe"] = True
+ data["trustTokens"] = []
+ if self.session_data.get("trust_token"):
+ data["trustTokens"] = [self.session_data.get("trust_token")]
+
+ headers = self._get_auth_headers()
+
+ if self.session_data.get("scnt"):
+ headers["scnt"] = self.session_data.get("scnt")
+
+ if self.session_data.get("session_id"):
+ headers["X-Apple-ID-Session-Id"] = self.session_data.get("session_id")
- # We authenticate every time, so "remember me" is not needed
- data.update({'extended_login': False})
+ try:
+ self.session.post(
+ "%s/signin" % self.AUTH_ENDPOINT,
+ params={"isRememberMeEnabled": "true"},
+ data=json.dumps(data),
+ headers=headers,
+ )
+ except PyiCloudAPIResponseException as error:
+ msg = "Invalid email/password combination."
+ raise PyiCloudFailedLoginException(msg, error) from error
+
+ self._authenticate_with_token()
+
+ self._webservices = self.data["webservices"]
+
+ LOGGER.debug("Authentication completed successfully")
+
+ def _authenticate_with_token(self):
+ """Authenticate using session token."""
+ data = {
+ "accountCountryCode": self.session_data.get("account_country"),
+ "dsWebAuthToken": self.session_data.get("session_token"),
+ "extended_login": True,
+ "trustToken": self.session_data.get("trust_token", ""),
+ }
try:
req = self.session.post(
- self._base_login_url,
- params=self.params,
- data=json.dumps(data)
+ "%s/accountLogin" % self.SETUP_ENDPOINT, data=json.dumps(data)
)
- resp = req.json()
- except PyiCloudAPIResponseError as error:
- msg = 'Invalid email/password combination.'
- raise PyiCloudFailedLoginException(msg, error)
-
- # {'domainToUse': 'iCloud.com'}
- domain_to_use = resp.get('domainToUse')
- if domain_to_use != None:
- msg = f'Apple insists on using {domain_to_use} for your request. Please use --domain parameter'
- raise PyiCloudConnectionException(msg)
+ self.data = req.json()
+ except PyiCloudAPIResponseException as error:
+ msg = "Invalid authentication token."
+ raise PyiCloudFailedLoginException(msg, error) from error
+
+ def _authenticate_with_credentials_service(self, service):
+ """Authenticate to a specific service using credentials."""
+ data = {
+ "appName": service,
+ "apple_id": self.user["accountName"],
+ "password": self.user["password"],
+ }
- self.params.update({'dsid': resp['dsInfo']['dsid']})
+ try:
+ self.session.post(
+ "%s/accountLogin" % self.SETUP_ENDPOINT, data=json.dumps(data)
+ )
- if not os.path.exists(self._cookie_directory):
- os.mkdir(self._cookie_directory)
- self.session.cookies.save()
- logger.debug("Cookies saved to %s", self._get_cookiejar_path())
+ self.data = self._validate_token()
+ except PyiCloudAPIResponseException as error:
+ msg = "Invalid email/password combination."
+ raise PyiCloudFailedLoginException(msg, error) from error
- self.data = resp
- self.webservices = self.data['webservices']
+ def _validate_token(self):
+ """Checks if the current access token is still valid."""
+ LOGGER.debug("Checking session token validity")
+ try:
+ req = self.session.post("%s/validate" % self.SETUP_ENDPOINT, data="null")
+ LOGGER.debug("Session token is still valid")
+ return req.json()
+ except PyiCloudAPIResponseException as err:
+ LOGGER.debug("Invalid authentication token")
+ raise err
+
+ def _get_auth_headers(self, overrides=None):
+ headers = {
+ "Accept": "*/*",
+ "Content-Type": "application/json",
+ "X-Apple-OAuth-Client-Id": "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ "X-Apple-OAuth-Client-Type": "firstPartyAuth",
+ "X-Apple-OAuth-Redirect-URI": "https://www.icloud.com",
+ "X-Apple-OAuth-Require-Grant-Code": "true",
+ "X-Apple-OAuth-Response-Mode": "web_message",
+ "X-Apple-OAuth-Response-Type": "code",
+ "X-Apple-OAuth-State": self.client_id,
+ "X-Apple-Widget-Key": "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+ }
+ if overrides:
+ headers.update(overrides)
+ return headers
- logger.info("Authentication completed successfully")
- logger.debug(self.params)
+ @property
+ def cookiejar_path(self):
+ """Get path for cookiejar file."""
+ return path.join(
+ self._cookie_directory,
+ "".join([c for c in self.user.get("accountName") if match(r"\w", c)]),
+ )
- def _get_cookiejar_path(self):
- # Get path for cookiejar file
- return os.path.join(
+ @property
+ def session_path(self):
+ """Get path for session data file."""
+ return path.join(
self._cookie_directory,
- ''.join([c for c in self.user.get('apple_id') if match(r'\w', c)])
+ "".join([c for c in self.user.get("accountName") if match(r"\w", c)])
+ + ".session",
)
@property
def requires_2sa(self):
- """ Returns True if two-step authentication is required."""
- return self.data.get('hsaChallengeRequired', False) \
- and self.data['dsInfo'].get('hsaVersion', 0) >= 1
- # FIXME: Implement 2FA for hsaVersion == 2
+ """Returns True if two-step authentication is required."""
+ return self.data.get("dsInfo", {}).get("hsaVersion", 0) >= 1 and (
+ self.data.get("hsaChallengeRequired", False) or not self.is_trusted_session
+ )
+
+ @property
+ def requires_2fa(self):
+ """Returns True if two-factor authentication is required."""
+ return self.data["dsInfo"].get("hsaVersion", 0) == 2 and (
+ self.data.get("hsaChallengeRequired", False) or not self.is_trusted_session
+ )
+
+ @property
+ def is_trusted_session(self):
+ """Returns True if the session is trusted."""
+ return self.data.get("hsaTrustedBrowser", False)
@property
def trusted_devices(self):
- """ Returns devices trusted for two-step authentication."""
+ """Returns devices trusted for two-step authentication."""
request = self.session.get(
- '%s/listDevices' % self._setup_endpoint,
- params=self.params
+ "%s/listDevices" % self.SETUP_ENDPOINT, params=self.params
)
- return request.json().get('devices')
+ return request.json().get("devices")
def send_verification_code(self, device):
- """ Requests that a verification code is sent to the given device"""
+ """Requests that a verification code is sent to the given device."""
data = json.dumps(device)
request = self.session.post(
- '%s/sendVerificationCode' % self._setup_endpoint,
+ "%s/sendVerificationCode" % self.SETUP_ENDPOINT,
params=self.params,
- data=data
+ data=data,
)
- return request.json().get('success', False)
+ return request.json().get("success", False)
def validate_verification_code(self, device, code):
- """ Verifies a verification code received on a trusted device"""
- device.update({
- 'verificationCode': code,
- 'trustBrowser': True
- })
+ """Verifies a verification code received on a trusted device."""
+ device.update({"verificationCode": code, "trustBrowser": True})
data = json.dumps(device)
try:
- request = self.session.post(
- '%s/validateVerificationCode' % self._setup_endpoint,
+ self.session.post(
+ "%s/validateVerificationCode" % self.SETUP_ENDPOINT,
params=self.params,
- data=data
+ data=data,
)
- except PyiCloudAPIResponseError as error:
+ except PyiCloudAPIResponseException as error:
if error.code == -21669:
# Wrong verification code
return False
raise
- # Re-authenticate, which will both update the HSA data, and
- # ensure that we save the X-APPLE-WEBAUTH-HSA-TRUST cookie.
- self.authenticate()
+ self.trust_session()
return not self.requires_2sa
+ def validate_2fa_code(self, code):
+ """Verifies a verification code received via Apple's 2FA system (HSA2)."""
+ data = {"securityCode": {"code": code}}
+
+ headers = self._get_auth_headers({"Accept": "application/json"})
+
+ if self.session_data.get("scnt"):
+ headers["scnt"] = self.session_data.get("scnt")
+
+ if self.session_data.get("session_id"):
+ headers["X-Apple-ID-Session-Id"] = self.session_data.get("session_id")
+
+ try:
+ self.session.post(
+ "%s/verify/trusteddevice/securitycode" % self.AUTH_ENDPOINT,
+ data=json.dumps(data),
+ headers=headers,
+ )
+ except PyiCloudAPIResponseException as error:
+ if error.code == -21669:
+ # Wrong verification code
+ LOGGER.error("Code verification failed.")
+ return False
+ raise
+
+ LOGGER.debug("Code verification successful.")
+
+ self.trust_session()
+ return not self.requires_2sa
+
+ def trust_session(self):
+ """Request session trust to avoid user log in going forward."""
+ headers = self._get_auth_headers()
+
+ if self.session_data.get("scnt"):
+ headers["scnt"] = self.session_data.get("scnt")
+
+ if self.session_data.get("session_id"):
+ headers["X-Apple-ID-Session-Id"] = self.session_data.get("session_id")
+
+ try:
+ self.session.get(
+ f"{self.AUTH_ENDPOINT}/2sv/trust",
+ headers=headers,
+ )
+ self._authenticate_with_token()
+ return True
+ except PyiCloudAPIResponseException:
+ LOGGER.error("Session trust failed.")
+ return False
+
+ def _get_webservice_url(self, ws_key):
+ """Get webservice URL, raise an exception if not exists."""
+ if self._webservices.get(ws_key) is None:
+ raise PyiCloudServiceNotActivatedException(
+ "Webservice not available", ws_key
+ )
+ return self._webservices[ws_key]["url"]
+
@property
def devices(self):
- """ Return all devices."""
- service_root = self.webservices['findme']['url']
+ """Returns all devices."""
+ service_root = self._get_webservice_url("findme")
return FindMyiPhoneServiceManager(
- service_root,
- self.session,
- self.params
- )
-
- @property
- def account(self):
- service_root = self.webservices['account']['url']
- return AccountService(
- service_root,
- self.session,
- self.params
+ service_root, self.session, self.params, self.with_family
)
@property
def iphone(self):
+ """Returns the iPhone."""
return self.devices[0]
+ @property
+ def account(self):
+ """Gets the 'Account' service."""
+ service_root = self._get_webservice_url("account")
+ return AccountService(service_root, self.session, self.params)
+
@property
def files(self):
- if not hasattr(self, '_files'):
- service_root = self.webservices['ubiquity']['url']
- self._files = UbiquityService(
- service_root,
- self.session,
- self.params
- )
+ """Gets the 'File' service."""
+ if not self._files:
+ service_root = self._get_webservice_url("ubiquity")
+ self._files = UbiquityService(service_root, self.session, self.params)
return self._files
@property
def photos(self):
- if not hasattr(self, '_photos'):
- service_root = self.webservices['ckdatabasews']['url']
- self._photos = PhotosService(
- service_root,
- self.session,
- self.params
- )
+ """Gets the 'Photo' service."""
+ if not self._photos:
+ service_root = self._get_webservice_url("ckdatabasews")
+ self._photos = PhotosService(service_root, self.session, self.params)
return self._photos
@property
def calendar(self):
- service_root = self.webservices['calendar']['url']
+ """Gets the 'Calendar' service."""
+ service_root = self._get_webservice_url("calendar")
return CalendarService(service_root, self.session, self.params)
@property
def contacts(self):
- service_root = self.webservices['contacts']['url']
+ """Gets the 'Contacts' service."""
+ service_root = self._get_webservice_url("contacts")
return ContactsService(service_root, self.session, self.params)
@property
def reminders(self):
- service_root = self.webservices['reminders']['url']
+ """Gets the 'Reminders' service."""
+ service_root = self._get_webservice_url("reminders")
return RemindersService(service_root, self.session, self.params)
- def __unicode__(self):
- return 'iCloud API: %s' % self.user.get('apple_id')
+ @property
+ def drive(self):
+ """Gets the 'Drive' service."""
+ if not self._drive:
+ self._drive = DriveService(
+ service_root=self._get_webservice_url("drivews"),
+ document_root=self._get_webservice_url("docws"),
+ session=self.session,
+ params=self.params,
+ )
+ return self._drive
def __str__(self):
- as_unicode = self.__unicode__()
- if sys.version_info[0] >= 3:
- return as_unicode
- else:
- return as_unicode.encode('ascii', 'ignore')
+ return f"iCloud API: {self.user.get('apple_id')}"
def __repr__(self):
- return '<%s>' % str(self)
+ return f"<{self}>"
diff --git a/src/icloudpd/authentication.py b/src/icloudpd/authentication.py
index e92ea43..861b03a 100644
--- a/src/icloudpd/authentication.py
+++ b/src/icloudpd/authentication.py
@@ -29,8 +29,7 @@ def authenticator(logger: logging.Logger, domain: str):
# If password not provided on command line variable will be set to None
# and PyiCloud will attempt to retrieve from its keyring
icloud = pyicloud_ipd.PyiCloudService(
- domain,
- username, password,
+ username, password=password,
cookie_directory=cookie_directory,
client_id=client_id,
)
diff --git a/src/pyicloud_ipd/exceptions.py b/src/pyicloud_ipd/exceptions.py
index ef9c917..80b7a69 100644
--- a/src/pyicloud_ipd/exceptions.py
+++ b/src/pyicloud_ipd/exceptions.py
@@ -1,39 +1,50 @@
-
-class PyiCloudException(Exception):
- pass
+"""Library exceptions."""
-class PyiCloudConnectionException(PyiCloudException):
- pass
-
-class PyiCloudNoDevicesException(PyiCloudException):
+class PyiCloudException(Exception):
+ """Generic iCloud exception."""
pass
-class PyiCloudAPIResponseError(PyiCloudException):
- def __init__(self, reason, code):
+# API
+class PyiCloudAPIResponseException(PyiCloudException):
+ """iCloud response exception."""
+ def __init__(self, reason, code=None, retry=False):
self.reason = reason
self.code = code
- message = reason
+ message = reason or ""
if code:
message += " (%s)" % code
+ if retry:
+ message += ". Retrying ..."
+
+ super().__init__(message)
- super(PyiCloudAPIResponseError, self).__init__(message)
+class PyiCloudServiceNotActivatedException(PyiCloudAPIResponseException):
+ """iCloud service not activated exception."""
+ pass
+
+# Login
class PyiCloudFailedLoginException(PyiCloudException):
+ """iCloud failed login exception."""
pass
-class PyiCloud2SARequiredError(PyiCloudException):
- def __init__(self, url):
- message = "Two-step authentication required for %s" % url
- super(PyiCloud2SARequiredError, self).__init__(message)
+class PyiCloud2SARequiredException(PyiCloudException):
+ """iCloud 2SA required exception."""
+ def __init__(self, apple_id):
+ message = "Two-step authentication required for account: %s" % apple_id
+ super().__init__(message)
-class NoStoredPasswordAvailable(PyiCloudException):
+class PyiCloudNoStoredPasswordAvailableException(PyiCloudException):
+ """iCloud no stored password exception."""
pass
-class PyiCloudServiceNotActivatedErrror(PyiCloudAPIResponseError):
+# Webservice specific
+class PyiCloudNoDevicesException(PyiCloudException):
+ """iCloud no device exception."""
pass
I've not tested these changes in a docker container yet.
from icloud_photos_downloader.
following - thanks
from icloud_photos_downloader.
Thanks a lot. Because, all this have a "family" impact. I need to have my dear photos and videos close to me on my NAS. If I can contribute one way or the other I will.
Thanks a looooooot. Merci 🤗.
from icloud_photos_downloader.
There is hope for a fix, and would appreciate others testing if they have a chance: #733
Notes:
- many automated tests are broken, and need fixing
- there is currently no way to select .com vs .cn domains for icloud (icloud.com is used by default)
- there is currently no way to select a photo library, so "All Photos" is used by default
I've had success running a docker container and pulling down all photos/movies from my iCloud photo library:
docker build . -t icloudpd_dev
docker run -it \
--rm --name icloudpd \
-v ~/icloudpd-download:/data \
-e TZ=America/New_York \
icloudpd_dev \
icloudpd \
--directory /data \
--username [email protected] \
--watch-with-interval 3600
2023-12-09 16:23:42 DEBUG Authenticating...
iCloud Password:
2023-12-09 16:24:00 INFO Two-factor authentication is required
Please enter two-factor authentication code: 123456
2023-12-09 16:24:07 WARNING Failed to parse response with JSON mimetype
2023-12-09 16:24:10 INFO Great, you're all set up. The script can now be run without user interaction until 2SA expires.
You can set up email notifications for when the two-step authentication expires.
(Use --help to view information about SMTP options.)
2023-12-09 16:24:13 DEBUG Looking up all photos and videos from album All Photos...
2023-12-09 16:24:13 INFO Downloading 81324 original photos and videos to /data ...
2023-12-09 16:24:16 DEBUG Downloading /data/2023/12/09/IMG_1083.MOV...
2023-12-09 16:24:18 INFO Downloaded /data/2023/12/09/IMG_1083.MOV
2023-12-09 16:24:18 DEBUG Downloading /data/2023/12/09/IMG_1082.JPG...
from icloud_photos_downloader.
Hi,
the fix almost works. I have not configured a domain, and the current auth_fix
branch fails with this error:
[...]
File "/home/xxxx/.local/lib/python3.11/site-packages/pyicloud_ipd/base.py", line 393, in _authenticate_with_token
domain_to_use = req.get('domainToUse')
^^^^^^^
AttributeError: 'Response' object has no attribute 'get'
I have not fully understood what the req.get
statement is supposed to do and expect there should be some kind of key check first.
My temporary fix is
domain_to_use = None
try:
domain_to_use = req.get('domainToUse')
except AttributeError as error:
print("domainToUse attribute not found.")
With this, I am able to download my icloud photos again 🎉
from icloud_photos_downloader.
Hi,
the fix almost works. I have not configured a domain, and the current
auth_fix
branch fails with this error:[...] File "/home/xxxx/.local/lib/python3.11/site-packages/pyicloud_ipd/base.py", line 393, in _authenticate_with_token domain_to_use = req.get('domainToUse') ^^^^^^^ AttributeError: 'Response' object has no attribute 'get'
I have not fully understood what the
req.get
statement is supposed to do and expect there should be some kind of key check first.[ . . . ]
I've fixed this in the latest commit. The get function will handle missing keys gracefully, unfortunately, it was being called on an object that doesn't have the method - so even if the key existed, it would still fail with the AttributeError
req.json()
does have the get
method (as it returns a dictionary), and that is set to self.data
just above, so calling self.data.get("domainToUse")
should return None
if it doesn't exist in the response.
from icloud_photos_downloader.
It didn't work for me at first.
I ran it with:
docker run -it --rm -v $(pwd)/mynas/Photos/iCloudBackup:/data -e TZ=America/New_York icloudpd icloudpd --directory /data --username [email protected]
I had the following error:
pyicloud_ipd.exceptions.PyiCloudNoStoredPasswordAvailableException: No pyicloud password for [email protected] could be found in the system keychain. Use the `--store-in-keyring` command-line option for storing a password for this username.
But I just needed something quick and dirty to do a backup so I hardcoded my password in base.py
and it worked.
from icloud_photos_downloader.
@Braincoke
You need to supply your own email to the docker run
command:
docker run -it \
--rm \
-v $(pwd)/mynas/Photos/iCloudBackup:/data \
-e TZ=America/New_York \
icloudpd icloudpd --directory /data --username [email protected]
You can see that in the command above, I pass a dummy email, --username [email protected]
You would want to replace that with your actual apple id email.
from icloud_photos_downloader.
Thanks @cfurrow, I did pass my real apple ID to the command.
from icloud_photos_downloader.
@Braincoke - I took a quick look and the icloudpd auth flow should prompt for a password if it's not supplied in the command line --password your_password
and also not found in the keyring.
With the changes in the pyicloud
exception names, this exception isn't being caught by icloudpd to prompt for the password. It was previously looking for pyicloud_ipd.exceptions.NoStoredPasswordAvailable
and should now be looking for pyicloud_ipd.exceptions.PyiCloudNoStoredPasswordAvailableException
.
I'll review the pyicloud
exceptions referenced throughout icloudpd
and update.
from icloud_photos_downloader.
I am on unraid and hence cant test any of this stuff being reported, but looking forward to testing it once an update goes to the unraid app store.
from icloud_photos_downloader.
遇到了同样问题
from icloud_photos_downloader.
The docker solution does not seem to work for me: I'm still getting invalid email/password combo when attempting this via a build docker container:
560 git clone https://github.com/icloud-photos-downloader/icloud_photos_downloader.git
561 cd icloud_photos_downloader/
562 ls
563 docker build . -t icloudpd_dev
564 history
565 docker run -it --rm --name icloudpd -v ~/Pictures/Photos:/data -e TZ=America/Vancouver icloudpd_dev icloudpd --directory /data --username [email protected] --watch-with-interval 1800
2023-12-12 07:35:41 DEBUG Authenticating...
iCloud Password:
2023-12-12 07:35:44 ERROR Unknown reason
Traceback (most recent call last):
File "pyicloud_ipd/base.py", line 220, in authenticate
File "requests/sessions.py", line 637, in post
File "pyicloud_ipd/base.py", line 105, in request
File "pyicloud_ipd/base.py", line 127, in _raise_error
pyicloud_ipd.exceptions.PyiCloudAPIResponseError: Unknown reason
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "starters/icloudpd_ex.py", line 110, in <module>
File "starters/icloudpd_ex.py", line 106, in main
File "click/core.py", line 1157, in __call__
File "click/core.py", line 1078, in main
File "click/core.py", line 1688, in invoke
File "click/core.py", line 1434, in invoke
File "click/core.py", line 783, in invoke
File "icloudpd/base.py", line 317, in main
File "icloudpd/base.py", line 744, in core
File "icloudpd/authentication.py", line 31, in authenticate_
File "pyicloud_ipd/base.py", line 204, in __init__
File "pyicloud_ipd/base.py", line 228, in authenticate
pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError('Unknown reason'))
[8] Failed to execute script 'icloudpd_ex' due to unhandled exception!
~/.src/icloud_photos_downloader $>
i successfully logged into icloud via browser to confirm that I wasn't fat-fingering my password.
from icloud_photos_downloader.
@winteriscariot I think you're not getting the repo with the fix. The PR that is waiting to merge is https://github.com/scaraebeus/icloud_photos_downloader.git right now.
Also, this don't work for me. I've got the same error:
git clone https://github.com/scaraebeus/icloud_photos_downloader.git
docker build . -t icloudpd_scarabeus --no-cache
docker run -it --rm --name icloudpd -v /tmp/icloudpd-download:/data -e TZ=America/New_York icloudpd_scarabeus icloudpd --directory /data --username [email protected] --watch-with-interval 3600 --password "PASSWORD"
2023-12-12 11:24:26 DEBUG Authenticating...
2023-12-12 11:24:27 ERROR Unknown reason
Traceback (most recent call last):
File "pyicloud_ipd/base.py", line 220, in authenticate
File "requests/sessions.py", line 637, in post
File "pyicloud_ipd/base.py", line 105, in request
File "pyicloud_ipd/base.py", line 127, in _raise_error
pyicloud_ipd.exceptions.PyiCloudAPIResponseError: Unknown reason
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "starters/icloudpd_ex.py", line 110, in <module>
File "starters/icloudpd_ex.py", line 106, in main
File "click/core.py", line 1157, in __call__
File "click/core.py", line 1078, in main
File "click/core.py", line 1688, in invoke
File "click/core.py", line 1434, in invoke
File "click/core.py", line 783, in invoke
File "icloudpd/base.py", line 317, in main
File "icloudpd/base.py", line 744, in core
File "icloudpd/authentication.py", line 31, in authenticate_
File "pyicloud_ipd/base.py", line 204, in __init__
File "pyicloud_ipd/base.py", line 228, in authenticate
pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseError('Unknown reason'))
[7] Failed to execute script 'icloudpd_ex' due to unhandled exception!
from icloud_photos_downloader.
@lcmartos git checkout auth_fix
from icloud_photos_downloader.
@michaelmolino Working! Thank you! I suppose it's just a matter of time before they try and accept the patch.
from icloud_photos_downloader.
I got it working too - following steps did the trick for me (checking out the head "auth_fix" is the trick really):
wget https://github.com/scaraebeus/icloud_photos_downloader/archive/refs/heads/auth_fix.zip
mv auth_fix auth_fix.zip
unzip auth_fix
docker build . -t icloudpd_dev
docker run -it --rm --name icloudpd -v /mnt/fotos:/data -e TZ=America/Vancouver icloudpd_dev icloudpd --directory /data --username [email protected] --watch-with-interval 1800 --password XXX
from icloud_photos_downloader.
I got it working too - following steps did the trick for me (checking out the head "auth_fix" is the trick really): wget https://github.com/scaraebeus/icloud_photos_downloader/archive/refs/heads/auth_fix.zip mv auth_fix auth_fix.zip unzip auth_fix docker build . -t icloudpd_dev docker run -it --rm --name icloudpd -v /mnt/fotos:/data -e TZ=America/Vancouver icloudpd_dev icloudpd --directory /data --username [email protected] --watch-with-interval 1800 --password XXX
I followed these steps and get the error Missing apple_id field
:
2023-12-12 16:03:27 DEBUG Authenticating...
2023-12-12 16:03:28 ERROR Missing apple_id field
Traceback (most recent call last):
File "pyicloud_ipd/base.py", line 365, in _authenticate_with_token
File "requests/sessions.py", line 637, in post
File "pyicloud_ipd/base.py", line 156, in request
File "pyicloud_ipd/base.py", line 185, in _raise_error
pyicloud_ipd.exceptions.PyiCloudAPIResponseException: Missing apple_id field
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "starters/icloudpd_ex.py", line 110, in <module>
File "starters/icloudpd_ex.py", line 106, in main
File "click/core.py", line 1157, in __call__
File "click/core.py", line 1078, in main
File "click/core.py", line 1688, in invoke
File "click/core.py", line 1434, in invoke
File "click/core.py", line 783, in invoke
File "icloudpd/base.py", line 317, in main
File "icloudpd/base.py", line 744, in core
File "icloudpd/authentication.py", line 31, in authenticate_
File "pyicloud_ipd/base.py", line 283, in __init__
File "pyicloud_ipd/base.py", line 345, in authenticate
File "pyicloud_ipd/base.py", line 371, in _authenticate_with_token
pyicloud_ipd.exceptions.PyiCloudFailedLoginException: ('Invalid authentication token.', PyiCloudAPIResponseException('Missing apple_id field'))
[7] Failed to execute script 'icloudpd_ex' due to unhandled exception!
I'm sure this is basic user error on my part but I thought I would share in case it isn't.
I also have to pass the password as an option. If I don't, it complains that it isn't in the keyfile.
from icloud_photos_downloader.
I followed these steps and get the error
Missing apple_id field
:[ . . . ]
I'm sure this is basic user error on my part but I thought I would share in case it isn't.
I also have to pass the password as an option. If I don't, it complains that it isn't in the keyfile.
I'm not sure all of the ways this error can manifest - so this may not apply, however, in some of my testing I was able to get this error if passing an invalid username and/or incorrect password. May want to double check there are no typos there.
from icloud_photos_downloader.
I got it working too - following steps did the trick for me (checking out the head "auth_fix" is the trick really): wget https://github.com/scaraebeus/icloud_photos_downloader/archive/refs/heads/auth_fix.zip mv auth_fix auth_fix.zip unzip auth_fix docker build . -t icloudpd_dev docker run -it --rm --name icloudpd -v /mnt/fotos:/data -e TZ=America/Vancouver icloudpd_dev icloudpd --directory /data --username [email protected] --watch-with-interval 1800 --password XXX
that worked for me and photos are syncing again. Thanks so much everyon for the great and quick team work on this one. Cheers.
from icloud_photos_downloader.
I followed these steps and get the error
Missing apple_id field
:
[ . . . ]
I'm sure this is basic user error on my part but I thought I would share in case it isn't.
I also have to pass the password as an option. If I don't, it complains that it isn't in the keyfile.I'm not sure all of the ways this error can manifest - so this may not apply, however, in some of my testing I was able to get this error if passing an invalid username and/or incorrect password. May want to double check there are no typos there.
Thanks for the tip. Just for the record, I've double- and triple-checked and I don't have any typos in username or password.
from icloud_photos_downloader.
Thanks for all the work on this everyone.
Seems like there are fixes in place for some.
I can test too it it gets to the unraid community app store.
from icloud_photos_downloader.
I think I found the issue I was having. My password has a special character and if I escape it it works. I should note that I didn't have to do this in 1.16.2.
It doesn't seem to be saving my session, but that's a problem for another day.
Thanks everyone :)
from icloud_photos_downloader.
Thanks for all the work on this everyone.
Seems like there are fixes in place for some.
I can test too it it gets to the unraid community app store.
Yeah I use unraid and I have no idea how to apply the fix lol
from icloud_photos_downloader.
Any solution yet?
from icloud_photos_downloader.
I got it working too - following steps did the trick for me (checking out the head "auth_fix" is the trick really): wget https://github.com/scaraebeus/icloud_photos_downloader/archive/refs/heads/auth_fix.zip mv auth_fix auth_fix.zip unzip auth_fix docker build . -t icloudpd_dev docker run -it --rm --name icloudpd -v /mnt/fotos:/data -e TZ=America/Vancouver icloudpd_dev icloudpd --directory /data --username [email protected] --watch-with-interval 1800 --password XXX
how to with this command add --delete-after-download, because its not working
from icloud_photos_downloader.
it's multiple commands. your "delete after download" belongs to icloudpd and belongs to the last command:
wget https://github.com/scaraebeus/icloud_photos_downloader/archive/refs/heads/auth_fix.zip
mv auth_fix auth_fix.zip
unzip auth_fix
docker build . -t icloudpd_dev
docker run -it --rm --name icloudpd -v /mnt/fotos:/data -e TZ=America/Vancouver icloudpd_dev icloudpd --directory /data --username [email protected] --watch-with-interval 1800 --password XXX --delete-after-download
from icloud_photos_downloader.
I really fine this just above my head, maybe I can do It but I dont want to mess things up, any idea when the unraid image would be ready, and I dont mean an exact time im justing wondering if anyone is working to update that part or is this manual fix is it, this is what we have either use the fix or dont?
from icloud_photos_downloader.
it's multiple commands. your "delete after download" belongs to icloudpd and belongs to the last command:
wget https://github.com/scaraebeus/icloud_photos_downloader/archive/refs/heads/auth_fix.zip
mv auth_fix auth_fix.zip
unzip auth_fix
docker build . -t icloudpd_dev
docker run -it --rm --name icloudpd -v /mnt/fotos:/data -e TZ=America/Vancouver icloudpd_dev icloudpd --directory /data --username [email protected] --watch-with-interval 1800 --password XXX --delete-after-download
its downloading photos but didnt delete them after download
from icloud_photos_downloader.
it's multiple commands. your "delete after download" belongs to icloudpd and belongs to the last command:
wget https://github.com/scaraebeus/icloud_photos_downloader/archive/refs/heads/auth_fix.zip
mv auth_fix auth_fix.zip
unzip auth_fix
docker build . -t icloudpd_dev
docker run -it --rm --name icloudpd -v /mnt/fotos:/data -e TZ=America/Vancouver icloudpd_dev icloudpd --directory /data --username [email protected] --watch-with-interval 1800 --password XXX --delete-after-download
Works with above for me on Linux and Docker. Great work regarding the workaround! All kudos!
But I can't figure out how to download shared libraries with this - is it possible?
from icloud_photos_downloader.
I got it working too - following steps did the trick for me (checking out the head "auth_fix" is the trick really): wget https://github.com/scaraebeus/icloud_photos_downloader/archive/refs/heads/auth_fix.zip mv auth_fix auth_fix.zip unzip auth_fix docker build . -t icloudpd_dev docker run -it --rm --name icloudpd -v /mnt/fotos:/data -e TZ=America/Vancouver icloudpd_dev icloudpd --directory /data --username [email protected] --watch-with-interval 1800 --password XXX
Can you elaborate on what exactly to do here? I can't move a file i don't have into a zip file, and so on..
from icloud_photos_downloader.
@NGC3982
Here you go: https://github.com/OrZidkani/icloudpd_auth_fix
Make it easier to run.
Big thanks to @scaraebeus
from icloud_photos_downloader.
Is there a update for the PIP (windows) Version? I don't use docker.
Or do I have to wait for next friday?
Thanks for all the efforts to fix it.
Frank
from icloud_photos_downloader.
Is there a update for the PIP (windows) Version? I don't use docker.
Or do I have to wait for next friday?Not sure on timing for the official update. Unfortunately for my fix to be pulled in, there are a few things that need to be in place on my branch for it to be accepted. I've been working through updating the various tests and code coverage and in doing so I am finding some other parts of the
pyicloud_ipd/base.py
auth flow that are no longer working correctly due to (possibly) recent changes in how the responses are coming through.I'm working through it but it will take some time - especially with the holidays coming up.
The only option I'm aware of at this time is to consume the auth_fix branch directly if your environment and setup allows you - with the risk that it has yet to be fully tested and vetted.
Works on Debian 12 with docker for me until some period of time where the Apple servers either:
hang up, as icloudpd appears to idle in the middle of downloading
attempt negotiate 2fa and send me an authentication code while icloudpd is in the middle of downloading
In both cases there isn't any debug or info logging printed indicating why icloudpd paused, but my network traffic goes to near zero, so I'm guessing the sessions is closed by apple.
Additionally, I'm attempting to download over 180k photos, so that might be why apple is hanging up on me periodically. It doesn't explain the 2fa prompt while icloudpd is running and actively downloading though.
The expected behavior where re-running icloupd picks up where it left off results in diminishing returns:
1st session, 4k downloads.
2nd session: 4k skipped and 2k additional downloads.
3rd session: 6k skipped and 500 additional completed downloads.
And so on until no additional downloads complete at all. This might be from pyicloud json handling?
from icloud_photos_downloader.
Works! For those using the above solution, a minor change is necessary (I'm on macOS): in the Shell script, change /mnt/iCloud
to $(pwd)
. Then, in Docker Desktop, add the current directory under Settings --> Resources --> File sharing.
ETA: The $(pwd)
comes from this documentation.
Edit 2: Don't use $(pwd)
! The documentation version works because it refers to where you are, but in the script, it cd
s to a temp directory so it would download there. Change it to the directory you want the downloads to go to!
from icloud_photos_downloader.
Is saving to the keychain not working for anyone else with the auth_fix branch?
Save password in keyring? [y/N]: y
Two-step authentication required. Your trusted devices are:
0: SMS to ********##
Which device would you like to use? [0]:
Previously I was able to see an option to use a 2FA code instead of SMS
from icloud_photos_downloader.
Is saving to the keychain not working for anyone else with the auth_fix branch?
Save password in keyring? [y/N]: y Two-step authentication required. Your trusted devices are: 0: SMS to ********## Which device would you like to use? [0]:
Previously I was able to see an option to use a 2FA code instead of SMS
Steps to reproduce? With the branch I just got a 2FA code on my devices like normal, and it prompted me in the terminal. I don't remember it giving me an option between SMS or 2FA code, though.
from icloud_photos_downloader.
Is saving to the keychain not working for anyone else with the auth_fix branch?
Save password in keyring? [y/N]: y Two-step authentication required. Your trusted devices are: 0: SMS to ********## Which device would you like to use? [0]:
Previously I was able to see an option to use a 2FA code instead of SMS
Steps to reproduce? With the branch I just got a 2FA code on my devices like normal, and it prompted me in the terminal. I don't remember it giving me an option between SMS or 2FA code, though.
With icloudpd --username "USERNAME" --password "PASSWORD"
it asks me for a 2FA right away
When I try to save in the keyring it only offers SMS instead of 2FA:
docker run --rm -it --name icloud \
-v /tmp/icloudpd/data:/data \
-v /tmp/icloudpd/cookies:/cookies \
-e "TZ=America/Los_Angeles" \
icloudpd:auth_fix \
icloud --username USERNAME
Enter iCloud password for USERNAME:
Save password in keyring? [y/N]: y
Two-step authentication required. Your trusted devices are:
0: SMS to ********##
Which device would you like to use? [0]
Edit: for now I've just resolved myself to running icloudpd
with a username and password until everything gets sorted out in the main branch
from icloud_photos_downloader.
It appears the command line flow for the base pyicloud_ipd
service (which you are invoking with icloud --username USERNAME
) was never updated in the pyicloud - 1.0.0
release to use the new 2FA feature, so it prompts as if it is expecting 2SA (choosing a device to send the code to first).
It also appears, from what I can tell, the icloudpd
flow doesn't offer to save in the keyring directly.
Based on the above, as you've saved the password in the keyring now, try using the icloudpd
directly without the --password
option and it should retrieve from the keyring - and ask for the 2FA without prompting for a device selection.
The option to offer saving in the keyring could be moved into the icloudpd
flow if needed. Not sure if there is already an issue for that or not.
from icloud_photos_downloader.
Previous behaviour was that saving to the keychain required SMS MFA, it did not have an option for Apple's built in MFA.
Performing a download would then trigger a second MFA prompt, which had two options for MFA; SMS and Apple.
I noticed this behaviour when attempting to relocate the MFA cookie in my container from $HOME/.local to /config. Took me a lot of deletes/recreates to get it working reliably.
Edit: I think it's also worth mentioning that I actually had two SMS numbers I could use for three of my containers. I have added my phone number as a trusted number to the three accounts of my family members. This allows me to re-authenticate their containers without me needing access to their devices. Very useful feature.
The current pyicloud - 1.0.0
2SA/2FA implementation doesn't appear to take this into account. The command line implementation for pyicloud
doesn't even test for 2FA (hence why it drops straight into asking which device to use). The underlying hooks in the pyicloud
2FA implementation also limit the flexibility for the above use case.
On top of this, there are two separate endpoints used to validate codes depending on whether the 2SA or 2FA flow is triggered. It's possible the endpoints don't really matter and the same endpoint can be used regardless - I'll have to test.
Proposed solution:
- Leave the default behavior as is and add a switch to list devices to alternatively send a code to
- When initiating
icloudpd
if you have a default iDevice and the account is enabled for HSA2 (I believe all accounts are now enabled for HSA2 at this point), prompt for 2FA code directly as it should be auto sent to the device- This would likely cover a majority of use cases
- There is an edge case where an account may not have an iDevice associated with it - in this case, there is likely still an SMS number to send to (potentially Android instead of iDevice). This is already covered in the default behavior and automatically triggers the list device flow.
- Adding a 'list devices' switch (
icloudpd --list-devices
) could trigger the flow to collect and show the available devices associated with the account, allowing the user to select a different device to send a code to (and displaying the option to bypass this and enter the code if you decide to go with the code auto sent to the default device)- With the above switch, a code would still likely be auto sent to an iDevice - choosing a different device (or SMS) to send a code to would need to be tested to verify which validation endpoint accepts the second code (manual request) - (or if both do)
- When initiating
from icloud_photos_downloader.
Based on the above, as you've saved the password in the keyring now, try using the
icloudpd
directly without the--password
option and it should retrieve from the keyring - and ask for the 2FA without prompting for a device selection.
I think in theory this should work but the code I get via SMS can never be verified successfully:
Enter iCloud password for USERNAME:
Save password in keyring? [y/N]: y
Two-step authentication required. Your trusted devices are:
0: SMS to ********##
Which device would you like to use? [0]: 0
Please enter validation code: ######
Failed to verify verification code
from icloud_photos_downloader.
I retract my statement about the pyicloud - 1.0.0
not taking the 2SA/2FA into account, it does, the minimal fix in the auth_fix branch does not include the updated cmdline.py to expose this in the pyicloud_ipd
direct invoking.
@tanookiben - when you attempt to verify the SMS code, have you also already received an auto-generated code to one of your iDevices?
If so, what happens if you enter that code at the prompt instead of the one sent through SMS?
This would at give me a bit of a clue as to where there is something end point related and/or if it's still expecting the first code.
from icloud_photos_downloader.
I'm not near my laptop to test, but IIRC, mine crashed at the point where it attempted to list the phone numbers.
from icloud_photos_downloader.
Is SMS based multifactor authentication no longer possible?
With the new method used in pyicloud - 1.0.0
it splits the flow between 2SA (using SMS) and 2FA (Auto-sent code) based on whether the iCloud account itself is set for HSA version 2 (it appears all accounts may now be set for this).
The way the base pyicloud
does this check, assuming all accounts are now set as HSA version 2, there is actually no way the 2SA flow would ever be called. I uncovered this awhile ago doing some other testing with an account that doesn't have an iDevice (using the free iCloud service as standalone). In this scenario, I identified a flag noting if the account has a qualifying device. Adding this in to the 2FA check (requiring this to be true) now allows the flow to fall back on the 2SA - but only if there isn't a qualifying device on the account.
I've started working on an enhancement to allow selecting an SMS for 2sa validation. A 2fa code will still be sent to the iDevice automatically, but you should be able to choose an SMS, get a code there, and validate with that instead.
If you want to check that out, see the 2fa_enhancements
branch under scaraebeus/icloud_photos_downloader
- keep in mind this is experimental though. In that branch, you can use the --list-devices
option to present the list of SMS devices and choose to send a code there (or choose to enter the auto sent 2FA code).
It may be worth starting a separate issue for this to be tracked outside of this thread.
from icloud_photos_downloader.
I depend on this tool and have been following this issue. Thanks so much for all the hard work and getting this fixed so quickly. You are appreciated! 🙏
from icloud_photos_downloader.
I've built a new container with iCloud 1.17.0, but the behaviour is not the same. I am no longer presented with the option to perform SMS based multifactor authentication. I also receive a warning about it not being able to parse JSON, which I've not seen before but I'm not too worried about, as the application downloads the photos regardless:
2023-12-20 12:08:01 DEBUG Configure password 2023-12-20 12:08:01 DEBUG Adding password to keyring file: /config/python_keyring/keyring_pass.cfg 2023-12-20 12:08:01 DEBUG Switched to icloudpd: 1.17.0 Enter iCloud password for [email protected]: Save password in keyring? [y/N]: y Two-step authentication required. Please enter validation code (string) --> 137446 2023-12-20 12:08:46 INFO Starting container initialisation 2023-12-20 12:08:46 DEBUG Generate MFA cookie using password stored in keyring file 2023-12-20 12:08:47 DEBUG Switched to icloudpd: 1.17.0 2023-12-20 12:08:47 ERROR Authentication required for Account. (421) Please enter two-factor authentication code: 119033 2023-12-20 12:09:20 WARNING Failed to parse response with JSON mimetype
Is SMS based multifactor authentication no longer possible?
Can you share the command you used here? I'm able to save credentials to the keyring with icloud --username "USERNAME"
but icloudpd --username "USERNAME"
without a --password
flag always prompts me for a password
from icloud_photos_downloader.
Just checking in on this issue, tried the lastest release today and it seems like it is still broken. Was #734, supposed to resolve this issue?
from icloud_photos_downloader.
Just checking in on this issue, tried the lastest release today and it seems like it is still broken
If you see the same issue then post here all details. Otherwise open new issue.
from icloud_photos_downloader.
Related Issues (20)
- Can't authenticate after redeploy HOT 1
- Shared object error on setup HOT 4
- Path not readable HOT 1
- Add option to create file listing all albums contents HOT 1
- Error: Invalid value for '-d' / '--directory': Path 'P:\\Photos\\neu\\09 Delta iClouds\\Steffi' does not exist. HOT 2
- Version `GLIBC_2.35' not found on Debian Bullseye HOT 2
- Authorization for not 2fa account failed with error 'Invalid authentication token.' HOT 2
- 1.17.1 introduces ModuleNotFoundError: No module named 'starters' HOT 9
- impossible to generate a two-factor token HOT 7
- Totally New and need help HOT 3
- Support for bursts HOT 4
- Container on Synology NAS does not start HOT 17
- PyPI does not work with user/pass any more HOT 5
- 1.17.3 introduces "Permission denied" on Synology NAS HOT 10
- Support loading parameters from environment variables
- [Latest Edits] Allow downloading of edited files HOT 1
- Authorization for not 2fa account failed with error 'Missing apple_id field (Missing apple_id field)' HOT 1
- Can't login "Bad username or password for" HOT 2
- icloudpd 1.17.3: Unhandled Exception While Authenticating HOT 10
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from icloud_photos_downloader.