instaloader / instaloader Goto Github PK

Extracting the followers and followees could be a great enhancement to the what Instaloader is.

*I'm not a coder and more of a data guy/linux admin so i don't know if i should be suggesting these enhancements when i can't code them or not. i hope these could be constructive :)

Would be nice if you can add support to download #hashtag pics and videos.

Since there could be 1000s or even millions for #hashtag feed, maybe you can have parameters for max # of media to download, or from x date to y date, etc.

Here's example Link: #WWF :
Has over 1 million media items.
Here's the URL to get it directly ( graph node json results, with next_cursor / has more pagination)

Pleeaase can you add it ? I can give you amazon gift card or something if you don't want $$ for compensation. Working on a class research project / AI stuff.

instaloader should be able to download the recent n photos the user liked.

Also save session (if useful) when script terminates because of KeyboardInterrupt.

Hi,

I have been able to download any star wars hashtags, and mix of star wars hashtags combined with others, but then I tried #starwars and #nsfw and the script completes without an error before downloading even one picture. I then tried #nsfw by itself and it did not work.

I feel the reason could be that you might need to be logged to access those restricted hashtags.

It could also be one reason why the script ended up saving less images than what was displayed in browser when we tested a few weeks ago.

1. Could you please confirm that it also happens to you.

2. If my assumption is correct, maybe as a workaround there could be a way to use the script to log in an account before downloading restricted hashtags?

Thanks

I have not seen that one before. I am on 2.2.3

[21428] #starwars #starwars/antonini.art.dzny_2017-08-12_10-54-15.jpg exists

Errors occured:
Unable to get owner name of post BXwcNmmDkok: 404 -- using 'UNKNOWN'.
Download hashtag #starwars: 404 when accessing https://instagram.fymy1-1.fna.fbcdn.net/t51.2885-15/e35/20686986_1636403346432445_1785624497169104896_n.jpg.

When downloading a profile, also store the profile's ID. This ID shall later be used to identify the profile if the name of the profile changed.

What comes naturally is when you fetch certain hashtags, you want to also know who has responded to that hashtag or in anyway was related to that hashtag. what is shown right now is pure set of comments pictures and their comments with that hashtag but not who has actually posted them:
instaloader "#emmawatson" --comments

POST'S CAPTION

A vontade de rir é grande, mas a de chorar é maior
~Daniel Antunes @dantunes.c🌲🌲🌲🌲🌲🌲🌲🌲🌲🌲🌲
#Castelobruxo #Castelobruxooficial #harrypotterbrasil #hogwarts #harrypotter #harrypotterbr #Harrypotterforever #jkrowling #potterhead #emmawatson #hp
🍁🍁🍁🍁🍁🍁🍁🍁🍁🍁🍁
-Siga nossa conta reserva exclusiva: @Castelobruxoreserva 🌴🌴🌴🌴🌴🌴🌴🌴🌴🌴🌴


COMMENTS

[
    {
        "created_at": 1504990313,
        "text": "Nesse dia",
        "owner": {
            "profile_pic_url": "https://instagram.fevn1-1.fna.fbcdn.net/t51.2885-19/s150x150/21372894_1075269222608866_4733110202788741120_a.jpg",
            "username": "_itsnaah_",
            "id": "2988718154"
        },
        "id": "17875132852141642"
    },
    {
        "created_at": 1504990305,
        "text": "Eu chorei tanto",
        "owner": {
            "profile_pic_url": "https://instagram.fevn1-1.fna.fbcdn.net/t51.2885-19/s150x150/21372894_1075269222608866_4733110202788741120_a.jpg",
            "username": "_itsnaah_",
            "id": "2988718154"
        },
        "id": "17888680897073913"
    },
    {
        "created_at": 1504990248,
        "text": "Oh @jupontesp \ud83d\ude2d\ud83d\ude02",
        "owner": {
            "profile_pic_url": "https://instagram.fevn1-1.fna.fbcdn.net/t51.2885-19/s150x150/21434031_1289801374463983_3543018590913429504_a.jpg",
            "username": "clauap29",
            "id": "4284789075"
        },
        "id": "17885256886115505"
    }
]

so i know the people who have responded to the post containing the hashtag but not the people who have actually posted that hashtag.

started testing with fetching comments for hashtags and faced the error below always ending after the first comment being processed. test both on English and Persian with the same error.

root@lucifer:~# instaloader --comments "#fourtyfour"
Retrieving pictures with hashtag #fourtyfour
[  1] #fourtyfour #fourtyfour/2017-08-19_18-30-52.jpg [Who doesn't love summer? NO O…] Traceback (most recent call last):
  File "/usr/local/bin/instaloader", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.5/dist-packages/instaloader.py", line 1384, in main
    args.profile_pic_only, args.fast_update, args.stories, args.stories_only, args.only_if)
  File "/usr/local/lib/python3.5/dist-packages/instaloader.py", line 1200, in main
    filter_func=filter_func)
  File "/usr/local/lib/python3.5/dist-packages/instaloader.py", line 1008, in download_hashtag
    downloaded = self.download_post(post, target='#' + hashtag)
  File "/usr/local/lib/python3.5/dist-packages/instaloader.py", line 805, in download_post
    self.update_comments(filename, post)
  File "/usr/local/lib/python3.5/dist-packages/instaloader.py", line 599, in update_comments
    comments.extend(post.get_comments())
  File "/usr/local/lib/python3.5/dist-packages/instaloader.py", line 301, in get_comments
    if self.comments == len(comments_in_metadata['edges']):
KeyError: 'edges'

There should be an option --liked-only, which, if --login is given, prevents downloading pictures which are not liked by the user. This should be applied to both regular download of profiles and downloads from the user's feed (option requested in issue #14).
In instaloader.download() function, this filter might by implemented with an filter_func parameter, just like in instaloader.download_feed_pics().

Just hit that error from branch v3-dev (while doing my regular daily downloads)

[790] #starwarscosplay #starwarscosplay/its.just.a_spark_2017-08-08_17-51-47.jpg [My almost finished mando twi'…]
JSON Query to graphql/query: HTTP error code 502. [retrying]
Traceback (most recent call last):
  File "c:\python\scripts\instaloader-script.py", line 11, in <module>
    load_entry_point('instaloader==3.0rc0', 'console_scripts', 'instaloader')()
  File "c:\python\lib\site-packages\instaloader.py", line 1224, in main
    args.profile_pic_only, args.fast_update, args.stories, args.stories_only)
  File "c:\python\lib\site-packages\instaloader.py", line 1039, in main
    self.download_hashtag(hashtag=pentry[1:], max_count=max_count, fast_update=fast_update)
  File "c:\python\lib\site-packages\instaloader.py", line 852, in download_hashtag
    for post in self.get_hashtag_posts(hashtag):
  File "c:\python\lib\site-packages\instaloader.py", line 834, in get_hashtag_posts
    lambda d: d['data']['hashtag']['edge_hashtag_to_media'])
  File "c:\python\lib\site-packages\instaloader.py", line 340, in graphql_node_list
    edge_struct = edge_extractor(data)
  File "c:\python\lib\site-packages\instaloader.py", line 834, in <lambda>
    lambda d: d['data']['hashtag']['edge_hashtag_to_media'])
TypeError: 'NoneType' object is not subscriptable

Encode username in default session file name, maybe hashed.

Firstly, thanks for this great tool 🙂

I have a question: would it be possible to save the profile ID in file names?

Example:

1. I'm looking for photos matching the hashtag #cat
2. These are the profiles/users with photos/videos matching that hashtag:

• @cute_cats
• @catsofjapan
• @catmania

3. These should be the downloaded files, saved as:

• cute_cats__2017-06-24_14-30-06.jpg
• catsofjapan__2017-06-24_14-30-20.jpg
• catmania__2017-06-24_14-30-36.jpg
• catmania__2017-06-24_14-31-24.mp4

I'd like to see this as a feature, being able to download all users that use a given hashtag. This option could result in huge amounts of data so a default setting of only get x amount of users with a flag to override would be a good idea.

Instaloader fails to detect and download videos inside carousels, so it automatically skips the videos and downloads pictures only. it's like i'm using -V option but i'm not.

Absolutely love this software! Something that would be helpful is if deleted posts had their filename changed to indicate they were deleted. For example, after downloading all the posts in a profile, when subsequently running InstaLoader with a certain flag then when it scans through all the photos in a profile and finds one that is missing based on the previous downloading, then that file that was previously downloaded is renamed with a postfix of "-deleted."
Looking forward to feedback! I think this would really amplify this tool.
Thanks!

Trying to download items from your own private profile while being logged in fails. Checking that data["entry_data"]["ProfilePage"][0]["user"]["media"]["nodes"] == [] before raise solves the issue.

When logging in, I can see the aforementioned error.

As I couldn't find any verbose flag in the program, I can't apport more backtraces than the following:

Session file does not exist yet - Logging in.
Fatal error: Login error! Connection error!

The error seems to come from here:

https://github.com/Thammus/instaloader/blob/9a72f85a0e31e2af54660f8469cce0db9e7db9c7/instaloader.py#L824

I think that Instagram changed something.

curl returns the HTTP 404 error when forging the login request.

hello,

whan an IG story has a video in it, the file doesn't get the correct .mp4 file extension, instead it's 3 random characters. videos from the normal feed appears named correctly.

Example:

[ 1/ 1] kelseyplum10/2017-11-07 17.48.32 BbM4hNxBBpt.jpg exists kelseyplum10/2017-11-07 17.48.32 BbM4hNxBBpt.6df

% file "kelseyplum10/2017-11-07 17.48.32 BbM4hNxBBpt.6df"
kelseyplum10/2017-11-07 17.48.32 BbM4hNxBBpt.6df: ISO Media, MP4 Base Media v1 [IS0 14496-12:2003]

Beneficial for those that upload the files to cloud.

Much like what youtube-dl does

Hi,

Great work on the username in filename or folder by username!

I have been using the previous version of script for about a month now and I was looking forward so much to get the username in the filename. This is so important to be able to keep track the original poster information. Love it, works perfectly.

Now here is my challenge when tracking hashtags: I would like to have a way separate my hashtags by folders just like it did before, but also having the option to have the images with username in filename placed automatically inside the hashtag folders.

Having the hashtag in the filename is problematic for when the same image appears in multiple folders under different hashtags. You want to be able to remove duplicate images with the same filename when they are moved from two different hashtag folders (temporary location) into a more permanent folder.

My work around for now is to create folders by hashtag manually and run the script from within each folder.

I am using a batch file to run instaloader like this, but then I end up with a batch file in each folder instead of having my batch files outside the image folders.

From the #starwars folder I run c:\python\scripts\instaloader #starwars --hashtag-username --no-profile-subdir
From the #lightsaber folder I run c:\python\scripts\instaloader #lightsaber --hashtag-username --no-profile-subdir

Example of an image of Darth Vader posted under two separate hashtags #starwars and #lightsaber
C:\python\Scripts\ #starwars\ username_date.jpg
C:\python\Scripts\ #lightsaber\ username_date.jpg

Then lets say I really like the image and I want to keep it. Now, I may not remember that I moved it to my organized images collection folder already (a Darth Vader folder in that case) when I see it again while looking at it from a different hashtag folder. When I move the second image again, I would expect the two images to overwrite each others.

Thank you again for the great work and help.

Side question: Is there an easy way to "batch run multiple scripts in separate command windows" but all at once? I manually initiate each of my batch file one at the time and it does not seems to be efficient.

Just wants to know how can I do/use a list of profiles that I wish to download from.
Should I use a .txt? How should I proceed?

when i search hashtags in instaloader the number of results received differ from what instagram actually shows:
https://www.instagram.com/explore/tags/salam/
instaloader "#salam"

UPDATE: my terminal took a persian character and turned it into an arabic one (they look the same but different in encoding) and that was why i received the previous results on persian language. what i do now is that i type my text inside a browser and then paste it into terminal...

Is it possible to fetch people tagged inside a instagram picture who might not be tagged inside the captions? since we can view this from the web i think it is possible but i had to ask anyways.
if yes i think it would be beneficial to add the feature since it would add to the completeness of data fetched while instaloading. a lot of instagrammers tag people inside the pictures but not the captions.

To facilitate using instaloader as a module, it would be a good idea to document the exported methods.

When a error like "user %s does not exist" occurs when downloading multiple profiles, it should just print a warning and continue instead of aborting.

the feature --only-if seems to be a powerful feature for combining different results with logic, yet it is very unclear for people not savvy enough, could you please clear what options can be given to --only-if, how and when we can use it and what are the capabilities? even when i read the explanation for this feature i cannot fully understand what it can or cannot do.

When attempting to download private profiles, and it reuses a session (thus does not ask for login credentials), and the profile is not followed (i.e. "no pics found"), it should ask for new login credentials.

Hi, I've been getting these errors message after a while, it happened last night on a different hashtag. Saw that 4-5 times within 24 hours.

Not sure why it is happening... I was just running 2 scripts concurrently without the fast download option.

[3591] #stormtrooper #stormtrooper/digitalanim4l_2017-07-24_01-45-26.jpg [My new coffee mug makes me sm…] #stormtrooper/digitalanim4l_2017-07-24_01-45-26.mp4 Traceback (most recent call last):
File "c:\python\lib\site-packages\urllib3\response.py", line 302, in _error_catcher
yield
File "c:\python\lib\site-packages\urllib3\response.py", line 384, in read
data = self._fp.read(amt)
File "c:\python\lib\http\client.py", line 449, in read
n = self.readinto(b)
File "c:\python\lib\http\client.py", line 493, in readinto
n = self.fp.readinto(b)
File "c:\python\lib\socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "c:\python\lib\ssl.py", line 1002, in recv_into
return self.read(nbytes, buffer)
File "c:\python\lib\ssl.py", line 865, in read
return self._sslobj.read(len, buffer)
File "c:\python\lib\ssl.py", line 625, in read
v = self._sslobj.read(len, buffer)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\python\scripts\instaloader-script.py", line 11, in
load_entry_point('instaloader==2.0', 'console_scripts', 'instaloader')()
File "c:\python\scripts\instaloader.py", line 1050, in main
args.fast_update)
File "c:\python\scripts\instaloader.py", line 897, in download_profiles
download_comments=download_comments)
File "c:\python\scripts\instaloader.py", line 744, in download_hashtag
download_comments=download_comments)
File "c:\python\scripts\instaloader.py", line 645, in download_node
date_epoch=date)
File "c:\python\scripts\instaloader.py", line 337, in download_pic
shutil.copyfileobj(resp.raw, file)
File "c:\python\lib\shutil.py", line 79, in copyfileobj
buf = fsrc.read(length)
File "c:\python\lib\site-packages\urllib3\response.py", line 401, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "c:\python\lib\contextlib.py", line 100, in exit
self.gen.throw(type, value, traceback)
File "c:\python\lib\site-packages\urllib3\response.py", line 320, in _error_catcher
raise ProtocolError('Connection broken: %r' % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)", ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

Thanks

hello,

When I save a file directly from instagram website, it uses a code (mediaid?) that's not the shortcode available in URL. I have an instagram archive created by another tool, and it uses that as basis for the filename.

Am I missing something, or is it not possible to get this into a filename pattern with instaloader?

Example of wanted filename: "2017-11-01 17.52.20 1638589741288153292_549775569.jpg" (The date part I've figured out)

If not possible, would it be possible to add it?

edit: I was slightly mistaken. The example filename I gave is not related to the original filename from instagram. Must be using some other method/code.

Maybe exposing original filename from instagram, would be better? I see it in the .json metadata, as part of the URL.

I think the absence of numbers like likes and comments when the abilities like dumping all comments and videos and stories exist is definitely a missing feature.

#EnhancementRequest

I have not figured yet how to prevent the text files from being saved along the images. Maybe we could have a "--no-text-file" function as well?

Thanks

I started with the new version 2.2.2 and after dumping a uarter of the profile it returned with this exception:

$instaloader 
.
.
.
.
[AFTER DOWNLOADING FOR A WHILE THIS ERROR CAME UP]
.
Request returned HTTP error code 404.
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/instaloader-2.2.2-py3.5.egg/instaloader.py", line 172, in _get_and_write_raw
    except (urllib3.exceptions.HTTPError, requests.exceptions.RequestException, ConnectionException) as err:
instaloader.ConnectionException: Request returned HTTP error code 404.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/instaloader-2.2.2-py3.5.egg/instaloader.py", line 172, in _get_and_write_raw
    except (urllib3.exceptions.HTTPError, requests.exceptions.RequestException, ConnectionException) as err:
instaloader.ConnectionException: Request returned HTTP error code 404.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/instaloader-2.2.2-py3.5.egg/instaloader.py", line 172, in _get_and_write_raw
    except (urllib3.exceptions.HTTPError, requests.exceptions.RequestException, ConnectionException) as err:
instaloader.ConnectionException: Request returned HTTP error code 404.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/instaloader-2.2.2-py3.5.egg/instaloader.py", line 1034, in download
    if username is not None:
  File "/usr/local/lib/python3.5/dist-packages/instaloader-2.2.2-py3.5.egg/instaloader.py", line 685, in download_node
    self._sleep()
  File "/usr/local/lib/python3.5/dist-packages/instaloader-2.2.2-py3.5.egg/instaloader.py", line 417, in download_pic
    except UnicodeEncodeError:
  File "/usr/local/lib/python3.5/dist-packages/instaloader-2.2.2-py3.5.egg/instaloader.py", line 178, in _get_and_write_raw
    
  File "/usr/local/lib/python3.5/dist-packages/instaloader-2.2.2-py3.5.egg/instaloader.py", line 178, in _get_and_write_raw
    
  File "/usr/local/lib/python3.5/dist-packages/instaloader-2.2.2-py3.5.egg/instaloader.py", line 176, in _get_and_write_raw
    self._sleep()
instaloader.NodeUnavailableException

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/instaloader", line 9, in <module>
    load_entry_point('instaloader==2.2.2', 'console_scripts', 'instaloader')()
  File "/usr/local/lib/python3.5/dist-packages/instaloader-2.2.2-py3.5.egg/instaloader.py", line 1258, in main
  File "/usr/local/lib/python3.5/dist-packages/instaloader-2.2.2-py3.5.egg/instaloader.py", line 1131, in download_profiles
    help='Rather than downloading regular posts of each specified profile, only download '
  File "/usr/local/lib/python3.5/dist-packages/instaloader-2.2.2-py3.5.egg/instaloader.py", line 1037, in download
    targets.update([followee['username'] for followee in followees])
KeyError: 'shortcode'

When instaloader is invoked to download a set of private profiles, it asks for username and password for each profile it attempts to download. It would be a better behavior to ask only once and then reuse my login credentials.

Profile X: no pics found. on public accounts.

Have an command line option to download all followees of a profile.

Download comments.

Maybe as separate XML files, together with other metadata such as the image caption.

When the search by hashtag retrieves a high number of results (> 1000 for example), not all posts are downloaded (and no error in printed).

I ran the following command.

$ instaloader "#boxingmeme"

and expected to download 5,218 posts as the original #boxingmeme feed on Instagram has 5,218 Posts (or atleast that's what IG is claiming it has).

However, the download exited without any error at post # 336.

See console output below.

...
........
[329] #boxingmeme 2016-07-05_16-00-40 [Who else does this ? Umm lemm▒]
[330] #boxingmeme 2016-06-19_06-54-01 [#happy #fathersday #happyfath▒]
[331] #boxingmeme 2016-06-03_10-37-07 txt mp4
[332] #boxingmeme 2016-05-18_03-40-46 [Ok one last one .. And I'll f▒]
[333] #boxingmeme 2016-05-17_17-31-16 [Since we're at it...The way F▒]
[334] #boxingmeme 2016-05-07_21-50-50 ["Views From The 6th... Round"▒]
[335] #boxingmeme 2016-05-03_08-29-04 txt mp4
[336] #boxingmeme 2016-04-25_09-49-34 [Would you rather fight in Oak▒]

That last post downloaded, # 336 has a datetime posted of almost exactly a year ago ( Apr 25th 2016 to be precise).

Wonder if the graph querying restricts us to only a sub-set of the data.

Thoughts?

NOTE: I didn't see any error in the cosole. Not sure what the http request status returned is....

when instaloading a profile if internet is disconnected the script keeps sending request and showing errors but keeps going to the next file instead of waiting to finalize the file (until the internet is connected again) and then moving to the next one.
i think lack of this feature can cause inconsistencies when using instaloader where data consistency and validity is important.

Hello there
I am going to download just .txt not pictures /videos I use:
instaloader --login YOUR-USERNAME --password YOUR-PASSWORD -s PROFILE
but this command download pictures too.
I add --stories-only instead of --stories it's not work like --stories when I add.
I need to download txt and skip to download pictures /videos
Thanks

It is 644. It should be 600.

Hi

Instagram compresses images on upload in terms of size and JPEG. I use the service since 2011, and compression rate for my early pictures is too high, some downloaded images have low quality. Since I have a local archive of all saved photos from my iPhones, I wrote a script to find better versions of fetched images here. The script is available here: https://gist.github.com/pavelkryukov/15f93d19a99428a284a8bcec27e0187b

I followed this example: https://github.com/realpython/image-fingerprinting. They use imagehash to detect similar images and shelve to store a database (so we do not have to iterate everything twice). Sometimes hashes between the original and the uploaded version are sligtly different, so we have to iterate through the whole database to find the best match :-(.

I would like to know if you see an opportunity to merge this functionality into InstaLoader. There is no problem to save a hash for each image to a .txt file during download, but looking over the database might have several options (how to generate a database and where to keep it for the next update? what if files were moved to other location?)

Thanks,
Pavel

There should be an option --feed=<limit> which downloads up to limit pictures from the user's feed, if --login is given.

I try to get followers from a random instagram profile but the program always gives this error:
Fatal error: ConnectionError(400): unable to gather followees.

I used the following command:
python instaloader.py --login=MY_USERNAME @TARGET_USERNAME

Hi,

Background: I've found online a list of popular hashtags to use with #starwars, such as #disney or #toys but then I would not want to download just #disney or just to download #toys... and just #starwars by itself is like 100,000 images a day, which is almost impossible to go through unless I look at posts through a different eye. I do not want to look at images of Mickey Mouse all day, unless it is a stormtrooper with Mickey Mouse ears :)

I don't know if the Instagram service used in the script would allow the implementation of a multiple hashtags "AND" function such as

"Download images from posts that have both #y AND #x", but exclude posts that have only #y or only #x?

The default folder name could be #y-#x to match the parameters passed to the function.

A work around could be to query the posts one by one that have the first hashtag in the list (#y), but BEFORE downloading the image, look into the post to see if all the other #hashtags are there (#x), and only download/save those that match all the hashtags (#y AND #x).

Just a thought, maybe there is a better way to get to it.

Thanks