n0madic / google-play-scraper Goto Github PK

There is probably some modification in GPlay, it's impossible to fetch the app data list for a developer.

First of all, I want to thank you for developing the 'google-play-scraper' library in Go. It has been extremely useful. However, I've noticed that the crawler is unable to retrieve all the permissions of an application from Google Play.

For example:

https://play.google.com/store/apps/details?id=com.einnovation.temu&hl=en

jsonArray:

[[["Microphone",[null,2,null,[null,null,"https://play-lh.googleusercontent.com/daUjqbSOr2QpaqXS2HQbNzYzzqN2yWGzM_7AZxwFaWLT7_kIhX95HKi_HSpjeeQDOmFMENZxJqblbu_4qg"]],[[null,"record audio"]],["perm_camera_mic"]],["Photos/Media/Files",[null,2,null,[null,null,"https://play-lh.googleusercontent.com/pHtIujPWxciAZcfYSwlrGGq14Z984rKLMgcm9RPATLiOlbrWy-tVlelEWgED7gpktgcD1tZizVeHiO5fkw"]],[[null,"read the contents of your USB storage"],[null,"modify or delete the contents of your USB storage"]],["perm_media"]],["Wi-Fi connection information",[null,2,null,[null,null,"https://play-lh.googleusercontent.com/U-_SG8pHTsqU_IyZTGQRkVMdLaAUeq1OnKGrB06KHF1z7vkkIQK3iF0HcbfTe1RnGlh-ajnZkbphl2W3Gdk"]],[[null,"view Wi-Fi connections"]],["signal_wifi_4_bar"]],["Storage",[null,2,null,[null,null,"https://play-lh.googleusercontent.com/aWNKQedLTpw6u6yyMjQObmuoKu67A1czWnIcvID86oAmMT02r5mNdRn6l9ZN2t2MIyH6tNy-01v7ukeQ"]],[[null,"read the contents of your USB storage"],[null,"modify or delete the contents of your USB storage"]],["storage"]],["Camera",[null,2,null,[null,null,"https://play-lh.googleusercontent.com/xbP_oGuJ21iG29iVh0p-UIZPzi_fYj8PMYiqDd9-LvaZ_a1tRcwp0I2-arfXvgX9YtfZTTaqwcLRWPNQM_c"]],[[null,"take pictures and videos"]],["camera"]]],[["Other",[null,2,null,[null,null,"https://play-lh.googleusercontent.com/pkKXoPl5q7n8T0s7KREtdvUZn1PLRgx-Ox0t4tkO8af4JpgGbyAxLBTsvEKKBCjwBACQsZisSYNmHPGbBA"]],[[null,"run at startup"],[null,"full network access"],[null,"prevent device from sleeping"],[null,"view network connections"],[null,"control vibration"]],["quiz"]]],[[null,"receive data from Internet"]]]

But it cannot extract the "Others" segment:

I've tried something like this:

func (app *App) LoadPermissions() error {
	payload := strings.Replace("f.req=%5B%5B%5B%22xdSrCf%22%2C%22%5B%5Bnull%2C%5B%5C%22{{appID}}%5C%22%2C7%5D%2C%5B%5D%5D%5D%22%2Cnull%2C%221%22%5D%5D%5D", "{{appID}}", app.ID, 1)

	js, err := util.BatchExecute(app.options.Country, app.options.Language, payload)
	if err != nil {
		return err
	}

	app.Permissions = make(map[string][]string)


	for _, path := range []string{"0", "1"} {

		for _, perm := range util.GetJSONArray(js, path) {
			key := util.GetJSONValue(perm.String(), "0")
			for _, permission := range util.GetJSONArray(perm.String(), "2") {
				app.Permissions[key] = append(app.Permissions[key], util.GetJSONValue(permission.String(), "1"))
			}
		}
	}


	return nil
}

And this code can extract all the permissions from "Others" except the first one, "receive data from Internet".

You can make use of this test to check this behviour.

Use here:

for _, app := range q.Results {
  fmt.Println(app.Title)
}

It will only print out 10 app names. The other app names (Title) are empty.
What is weird, is that app.URL are available for each app.

Hi,
Great library and thank you.
I was looking for a Google Scraper and found one.

Is there any way to extract the overall number and the average as well?
In other words, (from the screenshot below), it'd be nice to extract the 4.3 overall ratings and the distribution numbers, for example for rating 3 the following screenshot presents on hover with 1,137 reviews.

I did try and get all the reviews below

	r := reviews.New("xxx.application.ignore.it", reviews.Options{
		Number:   1000000,
		Language: "ja",
	})

	err := r.Run()
	if err != nil {
		panic(err)
	}

	for _, review := range r.Results {
		// fmt.Println(review.Reviewer, review.Timestamp, review.Score, review.Text)
		fmt.Println(review.Score)
	}

╰─$ go run main.go |grep 3|wc -l
     633

This one doesn't give exact number which is ok since it doesn't extract 100% of the reviews.

Current Get category example in the readme file is failing with a fatal error...

I would like to know if it will be remapped to a new interface or if there is a solution to this problem

{ "AdSupported": false, "AndroidVersion": "[[null,[[1800],null,[72000]]]]", "AndroidVersionMin": 0, "ContentRating": "", "ContentRatingDescription": "", "Description": "", "DescriptionHTML": "", "Developer": "", "DeveloperAddress": "", "DeveloperEmail": "", "DeveloperID": "", "DeveloperURL": "https://play.google.com", "DeveloperWebsite": "", "FamilyGenre": "", "FamilyGenreID": "", "Free": true, "Genre": "", "GenreID": "", "HeaderImage": "", "IAPOffers": false, "IAPRange": "", "Icon": "", "ID": "com.whatsapp",

app := New("com.kgi", Options{"tw", "zh-TW"})
err := app.LoadDetails()
if err != nil {
t.Error(err)
}

can't get all detail information

First of all, thanks for this amazing library.

Can you set the fpr header for searches to false by default?

I'm not exactly sure why it's used but adding params.Add("fpr", "false") to pkg/search/search.go gives better matches.

I believe it's used by Google for autocorrection (like "whatsa" to "whatsapp" or "yout" to "youtube"), and when you select the "No search for yout instead", it adds this header as &fpr=false.

I don't know why they use this though. The search results don't change much with or without it.

Please update the struct to have a new property (e.g. DeveloperInternalId), to have the internal developer Id,
It's useful because It remains the same even If the developer changes their name.

Here's the mapping 1.2.68.2

@n0madic Hi,

I'd created another issue and you'd closed it but there's still a fatal crash error as I mentioned in my comment.

Here's the example code with fatal crash:

clusters, err := category.New(store.ArtAndDesign, store.AgeNineUp, category.Options{ Country: "us", Language: "us", Number: 100, })

I was testing the code about collecting of new reviews with following configuration:

r := reviews.New("com.kolayrandevu.isletme", reviews.Options{
Number: 100,
Sorting: store.SortNewest,
Language: "zh",
})

When I run this, the code first collects the initial reviews successfully for 2 existing reviews on Google Play. Later on, the following code in reviews.go creates an infinite loop because the number property I've provided is 100 but the available reviews on Google Play is only 2.

for len(reviews.Results) != reviews.options.Number {

It is no longer possible to get the list data through category, search and other api, do you have any plans to re-implement them?