devRant/iMGSRC.RU/ImageShack/MeetMe Not Detecting Users That Exist about sherlock HOT 2 CLOSED

This appears to be a problem with the sites that redirect an attempt to view a non-existing username to the main site. If you try to go to https://devrant.com/users/dfoxxxxxxxxx (a user name that does not exist), then we get a redirect to the https://devrant.com/ root of the site. Yet, the "response_url" checking algorithm is only looking for the configured error URL being included in the response:

        elif error_type == "response_url":
            error = net_info.get("errorUrl")
            # Checks if the redirect url is the same as the one defined in data.json
            if not error in r.url:
                print_found(social_network, url, response_time, verbose)
                write_to_file(url, f)
                exists = "yes"
                amount = amount+1
            else:
                print_not_found(social_network, response_time, verbose)
                exists = "no"

Since "https://devrant.com/" is in "https://devrant.com/users/dfox", it is marked as an unknown username.

Pinterest (for example) has a response URL of "https://www.pinterest.com/?show_error=true". So this does not cause the problem, as there is no overlap between a good username and the error URL.

from sherlock.

To fix this problem requires a complete change to the "response_url" detection strategy...

Previously, there was a problem with sites that redirect an attempt to view a non-existing username to the main site. For example, if you try to go to https://devrant.com/users/dfoxxxxxxxxx (a user name that does not exist), then we get a redirect to the https://devrant.com/ root of the site. But, the "response_url" checking algorithm was only looking for the configured error URL being included in the response. So, these sites always indicated that the username was not found.

Update the "response_url" detection method so that the request does not allow redirects. If we get a 200 response of some type, then the username has been found. However, if we get something like a 302, then we know that the username was not found as we are being redirected.

This whole method seems fragile, but I did exhaustively test all of the supported sites, and they all work. So, this change is clearly an improvement.

from sherlock.