Git Product home page Git Product logo

Comments (12)

giosh94mhz avatar giosh94mhz commented on August 16, 2024

Thanks for the report. I've fixed it.
Please note that I'm still in the early stages of this bundle and many thing will change. I think that in the near future I will migrate to the Guzzle library for downloading.

from geonamesbundle.

mekras avatar mekras commented on August 16, 2024

It seems that problem still not fixed. Now downloadsSize is always equal to zero if curl_getinfo returns -1. So nothing will be downloaded at all.

from geonamesbundle.

giosh94mhz avatar giosh94mhz commented on August 16, 2024

@mekras I'm not able to reproduce the issue, both with my unit testing and with "dump files" download.

The call to CurlDownload::requestContentLength is just for displaying progress, so it should not prevent download. The curl_info which trigger the error is bound to an HEAD request to the Geonames dump files folder. Just to exclude some connection/configuration error, try this in a shell on your side:

curl -I http://download.geonames.org/export/dump/allCountries.zip

from geonamesbundle.

mekras avatar mekras commented on August 16, 2024 
mekras@mekras: curl -I http://download.geonames.org/export/dump/allCountries.zip 
HTTP/1.1 200 OK    
Date: Fri, 07 Feb 2014 11:57:13 GMT
Server: Apache/2.2.17 (Linux/SUSE)
Last-Modified: Fri, 07 Feb 2014 05:12:18 GMT
ETag: "33c18b2-ebb7b82-4f1ca08abb080"
Accept-Ranges: bytes
Content-Length: 247167874
Content-Type: application/zip                 
X-Pad: avoid browser bug

But if I insert code

print_r(curl_info($ch)); die;

into requestContentLength method then I got:

Array
(
    [url] => http://download.geonames.org/export/dump/featureCodes_en.txt
    [content_type] => 
    [http_code] => 0
    [header_size] => 0
    [request_size] => 0
    [filetime] => -1
    [ssl_verify_result] => 0
    [redirect_count] => 0
    [total_time] => 0.000123
    [namelookup_time] => 0
    [connect_time] => 0
    [pretransfer_time] => 0
    [size_upload] => 0
    [size_download] => 0
    [speed_download] => 0
    [speed_upload] => 0
    [download_content_length] => -1
    [upload_content_length] => -1
    [starttransfer_time] => 0
    [redirect_time] => 0
    [certinfo] => Array
        (
        )

    [primary_ip] => 
    [primary_port] => 0
    [local_ip] => 
    [local_port] => 0
    [redirect_url] => 
)

Backtrace:

#0 vendor/giosh94mhz/geonames-bundle/Giosh94mhz/GeonamesBundle/Utils/CurlDownload.php(132): Giosh94mhz\GeonamesBundle\Utils\CurlDownload->requestContentLength()
#1 vendor/giosh94mhz/geonames-bundle/Giosh94mhz/GeonamesBundle/Import/ImportDirector.php(148): Giosh94mhz\GeonamesBundle\Utils\CurlDownload->download()
#2 vendor/giosh94mhz/geonames-bundle/Giosh94mhz/GeonamesBundle/Import/ImportDirector.php(59): Giosh94mhz\GeonamesBundle\Import\ImportDirector->download()
#3 vendor/giosh94mhz/geonames-bundle/Giosh94mhz/GeonamesBundle/Command/AbstractImportCommand.php(50): Giosh94mhz\GeonamesBundle\Import\ImportDirector->import()
#4 vendor/symfony/symfony/src/Symfony/Component/Console/Command/Command.php(241): Giosh94mhz\GeonamesBundle\Command\AbstractImportCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#5 vendor/symfony/symfony/src/Symfony/Component/Console/Application.php(888): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#6 vendor/symfony/symfony/src/Symfony/Component/Console/Application.php(191): Symfony\Component\Console\Application->doRunCommand(Object(Giosh94mhz\GeonamesBundle\Command\ImportCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#7 vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/Console/Application.php(96): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#8 vendor/symfony/symfony/src/Symfony/Component/Console/Application.php(121): Symfony\Bundle\FrameworkBundle\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#9 app/console(27): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput))

From console:

mekras@mekras: curl -I http://download.geonames.org/export/dump/featureCodes_en.txt
HTTP/1.1 200 OK
Date: Fri, 07 Feb 2014 12:08:37 GMT
Server: Apache/2.2.17 (Linux/SUSE)
Last-Modified: Fri, 07 Feb 2014 04:54:29 GMT
ETag: "33c18ad-deba-4f1c9c8f40b40"
Accept-Ranges: bytes
Content-Length: 57018
Content-Type: text/plain; charset=utf-8

Some more info:

  • PHP: 5.4.20
  • cURL: 7.32.0-2.4.1

from geonamesbundle.

giosh94mhz avatar giosh94mhz commented on August 16, 2024

I have a less recent version of both PHP and cURL, so I don't think (hope :) it is a regression.

There are to weird things:

  1. namelookup_time and connect_time are 0, but in my tests are something meaningful;
  2. command line works perfectly.

So maybe it's just a PHP module problem. Just to be sure check that the configuration contain something like (reduced):

curl

cURL support => enabled
Largefile => Yes
libz => Yes
Protocols => dict, file, ftp, ftps, gopher, http, https, imap, imaps, ldap, pop3, pop3s, rtmp, rtsp, scp, sftp, smtp, smtps, telnet, tftp
ZLib Version => 1.2.7

Also, you can try to add a simple "return 1" instead of the body of requestContentLength. You'll get a weird progress bar, but the download should happens anyway. If that's the case maybe there is some problem with ''curl_copy_handle''

from geonamesbundle.

mekras avatar mekras commented on August 16, 2024

The same problem on FreeBSD with PHP 5.3.28 and cURL 7.35.0
Configuration is OK.

from geonamesbundle.

giosh94mhz avatar giosh94mhz commented on August 16, 2024

I reopen the issue since it is not fixed.

Still I'm clueless on where to start.

  1. Have you tried the return 1 trick from the function requestContentLength?
  2. The UnitTests of CurlDownload are affected of the issue?

In the meanwhile, I'm working on some code changes to use Guzzle as a download adapter. CurlDownload will still be there anyway, so I hope we get this fixed.

from geonamesbundle.

mekras avatar mekras commented on August 16, 2024
  1. Have you tried the return 1 trick from the function requestContentLength?

Not yet. I'm trying to determine the cause of the error.

  1. The UnitTests of CurlDownload are affected of the issue?

I'll check it on the next week.

from geonamesbundle.

mekras avatar mekras commented on August 16, 2024

Download tests failed:

There were 2 failures:

1) Giosh94mhz\GeonamesBundle\Tests\Utils\CurlDownloadTest::testDownload
Failed asserting that 0 is greater than 0.

GeonamesBundle/Tests/Utils/CurlDownloadTest.php:77

2) Giosh94mhz\GeonamesBundle\Tests\Utils\CurlDownloadTest::testMultiDownload
Failed asserting that 0 is greater than 0.

GeonamesBundle/Tests/Utils/CurlDownloadTest.php:145

from geonamesbundle.

giosh94mhz avatar giosh94mhz commented on August 16, 2024

I've just pushed some updates and refactoring.
Now you can switch to the Guzzle as a download provider, by setting:

giosh94mhz_geonames:
    download:
        adapter:    guzzle

For now the instance of Guzzle::Http::Client is not injected, but it defaults to cURL. So we can find if it is a cURL problem or actually a bug in CurlDownloaderAdapter (name refactored :)

from geonamesbundle.

mekras avatar mekras commented on August 16, 2024

Using Guzzle solves the problem. But it requires to set memory_limit at least 1G.

from geonamesbundle.

giosh94mhz avatar giosh94mhz commented on August 16, 2024

Mmm... if Guzzle solves it than there is definitely something wrong in CurlDownloadAdapter.
Still I cannot reproduce it so... are you able to track down the issue by comparing Guzzle cURL classes to mine? By doing a quick check I see that it doesn't use the CURLOPT_FILE option to set the destination path, but use php://temp...

For the memory_limit the problem is the cache which call response->getBody on the dump. I think this will need some patch on Guzzle (when using the save_to option + caching) so I'll check it out in the next days.

from geonamesbundle.

Related Issues (4)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.