some samples fail with --force_sratools_download due to changes in prefetch results about fetchngs HOT 13 CLOSED

Okay this is the same thing I get connected to this issue. What I did to get around this is locally changing the bin/sra_ids_to_runinfo.py file so that the ENA FTP is preferred to SRA which can simply be done by changing line 229 from return cls._id_to_srx(identifier) to return cls._id_to_erx(identifier).
However, be aware that this only works for SRR identifiers and not for GSM.

from fetchngs.

It works now. Thanks a lot!

from fetchngs.

Hi @dmalzl ! Thanks for looking into a fix. Yes, the NCBI changed their APIs yet again with a breaking change. Given this pipelines and other tools make assumptions about the API calls unfortunately the only thing we can do is to patch fix on the fly...

Have you by any chance found a backwards compatible fix? If so, we can do a patch release straight away.

We have plans to use and contribute to ffq to fetch the metadata in the future but that won't be immune to these sorts of changes either. It will however standardise the way we fetch the metadata.

from fetchngs.

Hi. I confirm that the fetchngs is not useable as it is now for SRA ids. Neither version 1.6 or 1.4.
In my case all my inputs are in SRA format so I can't use the pipeline unless I convert them I guess.
Any idea of the timescale when this bug will be fixed?

from fetchngs.

so for me most of them still work just some are not available. did you try use teh --force_sratools_download option

from fetchngs.

Hi. Yes it doesn't work. I double checked and in fact all my IDs are Geo Sample IDs (GSM).

from fetchngs.

Although I think fetchngs is designed to also link these back SRA accessions I would suggest using the SRA run selector or the Entrez API to link GSM to SRA accessions maybe that solves the problem

from fetchngs.

Thanks for the suggestion. I managed to convert the IDs using the Entrez API as you suggested and run the command again with or without the --force_sratools_download option, but I get the same error message as before.

from fetchngs.

Could you post the exact error you get and for which process this occurs?

from fetchngs.

Yes, here is the error:

Error executing process > 'NFCORE_FETCHNGS:SRA:SRA_IDS_TO_RUNINFO (SRR5000684)'

Caused by:
  Process `NFCORE_FETCHNGS:SRA:SRA_IDS_TO_RUNINFO (SRR5000684)` terminated with an error exit status (1)

Command executed:

  echo SRR5000684 > id.txt
  sra_ids_to_runinfo.py \
      id.txt \
      SRR5000684.runinfo.tsv \


  cat <<-END_VERSIONS > versions.yml
  "NFCORE_FETCHNGS:SRA:SRA_IDS_TO_RUNINFO":
      python: $(python --version | sed 's/Python //g')
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  [ERROR] The server couldn't fulfill the request.
  [ERROR] Status: 400 Bad Request. Both list of IDs and query_key are empty

from fetchngs.

Hi @drpatelh,

Unfortunately, the only things I could come up with is using the ENA FTP by default and falling back on the FTP when prefetch failed. So what I did was first ignoring all fails of prefetch due to it only downloading the *.sralite file and then downloading the ones that failed via the ENA FTP which was a very quick fix for me but I already thought about trying to feed the *.sralite file to fasterq-dump to see what happens. If it shows the expected behaviour one might be able to just check for either of the outcomes of prefetch and act accordingly but I haven't tried yet.

Such API changes are so annoying and I know this is out of your hands. Just wanted to point it out so that other users know what's going on.

Thanks for looking into it though

from fetchngs.

return cls._id_to_erx(identifier)

This also worked for me.

from fetchngs.

This issue should mostly be solved I think after the API was fixed. Feel free to re-open if the problem persists.

from fetchngs.