`Response is missing required header Content-Length: for url ` about kent HOT 11 OPEN

This is the relevant error message :

Error in seqinfo(ranges) : UCSC library operation failed
In addition: Warning messages:
1: In seqinfo(ranges) :
  Response is missing required header Content-Length: for url https://nottlab.dsi.ic.ac.uk/RMY_cancer//cancer/hg38/A549_H3K27ac_R1_tag.ucsc.bigWig

It seems hashFindValUpperCase couldn't find hashed key for Content-Length and failing because of it

So I went further to investigate whether the value of Content-Length was fetch and hashed:

I used printf to log values before call hashAdd and It seems there's some issue with it.
As it didn't show Content-Length but I'm not sure which component is the responsible for the issue (parsing[netUrlHeadExt] or fetching[lineFileAttach]).

I'm happy to collaborate but for that I need someone help who's more familiar with the nitty-gritty details of the codebase.

from kent.

Thanks! Yes, this confirms that this is your call: struct bbiFile * file = bigWigFileOpen((char *)CHAR(asChar(r_filename)));

And we didn't change bigWigFileOpen. We made a single change 4-5 years ago to bigWigFileOpen and that broke another package. The engineer who made the most recent change confirmed that he will never touch these function signatures. I guess only you can find out what exactly broke here. We definitely made a small change, but not to any of the function headers that you're calling.

from kent.

Not quite sure who the primary maintainer is currently, but tagging the developers with the most overall contributions for assistance:
@NullModel @galt @katerose @JimKent

from kent.

Good Evening Brian: I'm trying to follow what it is you are describing here. It looks like you are running
an R script that is using rtracklayer code. I am not aware of how the rtracklayer code uses the
kent source. I'm assuming it is somehow built into rtracklayer. I guess my question would be, which
version of the kent source is being used ? When I try the ordinary kent code command line tools
on the URL you display here, it all appears to work just fine. The error you describe appears to be
in the network connection business. From that same system you are working from, do the ordinary
kent command line tools work with the URL ? For example, bigWigInfo should return:

bigWigInfo "https://nottlab.dsi.ic.ac.uk/RMY_cancer//cancer/hg38/A549_H3K27ac_R1_tag.ucsc.bigWig" version: 4 isCompressed: yes isSwapped: 0 primaryDataSize: 38,394,145 primaryIndexSize: 226,392 zoomLevels: 10 chromCount: 198 basesCovered: 520,652,859 mean: 1.871263 min: 0.050000 max: 574.520020 std: 7.786024

from kent.

Thanks @NullModel for the pointers and for clarifying that this issue is not related to kent.
At some point, rtracklayer somewhat diverge from the kent source. I will look into the networking module in rtracklayer.

from kent.

Hi @sanchit-saini, wow, this is interesting. We had no idea that you're calling our C functions from R directly, here, for the bigWig data import: https://github.com/lawremi/rtracklayer/blob/master/R/bigWig.R#L240 (the only thing that I can't figure out is which function you're calling exactly, it's in the string/macro "bwgFile_query" but I can't find the value.

If we changed a function header and that broke something, let us know. We can give you a stable function call, e.g. bwQuery_stable, and can promise to never change the header of that. But we haven't ever touched the headers, as far as I can see from the git history, so maybe this has to do with something related to rtracklayer?

from kent.

The only change was

-struct bbiFile *bigWigFileOpenAlias(char *fileName, struct hash aliasHash);
-/ Open up big wig file. Free this up with bbiFileClose. Use aliasHash if non-NULL */
+struct bbiFile *bigWigFileOpenAlias(char fileName, aliasFunc aliasFunc);
+/ Open up big wig file. Free this up with bbiFileClose. Use aliasFunc if non-NULL */

in commit b622d14 but this is not a function you use (as far as I can tell)

from kent.

@braneyboo I think we've had this issue before, that whenever we change something in bigBed/bigWig, rtracklayer runs into trouble. I wonder if we can think about a system to reduce this in the future... not sure what. Maybe make separate, stable function calls that we never touch and that are marked as such in the code?

from kent.

Hi @maximilianh ,

the only thing that I can't figure out is which function you're calling exactly, it's in the string/macro "bwgFile_query" but I can't find the value.

Calling to this function which uses kent library functions
https://github.com/lawremi/rtracklayer/blob/05017acd2b4abc934b7e8c9873bdedcc099875aa/src/bigWig.c#L226

Kent source that rtracklayer relies on resides at :
https://github.com/lawremi/rtracklayer/tree/master/src/ucsc

If we changed a function header and that broke something, let us know. We can give you a stable function call, e.g. bwQuery_stable, and can promise to never change the header of that. But we haven't ever touched the headers, as far as I can see from the git history, so maybe this has to do with something related to rtracklayer?

I don't think it is related to the modification of function headers. I'm still not sure what's causing this issue and need some time to investigate it.

Apart from that, I like the idea of having stable functions. It would be great. We could do it at some time. First, we've to factor out the list of used functions.

from kent.

from kent.

@sanchit-saini have you had any luck with this?

from kent.