Hi Ying I am trying to run IBDkin in a data set of 85K individuals (2.8M SNPs). Th

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

Segmentation fault even for small file and huge amount of memory about ibdkin HOT 8 CLOSED

vicbp1 commented on August 28, 2024

Segmentation fault even for small file and huge amount of memory

from ibdkin.

Comments (8)

YingZhou001 commented on August 28, 2024 1

Thanks Bing, that is a super catch! Welcome to modify my scripts in any ways! Best, Ying

…

On Tue, Oct 19, 2021 at 5:14 PM Bing Guo ***@***.***> wrote: Hi @YingZhou001 <https://github.com/YingZhou001>, It has been a while since we met via Zoom in the MCMC workshop this summer :). Victor and I are from the same group. We looked at this issue together this afternoon. We figured that the issue was caused by the memory overflow from str1 to chr. https://github.com/YingZhou001/IBDkin/blob/61ccecbf3cae8b5dd4b42716aeed099e35a3c3ed/src-v2.8.7.7/parallel.c#L623-L625 https://github.com/YingZhou001/IBDkin/blob/61ccecbf3cae8b5dd4b42716aeed099e35a3c3ed/src-v2.8.7.7/parallel.c#L634-L636 It would not be an issue if the sample names are short. But we happened to have long sample names in our IBD input files (as long as 58 bytes, longer than buffer size (BUFF1 or 32) . https://github.com/YingZhou001/IBDkin/blob/61ccecbf3cae8b5dd4b42716aeed099e35a3c3ed/src-v2.8.7.7/head.h#L17 After we edited #define BUFF1 32 to #define BUFF1 128 and recompiled the code, it runs without crashing immediately. Hope it runs smoothly to the end. @vicbp1 <https://github.com/vicbp1> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHO6E2FPKM4VDCOTJ5S54CLUHYCWDANCNFSM5F6PIAYQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

from ibdkin.

YingZhou001 commented on August 28, 2024

Hi Victor,

Thanks for reaching.
Based on your situation, I will try to add option "--part 10 1" to your command. If the problem is still there, I will check the input files.

Best,
Ying

from ibdkin.

vicbp1 commented on August 28, 2024

Thank for your answer!
I double checked the inputs and they look ok. For values like --part 10, 20, or 50 the result is the same even when I increase the memory availability (700G). Just for one chr that I had a different error : Error!, maker ranges for chromosome do not cover the position 21138926

Even when my data have that region

Any thoughts?

from ibdkin.

YingZhou001 commented on August 28, 2024

Hi Victor, I don't think this is a memory problem.
Did you check the range file? it should cover all your markers.
And also check the chromosome identifier, it should be in the same codes for all input files.
If the range file is good, I will cehck 1) ibd file, compare the format with the example to look difference. 2) for each IBD segment, the end position should be larger than the start position. 3) the genetic map, both the physical and genetic positions should be in increasing order.

Best,
Ying

from ibdkin.

vicbp1 commented on August 28, 2024

Hi Ying,

I found that there was something wrong during the compression of the files and now is working
Thank you so much for your time!

from ibdkin.

vicbp1 commented on August 28, 2024

Hi Ying,

After checking all my files and running again I have found this error:

Error!, maker ranges for chromosome do not cover the position 4160868

What it means exactly? Do I have to modify my range sfile?

from ibdkin.

YingZhou001 commented on August 28, 2024

Hi Victor, Yes, please try to modify the range file. The range file is used for allocate memory so that it can save time for ibdkin for reading ibd file. Best, Ying

…

On Mon, Oct 18, 2021 at 12:57 PM Victor Borda ***@***.***> wrote: Hi Ying, After checking all my files and running again I have found this error: *Error!, maker ranges for chromosome do not cover the position 4160868* What it means exactly? Do I have to modify my range sfile? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHO6E2ENQOMKYDTYAHNQEA3UHR34LANCNFSM5F6PIAYQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

from ibdkin.

bguo068 commented on August 28, 2024

Hi @YingZhou001,

It has been a while since we met via Zoom in the MCMC workshop this summer :).
Victor and I are from the same group. We looked at this issue together this afternoon. We figured that the issue was caused by the memory overflow from str1 or str2 to chr.

IBDkin/src-v2.8.7.7/parallel.c

Lines 623 to 625 in 61ccecb

 char str1[BUFF1], str2[BUFF1]; 

 int tmpid; 

 int chr, id1, id2;

IBDkin/src-v2.8.7.7/parallel.c

Lines 634 to 636 in 61ccecb

 for(i = 0; i < buffi; i++){ 

 if(i % N != th_index)continue; 

 sscanf(strbuff[i], "%s %*s %s %*s %*s %f %f %f", str1, str2, &p1, &p2, &l);

It would not be an issue if the sample names are short. But we happened to have long sample names in our IBD input files (as long as 58 bytes, longer than buffer size (BUFF1 or 32) .

IBDkin/src-v2.8.7.7/head.h

Lines 16 to 17 in 61ccecb

 #define BUFF 128 

 #define BUFF1 32

After we edited #define BUFF1 32 to #define BUFF1 128, #define BUFF 128 to #define BUFF 512 and recompiled the code, it runs without crashing immediately. Hope it runs smoothly to the end. @vicbp1

from ibdkin.

Segmentation fault even for small file and huge amount of memory about ibdkin HOT 8 CLOSED

Comments (8)

Related Issues (6)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	for(i = 0; i < buffi; i++){
	if(i % N != th_index)continue;
	sscanf(strbuff[i], "%s %s %s %s %*s %f %f %f", str1, str2, &p1, &p2, &l);