Out of memory error about jscpd HOT 20 CLOSED

Thanks for pointing this out! I just figured out I installed leveldb locally while jscpd was installed globally, after I install them both globally, it ran successfully!

Thanks again for the help! Appreciate

from jscpd.

thank you for report, will investigate and fix

from jscpd.

I encountered this issues today and solved it by increasing the amount of memory node.js uses.

node --max-old-space-size=8192 node_modules/.bin/jscpd

It might be a nice idea to add it to the README file.

from jscpd.

Just released [email protected] with persistence db inside, for now, you can run jscpd on really big projects

from jscpd.

facing this issue while running jscpd scan for big repo in jenkins ,Could you just tell me where the changes needs to be made exactly ?

from jscpd.

@nishwalshetty could you please provide more information about jscpd version and node.js env?

from jscpd.

@kucherenko jscpd version 2.0.16 and it is running in linux environment with node 8.15.0

from jscpd.

<--- Last few GCs --->

[17914:0x307ba90] 3918376 ms: Mark-sweep 1369.0 (1463.0) -> 1368.6 (1471.0) MB, 350.7 / 0.1 ms allocation failure GC in old space requested
[17914:0x307ba90] 3918678 ms: Mark-sweep 1368.6 (1471.0) -> 1368.6 (1433.0) MB, 301.4 / 0.4 ms last resort GC in old space requested
[17914:0x307ba90] 3918992 ms: Mark-sweep 1368.6 (1433.0) -> 1368.6 (1432.0) MB, 313.7 / 0.1 ms last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x3258c24258b9
0: builtin exit frame: entries(this=0x3258c2404831 <JSFunction Object (sfi = 0x3bf429509cf9)>,0x5b4b37d9f01 )

1: updatePercentage [/opt/node-v8.15.0-linux-x64/lib/node_modules/jscpd/build/listeners/statistic.js:~86] [pc=0x232ba937d3c8](this=0x112f14604f81 <StatisticListener map = 0x2b58dd110c19>,format=0x29581a87b859 <String[10]: javascript>)
2: matchSou...

from jscpd.

Hi @kucherenko , above is the error stack we are facing when we(@nishwalshetty) did the scan on our projects.

from jscpd.

Hi @himanshu2416, thank you for the report, I will investigate, looks like I should reopen the issue.

from jscpd.

@kucherenko did you find any workaround

Hi @kucherenko , above is the error stack we are facing when we(@nishwalshetty) did the scan on our projects.

@kucherenko did you find any workaround for this ?

from jscpd.

try to upgrade version of node.js to latest

from jscpd.

also you can try to use command:

node --max-old-space-size=8192 node_modules/.bin/jscpd

from jscpd.

@kucherenko no luck with latest node version 10.17.0 also the above command was also tried. please find the stacktrace below

jscpd --min-tokens 200 --reporters html --output . --mode strict --format javascript,java --ignoreCase true . node --max-old-space-size=8192 node_modules/.bin/jscpd

<--- Last few GCs --->

[5438:0x39c01e0] 12884970 ms: Mark-sweep 1393.3 (1445.2) -> 1393.3 (1414.2) MB, 138.9 / 0.0 ms (average mu = 0.143, current mu = 0.000) last resort GC in old space requested
[5438:0x39c01e0] 12885129 ms: Mark-sweep 1393.3 (1414.2) -> 1393.3 (1414.2) MB, 159.3 / 0.0 ms (average mu = 0.080, current mu = 0.000) last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

0: ExitFrame [pc: 0x187f5c87e441]

Security context: 0x2f870819e6e9
1: _put [0xcd0240ef389] [/usr/lib/node_modules/jscpd/node_modules/leveldown/leveldown.js:~52] [pc=0x187f5f528491](this=0x031ff1c3bdf1 ,key=0x34ad72916221 <String[24]: 2019-11-13T11:30:50.721Z>,value=0x33582d33c809 <Very long string[9838981]>,options=0x33582d33ca49 ,callback=0x33582d33ca81 <JSFu...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: 0x8fa050 node::Abort() [node]
2: 0x8fa09c [node]
3: 0xb0020e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xb00444 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0xef4952 [node]
6: 0xf0416f v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
7: 0xecc3b5 [node]
8: 0xed3c2a v8::internal::Factory::NewRawOneByteString(int, v8::internal::PretenureFlag) [node]
9: 0x101d69d v8::internal::String::SlowFlatten(v8::internal::Handlev8::internal::ConsString, v8::internal::PretenureFlag) [node]
10: 0xafdbb4 v8::internal::String::Flatten(v8::internal::Handlev8::internal::String, v8::internal::PretenureFlag) [node]
11: 0xb0b580 v8::String::Utf8Length() const [node]
12: 0x90cea8 napi_get_value_string_utf8 [node]
13: 0x7fc2884ca666 [/usr/lib/node_modules/jscpd/node_modules/leveldown/prebuilds/linux-x64/node.napi.node]
14: 0x7fc2884cc1bd db_put(napi_env__, napi_callback_info__) [/usr/lib/node_modules/jscpd/node_modules/leveldown/prebuilds/linux-x64/node.napi.node]
15: 0x905d15 [node]
16: 0x187f5c87e441

from jscpd.

jscpd --min-tokens 200 --reporters html --output . --mode strict --format javascript,java --ignoreCase true . node --max-old-space-size=8192 node_modules/.bin/jscpd

Is not right command, the following is right:

 node --max-old-space-size=8192 node_modules/.bin/jscpd --min-tokens 200 --reporters html --output ./report/ --mode strict --format javascript,java --ignoreCase .

from jscpd.

fixed at [email protected]

from jscpd.

@kucherenko Hi, I just ran into this issue with jscpd: 3.3.0-rc.10, node: v14.5
And my Mac has 64 GB memory and I tried node --max-old-space-size=12288 node_modules/.bin/jscpd, but didn't help.

I googled around and found this on StackOverflow, saying this might happen because of one single oversized object, like Map or Array
https://stackoverflow.com/questions/55465821/getting-heap-out-of-memory-error-even-when-available-heap-memory-is-much-larger

It would be greatly appreciated if you could reopen this issue and look into it.
Thanks for the great work you put into maintaining this project! !

Cheers

<--- Last few GCs --->

[80419:0x108008000] 303959 ms: Mark-sweep 4069.1 (4121.4) -> 4053.5 (4121.9) MB, 6384.0 / 0.0 ms (average mu = 0.102, current mu = 0.001) allocation failure scavenge might not succeed
[80419:0x108008000] 311113 ms: Mark-sweep 4069.7 (4121.9) -> 4054.8 (4123.6) MB, 7077.7 / 1.5 ms (average mu = 0.053, current mu = 0.011) allocation failure scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0x100bfea43 node::Abort() (.cold.1) [/usr/local/bin/node]
2: 0x10008609d node::FatalError(char const*, char const*) [/usr/local/bin/node]
3: 0x100086206 node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
4: 0x10018cc95 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
5: 0x10018cc3f v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
6: 0x1002b2ff1 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
7: 0x1002b4354 v8::internal::Heap::MarkCompactPrologue() [/usr/local/bin/node]
8: 0x1002b1d3b v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
9: 0x1002b04e4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
10: 0x1002b8654 v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
11: 0x1002b86aa v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
12: 0x1002985a3 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/local/bin/node]
13: 0x10050a25f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
14: 0x10077d659 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/usr/local/bin/node]
15: 0x1007ff6ab Builtins_CreateObjectLiteralHandler [/usr/local/bin/node]
[1] 80419 abort jscpd --min-lines 40 -r json,html -o “/Users/jenkins_node/Sites” -b

from jscpd.

I suggest install @jscpd/leveldb-store

npm i @jscpd/leveldb-store

and run following command

jscpd --store leveldb /path/to/files

from jscpd.

I suggest install @jscpd/leveldb-store

npm i @jscpd/leveldb-store

and run following command

jscpd --store leveldb /path/to/files

@kucherenko Hi, thanks for the quick response!

I see a warning store name leveldb not installed. and end up with the memory issue again, but I already ran npm install @jscpd/leveldb-store --save and also I can see leveldb is intalled when I run npm list. Did I miss something?

└─┬ @jscpd/[email protected]
  ├─┬ @jscpd/[email protected]
  │ └── [email protected]
  ├─┬ @jscpd/[email protected]
  │ ├── @jscpd/[email protected] deduped
  │ ├── [email protected]
  │ └── [email protected]
  ├─┬ [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├─┬ [email protected]
  │ │ ├── [email protected] deduped
  │ │ └── [email protected] deduped
  │ └── [email protected]

from jscpd.

Could you please tell me the command what you run for detect duplications? Is it globally installed jscpd or locally?

from jscpd.