optimizer "tracing dependencies" multiple times for same modules about r.js HOT 8 CLOSED

on my specific case I have 90+ modules that I want to combine into 10 files, the build process takes around 17min on a MBP quad-core running through Rhino (I'm listing 14 modules on each exclude). If I remove the "exclude" config each file builds in less than 20 seconds (total time 3min)... - I would expect the performance to be similar (or even better) with the excludes (since it could ignore the content of these files).

PS: For some reason the build isn't working for me on node for this project so I don't know how fast it would be. (didn't had time to track what is causing the bug)

from r.js.

It is a bit tricky since the exclude lists can be different across build layers. It would have to be the exact same list, otherwise, what needs to be tracked is the dependency list for each item in the exclude list, and each sub-dependency. It is within the realm of possibility to see how that may work, but it would take more machinery to make it happen.

Doing an "exact list" exclude reuse is easier, worth investigating more. It is not high priority at the moment, but I am willing to look at patches.

I would definitely like to see how it performs under Node, it should be noticeably faster. If you do get it to run with Node, it would be great to know the build time.

from r.js.

The "exact list" approach wouldn't help that much since main will have a different list than
other modules.

I was thinking on some sort of dictionary to store dependencies of each module, something like:

//at first pass get flat dependency list (this should be cached across modules)
//array contain dependencies found inside `define` and paths normalized to always
//point to same modules even if using a path alias or relative paths.
var flatDeps = {
  "foo/bar" : ["foo/bar/lorem", "core/ipsum"],
  "foo/bar/lorem" : [],
  "core/ipsum" : ["core/asd"],
  "core/asd" : [],
};

//second module would just add items that are not  on the list
//since "core/ipsum" was already traced before just add
//"dolor/amet" and it's dependencies
flatDeps["dolor"] = ["core/ipsum", "dolor/amet"];
flatDeps["dolor/amet"] = ["core/ipsum", "dolor/bar"];
flatDeps["dolor/bar"] = [];

doing that you don't need to check dependencies again, just need to check if all the items of
flatDeps were already traced before, if not, just add the new items to the list.

than to output module "dolor" as a separate file with "core/ipsum" listed on the excludes
you would check flatDeps["dolor"] and load all the flatDeps excluding "core/ipsum":

function getNestedDeps(moduleId, excludes){
  excludes = excludes || [];

  var deps, 
      checked = [], 
      pending;

  function notChecked(val){
    return checked.indexOf(val) === -1;
  }

  function notExcluded(val){
    return excludes.indexOf(val) === -1;
  }

  function isUnique(val, i, arr){
    return arr.indexOf(val, i + 1) === -1;
  }

  function getFlatDeps(moduleId){
    return flatDeps[moduleId].filter(notExcluded);
  }

  function pushDeps(id){
    checked.push(id);
    deps = deps.concat( getFlatDeps(id) );
  }

  //grab dependencies

  deps = getFlatDeps(moduleId);

  while(true){
    pending = deps.filter(notChecked);
    if(! pending.length){
      break;
    }
    pending.forEach(pushDeps);
  }

  return deps.filter(isUnique);
}


console.log( getNestedDeps("dolor", ["core/ipsum"]) ); // ["dolor/amet", "dolor/bar"]

I know it shouldn't be an easy change but it can probably improve optimizer performance a lot.

from r.js.

just managed to make it work on node. it takes 30s to run, 27s tracing the dependencies.. - incredibly faster

without excludes it takes 5s to trace dependencies and 18s to finish (since it needs to compress all the duplicate code as well)

I still think that it is worth to trace dependencies only once. Build would be considerably faster on both environments and node is hard to install on windows.

BTW my problem was related with issue #12

from r.js.

Great, thanks for the timing info, it helps create a baseline. Your suggestion of a simple map for direct dependencies for each module might be used will avoid multiple file IO and AST conversion work. I'll see about getting this in, and then if you can try it with the new code, we can get a rough idea on the speed improvement.

from r.js.

I went ahead and tried a real simple content cache for the modules, and just updated the snapshot for today. Can you try the latest snapshot and report back numbers? Be sure to get the latest snapshot, commit 4575c6b (I have uploaded a few snapshots for today, all at the same location -- I just pushed the change).

It is not quite what you have above, but it should be saving file IO and AST parsing. I'm interested to know the results, if it gets to a decent speedup.

from r.js.

it increased the performance a lot! the same build that was taking 17min on Rhino now takes 42s, around 24s tracing dependencies.. on node it takes 1s to trace dependencies and 5s to complete optimization..

It still traces the same dependencies multiple times but I'm happy with the result :D - you can close the issue if you want, unless you think that the dictionary technique will help in some cases (I don't have that many nested or shared dependencies on my project, other people may still have issues with performance).

thanks!

from r.js.

Great to hear! I'll close the ticket now, and let someone else open a new ticket if they notice a problem. I think the 80% part of this optimization is done, and the current approach feels more accommodating for loader plugins that may actually need to execute code.

from r.js.