ayld / facade Goto Github PK

Because the Temp folder is hardcoded to /tmp in the config.

The project should be operable via console interface.

Something basic is needed currently, like the ability to do something of the sort:

java -jar facade.jar optimize /source/dir lib/dir

Which should call the current service implementations and create a facade jar for the /source/dir removing uneeded classes from the /lib/dir

Concurrent components should be selectable and usable via the API.

In other words something like.

final JarFile outJar = LibraryMinimizer
    .concurrentSourceResolution(threadsCount)
    .concurrentExtraction(threadsCount)
    .forSourcesAt("/abs/path/to/src/dir")
    .withLibs("/abs/path/to/libs")
    .getFile();

should be possible

The API should provide something like

FacadeApi
  .buildWithDefaultConfig()
  .dependencies()
  .fromSource(SourceFile.fromFile(aFile));

and

FacadeApi
  .buildWithDefaultConfig()
  .dependencies()
  .fromClass(ClassFile.fromFile(aFile));

Something simple please as this is a library not an app server :)

If you thing that most logging frameworks are too complex (as I do :D) a simple custom solution would also be nice.

Also when done check for any leftover system.outs

No seriously ... the current implementation is cluttered, unclear and undocumented.
More unit tests should be implemented and the implementation should be re factored to improve clarity.

Currently all concurrent components use very similar code (a thread pool to which they submit some Callables).

A utility can be made to hold the common code in a single place.

Multiple class resolvers can work on several .class files at the same time.

We (ApiBuilder) need a 'better' API. It is currently inconsistent and semantically incorrect.

After the redesign only additions and very minor changes should be done.

Currently the implementation only checks whether the class filename matches the given dependency name.
So if we're looking for com.something.SomeClass it will currently match the path extracted.jar/org/something/else/SomeClass.class which is incorrect. This should be fixed.

Also DependencyBundleResolver should provide the ability to change the match behavior easily.

GPL or LGPL

You fool !!!

Add feature for Facade to be able to separately slim down each jar independently and result in the original number of jars just 'slimmed' down.

Set up CI for Facade.
Any 'good one' will do (Jenkins/Bamboo/Team City) at the discretion of the implementer.

Currently the bean context is initialized when needed.
It would be better if it's initialization is extracted to a common place (a singleton enum maybe ?!).

Facade needs to be callable through maven in a way like:

mvn clean install facade:optimize

Which will compile and package the project with the facade.jar only.

A service is required which can, given a source directory resolve the dependencies of all source files inside the directory tree.

It should then return them as strings which are fully qualified class names.

The JarExploder is huge and cluttered, 2 huge methods no comments and bad var names.
Fix it naaaaao !!!

Something like

Minimizer
    .sources(srcDir)
    .libs(libDir)
    .includeClasses("some.runtime,Dependency", "some.other.runtime.Dep")
    .includeAllOf(aJar, anotherJar)

And the output jar would contain some.runtime,Dependency, some.other.runtime.Dep and all classes from aJar and anotherJar

Instances of logic in which ListenableComponents update their listener for status changes should not be 'in the same place' as business logic.

Probably via AOP as this is a completely separate aspect.

Multiple source resolvers can work on several files at the same.

We just need to modify the LICENSE file in the root dir.

UI commands should be able to return a result when appropriate.

A Result class for the console UI commands needs to be designed and implemented.
The commands which currently return void should also be changed.

What the title says.

Consider the method:

    public Object ohShit() throws ClassNotFoundException, InstantiationException {
        return Class.forName("hidden.runtime.Dependency").newInstance();
    }

With the current implementation the class hidden.runtime.Dependency will not be present in the output Jar.

So generally it must be decided whether we try to support some runtime dependencies (and how) or not.

Any suggestions/opinions are welcome in the comments.

We agreed that the new name should be Diet.

There should be a separate UI project for the console UI.

This way Facade can be distributed as a separate lib and the UIs can depend on it.

The minimizer usually uses the OS's temp folder to unzip lib jars. It then uses them to package the facade.jar.

It should remove the extracted folders after the packaging because consecutive runs on the same PC over a different lib will cause significant slowdowns since the resolvers have to query the previous resolution leftovers.

And also it is just leaves a mess after itself currently :)

Problem

java.io.FileNotFoundException: /home/ayld/.m2/repository/.../aether-util-1.7.jar (Too many open files)

This happens when you run Facade on a lib folder with a number of jars larger than the OS's ulimit. Which is not that rare actually. A local Maven repo on a machine seen several projects can have well over a 1000 jars.

On my linux:

ayld@pwn:~/m2/repository$ find . -name '*.jar' -type f | wc -l
1533

and

ayld@pwn:~/m2/repository$ ulimit -n
1024

Cause

The thing is that Java's ZipFile type actually opens the file when you create an instance (!!!) even if you don't use it later on.

So this loop that looks like it's searching for files is actually opening them:

for (File jarFile : Files.in(libDir).withExtension(JarMaker.JAR_FILE_EXTENSION).list()) {
    libJars.add(new JarFile(jarFile));
}

because of:

public JarFile(File file) throws IOException {
        this(file, true, ZipFile.OPEN_READ);
}

and

public ZipFile(File file, int mode, Charset charset) throws IOException
    {
        if (((mode & OPEN_READ) == 0) ||
            ((mode & ~(OPEN_READ | OPEN_DELETE)) != 0)) {
            throw new IllegalArgumentException("Illegal mode: 0x"+
                                               Integer.toHexString(mode));
        // omitted
        jzfile = open(name, mode, file.lastModified(), usemmap);
        // omitted 
    }

Why JarFile and not just File ?

Because it actually makes sure that the file is a jar and is singed properly (if needed).

Fixing

So is 1533 libs a valid case ? A user can easily work around this limitation if he backs up repo->builds his last project->runs Facade again or setting his ulimit higher (which is simple).

Should we start using File over JarFile or maybe out own JarFile implementation that doesn't open the files or opens->validates->closes them ?

Or some other fix ?

What the title says.

The API can contain a public enum for each listenable component.
Something like:

api.addComponentListener(FacadeApi.Component.CLASS_DEP_RESOLVER, new Listener() { // bla bla bla })

A service that can create a jar file from a list of given classes is required.
The list can be just strings with proper qualified name class format (like com.i.like.pie.Something), Class objects or file paths.

Most important components should extend the listenable interface.

Currently library jars are extracted by a single thread. Speed can be significantly improved if there are several threads extracting separate jars.

For the class commons-lang3-3.1.jar/org/apache/commons/lang3/CharRange$1.class the jar maker creates

org/apache/commons/lang3/CharRange$1/
org/apache/commons/lang3/CharRange$1/CharRange$1.class
org/apache/commons/lang3/CharRange.class

in the output jar which is incorrect.

It should only create

org/apache/commons/lang3/CharRange$1.class

Currently the source resolver iterates every line of every .java file while it only needs to look at the imports. Resolution can be terminated early when it reaches the class definition.

Think of a name for the organization that will host the project.

Just propose stuff I will count votes.

We also need a logo btw, I was thinking of using these guys as they rule the universe.
Also when done we need to create an account for the organization on Gittip.

At least for the interfaces for the love of god ...

A service is required which can, given a source directory resolve the dependencies of all source files inside the directory tree.

It should then return them as strings which are fully qualified class names.

Design a Listenable interface that should be implemented by the 'major components' (resolvers, matchers and bundle manipulators (jar maker/exploder)). This interface should require implementers to provide something like a notifyListeners() method.

Design a Listener interface which should be implemented in logic interested in component 'processing status' updates.

The source resolver can not handle wildcard imports.

Currently for the import:

import com.company.something.*;

It will resolve the dependency:

com.company.Something.class

Which is just bad ... :)

If we have say

some.package.Class.class

and

another.package.Class.class

these names collide in the output jar because of it's flat single directory structure.

The out jar should not be flatly structured.

We could print something like:

[INFO] You were using: 453 jars (322 transitive), with 5321 classes.
[INFO] Now using: 1 jar, 122 classes.

(these are real statistics from one of the tests btw ;) )

This can help the feel (lol!).

So since I decided (!) to distribute Facade not only as a Maven plugin, but as a lib also, there should be an API that given a list of sources and a list of .jars, can create the facade.jar and return it to the caller. This should be just s ''simple'' class.

So do that.

So as per discussion we'll use the Logowski guys.

There is a form at the end of the page at the above link. We need to decide stuff for the form.
So this issue is about that :)

Why : it is good to think up a catchy name before the project becomes very popular, because it would be harder to do it later
Propososition: IMHO the current name does not give much information aboit the project and what worries me most is that it might suggest different things die to the fact that there is a sofware develepment pattern with the same name
How: Open discussion with everybody contributing a suggestion. If we do not think of a good one with stick with thecurrent one

My suggestion : Diet
Slogan : Put your project on a diet
Why : short, kind of provokative, no double meaning

For instance it considers java.io.Serializable to be invalid.

This is the current regex ([a-zA-Z_$][a-zA-Z\\d_$]*\\.)*[a-zA-Z_$][a-zA-Z\\d_$]*.

Currently if a library contains about ~500 jars the used heap can go over 1.5G and the CPU load could be about 90% during resolution.

The reasons for this must be found and eliminated (mwaahahah !) if possible.

Since I'm late with all the milestones I decided to add more pointless features which will slow me down further (but are fun to write).

So I need a file util that can do the following:

Say we have a:
final File dir = new File("/somedir");

given the above, the following:

Files.in(dir).withExtension("class").named("ClassName").exclusive();

Would return all files in /somedir that are named ClassName and have the class extention. (which should really be just one file)

And

Files.in(dir).withExtension("class").named("ClassName").inclusive();

Would return all files in /dir that have the class extension and also all files named ClassName.

This checklist will become tasks when we decide what exactly we need.

This is intended as a discussion on what we need:

• Change the name
• Create GitHub organization
• Switch to LGPL
• Organization and project logo
• Better site
• Maven plugin deployed to a repo
• Demo project: pretty example with many graphs showcasing how a project with 50 libs becomes a project with 1

Any logging would do for now.

Either Logback or log4j. Preferably something that implements slf4j, but if there is something newer and better its ok.

All the System.out.print calls must be removed and replaced with the new logger.