Git Product home page Git Product logo

etc2comp's Introduction

Etc2Comp - Texture to ETC2 compressor

This repo is no longer maintained.

Etc2Comp is a command line tool that converts textures (e.g. bitmaps) into the ETC2 format. The tool is built with a focus on encoding performance to reduce the amount of time required to compile asset heavy applications as well as reduce overall application size.

This repo provides source code that can be compiled into a binary. The binary can then be used to convert textures to the ETC2 format.

Important: This is not an official Google product. It is an experimental library published as-is. Please see the CONTRIBUTORS.md file for information about questions or issues.

Setup

This project uses CMake to generate platform-specific build files:

  • Linux: make files
  • OS X: Xcode workspace files
  • Microsoft Windows: Visual Studio solution files
  • Note: CMake supports other formats, but this doc only provides steps for one of each platform for brevity.

Refer to each platform's setup section to setup your environment and build an Etc2Comp binary. Then skip to the usage section of this page for examples of how to use the library.

Setup for OS X

build tested on this config: OS X 10.9.5 i7 16GB RAM Xcode 5.1.1 cmake 3.2.3

Start by downloading and installing the following components if they are not already installed on your development machine.

  • Xcode version 5.1.1, or greater
  • CMake version 3.2.3, or greater

To build the Etc2Comp binary:

  1. Open a Terminal window and navigate to the project directory.
  2. Run mkdir build_xcode
  3. Run cd build_xcode
  4. Run cmake -G Xcode ../
  5. Open Xcode and import the build_xcode/EtcTest.xcodeproj file.
  6. Open the Product menu and choose Build For -> Running.
  7. Once the build succeeds the binary located at build_xcode/EtcTool/Debug/EtcTool can be executed.

Optional Xcode EtcTool ‘Run’ preferences note: if the build_xcode/EtcTest.xcodeproj is manually deleted then some Xcode preferences will need to be set by hand after cmake is run (these prefs are retained across cmake updates if the .xcodeproj is not deleted/removed)

  1. Set the active scheme to ‘EtcTool’
  2. Edit the scheme
  3. Select option ‘Run EtcTool’, then tab ‘Arguments’. Add this launch argument: ‘-argfile ../../EtcTool/args.txt’
  4. Select tab ‘Options’ and set a custom working directory to: ‘$(SRCROOT)/Build_Xcode/EtcTool’

SetUp for Windows

  1. Open a Terminal window and navigate to the project directory.
  2. Run mkdir build_vs
  3. Run cd build_vs
  4. Run CMAKE, noting what build version you need, and pointing to the parent directory as the source root; For VS 2013 : cmake -G "Visual Studio 12 2013 Win64" ../ For VS 2015 : cmake -G "Visual Studio 14 2015 Win64" ../ NOTE: To see what supported Visual Studio outputs there are, run cmake -G
  5. open the 'EtcTest' solution
  6. make the 'EtcTool' project the start up project
  7. (optional) in the project properties, under 'Debugging ->command arguments' add the argfile textfile thats included in the EtcTool directory. example: -argfile C:\etc2\EtcTool\Args.txt

Setup For Linux

The Linux build was tested on this config: Ubuntu desktop 14.04 gcc/g++ 4.8 cmake 2.8.12.2

  1. Verify linux has cmake and C++-11 capable g++ installed
  2. Open shell
  3. Run mkdir build_linux
  4. Run cd build_linux
  5. Run cmake ../
  6. Run make
  7. navigate to the newly created EtcTool directory cd EtcTool
  8. run the executable: ./EtcTool -argfile ../../EtcTool/args.txt

Skip to the Usage section for more information about using the tool.

Usage

Command Line Usage

EtcTool can be run from the command line with the following usage: etctool.exe source_image [options ...] -output encoded_image

The encoder will use an array of RGBA floats read from the source_image to create an ETC1 or ETC2 encoded image in encoded_image. The RGBA floats should be in the range [0:1].

Options:

-analyze <analysis_folder>
-argfile <arg_file>           additional command line arguments read from a file
-blockAtHV <H V>              encodes a single block that contains the
                              pixel specified by the H V coordinates
-compare <comparison_image>   compares source_image to comparison_image
-effort <amount>              number between 0 and 100 to specify the encoding quality 
                              (100 is the highest quality)
-errormetric <error_metric>   specify the error metric, the options are
                              rgba, rgbx, rec709, numeric and normalxyz
-format <etc_format>          ETC1, RGB8, SRGB8, RGBA8, SRGB8, RGB8A1,
                              SRGB8A1 or R11
-help                         prints this message
-jobs or -j <thread_count>    specifies the number of threads (default=1)
-normalizexyz                 normalize RGB to have a length of 1
-verbose or -v                shows status information during the encoding
                              process
-mipmaps or -m <mip_count>    sets the maximum number of mipaps to generate (default=1)
-mipwrap or -w <x|y|xy>       sets the mipmap filter wrap mode (default=clamp)
  • -analyze will run an analysis of the encoding and place it in folder "analysis_folder" (e.g. ../analysis/kodim05). within the analysis_folder, a folder will be created with a name of the current date/time (e.g. 20151204_153306). this date/time folder is used to compare encodings of the same texture over time.
    within the date/time folder is a text file with several encoding stats and a 2x png image showing the encoding mode for each 4x4 block.

  • -argfile allows additional command line arguments to be placed in a text file

  • -blockAtHV selects the 4x4 pixel subset of the source image at position (H,V).
    This is mainly used for debugging

  • -compare compares the source image to the created encoded image. The encoding will dictate what error analysis is used in the comparison.

  • -effort uses an "amount" between 0 and 100 to determine how much additional effort to apply during the encoding.

  • -errormetric selects the fitting algorithm used by the encoder. "rgba" calculates RMS error using RGB components that are weighted by A. "rgbx" calculates RMS error using RGBA components, where A is treated as an additional data channel, instead of as alpha. "rec709" is similar to "rgba", except the RGB components are also weighted according to Rec709. "numeric" calculates RMS error using unweighted RGBA components.
    "normalize" calculates error based on dot product and vector length for RGB and RMS error for A.

  • -help prints out the usage message

  • -jobs enables multi-threading to speed up image encoding

  • -normalizexyz normalizes the source RGB to have a length of 1.

  • -verbose shows information on the current encoding process. It will then display the PSNR and time time it took to encode the image.

  • -mipmaps takes an argument that specifies how many mipmaps to generate from the source image. The mipmaps are generated with a lanczos3 filter using edge clamping. If the mipmaps option is not specified no mipmaps are created.

  • -mipwrap takes an argument that specifies the mipmap filter wrap mode. The options are "x", "y" and "xy" which specify wrapping in x only, y only or x and y respectively. The default options are clamping in both x and y.

Note: Path names can use slashes or backslashes. The tool will convert the slashes to the appropriate polarity for the current platform.

API

The library supports two different APIs - a C-like API that is not heavily class-based and a class-based API.

main() in EtcTool.cpp contains an example of both APIs.

The Encode() method now returns an EncodingStatus that contains bit flags for reporting various warnings and flags encountered when encoding.

Copyright

Copyright 2015 Etc2Comp Authors.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

etc2comp's People

Contributors

hnknta avatar kablammyman avatar mainroach avatar paulrashidi avatar tjohns avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

etc2comp's Issues

Compressing small images (8x8 px png file)

Hi,
If i use the tool to compress a png image of 8x8 pixels, the result is wrong. I can see it by using analyze and compare options. I attach the image used for the test.
image (very small png)

Call to constructor in method body.

In file EtcBlock4x4.cpp, line 88 (and many other files and methods) there is call to constructor of that class.
This call does not changes current object (this), instead it creates a temporary object and calls constructor for it, so this call doesn't have effect that you expect from it.

Sorry for my bad English.
BR, Leon.

Add support for arrays

It will be nice if this tools supports also arrays of images (similar with PVRTexToolCLI)

Add support for output in PKM format

Code for determining output file format is incomplete - ETCTool always saves output in KTX format. Procesing of file format Format::INFER_FROM_FILE_EXTENSION in EtcFile.cpp has no proper implementation (actually, it just has a todo at line 19).

build instructions for Linux

In the README.md file under Setup For Linux, we are told to "Invoke cmake to create the make files". Being unfamiliar with cmake, it took me a while to figure out that one is supposed to issue the command "cmake .." here. Would you please update the README file?

Memory leak of m_paucEncodingBits in Etc::Image

In Image::~Image, the code to free m_paucEncodingBits is commented out. Presumably this is because one of the Image::Image constructors allows the caller to pass in their own encoding bits so calling delete[] would be unsafe in this scenario. However, in other codepaths, the class allocates it's own m_paucEncodingBits in the Image::Encode function which is never freed.

A possible solution would be to add a bool to Etc::Image to track whether or not it allocated m_paucEncodingBits, and free when necessary.

Block should not know Image

Currently a block needs to reference an Image, which makes writing Unit Tests touch way more code than it should be.

-output flag not handling paths with spaces

The mkdir call is being made with unquoted path arguments causing paths with spaces to break the directory creation.

EtcTool.cpp(824)
old:
sprintf_s(strCommand, "if not exist %s %s %s", path, ETC_MKDIR_COMMAND, path);
fix:
sprintf_s(strCommand, "if not exist "%s" %s "%s"", path, ETC_MKDIR_COMMAND, path);

Arm64 support

Given Apple's recent announcement, are there any plans to support mac arm64 library building?

Compressing normalmaps

What is a correct way to compress normalmaps?
From reading readme it looks like something like "-normalizexyz -errormetric normalxyz -format RGB8" should work better than "-error metric rgba". But with this errormetric I get some artefacts: https://ibb.co/dyL4dk (left is a source image, center is "-normalizexyz -errormetric normalxyz -effort 50 -format RGB8" and right is "-errormetric rgba -effort 50 -format RGB8").
Source image for tests: https://ibb.co/gkTqJk

Option to compress in ETC1 mode for speed

As the previos issue (#20) was closed, opening this one with a better explanation of the desired feature.
As far as I understant, ETC2 supports the ETC1 specification subset.

Would it be possible to compress ETC2 textures as ETC1, as in, less quality/modes but having the benefit of alpha support, R and RG formats, etc. so compression speed can be increased in exchange for quality?

Thanks!

Make sure Colt McAnlis properly reads issues before closing them

I posted two times, you closed the issues #21 and #20 and did not even bother to read what I write when I answer you..
I will use bullet points to make sure this time you read them.

  • NEED TRANSPARENCY, R and RG COMPRESSION Please get this into your head before answering I need to go and find a ETC1 compressor.
  • NEED OPTION FOR FASTER INITIAL COMPRESSION SPEED I was just suggesting that, as ETC2 is backwards compatible with ETC1, to just optionally compress in ETC1 WHILE KEEPING SUPPORT FOR TRANSPARENCY, R and RG CHANNELS, WHICH ARE ETC2 FEATURES STILL NEEDED FOR THIS

Thanks a lot for your time, sorry for using caps and re-posting the issue, but I hope this time the message is clearer.

PNG and EAC_RG11 support

Do most people really have float image data to feed into an encoder? Shouldn't this take png files containing data and compress them like all of the other tools.

I also don't see EAC_RG11 support, but there is EAC_R11. The former is useful for normal maps which only have two uncorrelated channels.

relative path for output

When I tried my first file conversion with EtcTool, all I got was "couldnt find a place to put converted images". Looking in the source code, I found this is because EtcTool expects to find a slash in the output filename. It would be nice if something like
EtcTool infile.png -output outfile.etc2
worked.

Slow

ETC2Comp is still slow compared to other texture compressors, even at effort 0 (effort > 0.5 is pretty much unusably slow). For a 2048x2048 in a AMD A8 takes 20 seconds. Squish takes less than one for BC compression.

Is there any chance to add a faster but less quality mode? This way user can pick which textures he/she wants in higher quality manually later.

Allow alpha channel with data other than opacity

I have textures that have an alpha channel that represents data other than opacity. For example part of the model is colorized with a predefined color in the shader according to the texture alpha channel.

Now if the alpha value is 0 (no colorization in my example case) the RGB values are not valid anymore after the compression.

R11 and RG11 seem to be computing data as 8-bits instead of 11-bit data.

I don't know if this is artifact of copy-pasting the A8 code over to form the R11/G11 calculators. This library seems to be treating incoming channel data as 8-bit for R11 and RG11 encoding. Also constant tables seem to be pre-divided by 255.0f. This means the higher precision of these modes may not be preserved, since the library takes full float unorm data in as input.

This may be an artifact of the ETC2 format storing the base in 8-bits, but I'm trying to lower up the iteration count by computing error in integer space, so I scale some other terms bye 2047.0f.

In CaclculateR11/G11 source:

if (isSnorm)
{
m_fRedBase = (fBase * 255) - 128; // TODO: why 255, shouldn't this be * 1023 - 1024
}
else
{
m_fRedBase = 255.0f * fBase; // TODO: * 2047.0f?
}

Not all mipmap levels are generated for non-square images

I'm using EtcTool to compress rectangular, power of 2 images (2048x1024) with mipmaps for use in WebGL. WebGL Refuses to load the textures when one of the mipmap-based minFilters is selected.

The generated KTX file contains mipmaps from 2048x1024 down to 2x1. WebGL seems to require a 1x1 mipmap.

I've changed this in my fork and the textures now load correctly. The changes are trivial. If there's any interest, I can try to create a pull request, although my open source contribution skills are rusty

assert() abort with ETC2A1 & RGBX

There is an internal inconsistency in the handling of m_boolPunchThroughAlpha here that triggers with the RGBX error metric.

We could probably fix it fairly easily, but there's little indication that pull requests are being accepted here anymore. Is there a fork that's actively maintained?

EtcTool crash

Hi guys, sometimes i have a crash in the EtcTool.

I reviewed the code and i found a char array with a bad size.

In the EtcTool.cpp in function ProcessCommandLineArguments
if (pstrOutputFilename[c] == ETC_PATH_SLASH) { c++; ptrOutputDir = new char[c]; strncpy(ptrOutputDir, pstrOutputFilename, c); ptrOutputDir[c] = '\0'; CreateNewDir(ptrOutputDir); break; }

The ptrOutputDir variable has a bad size, the must be c+1, i changed my code to:

if (pstrOutputFilename[c] == ETC_PATH_SLASH) { c++; ptrOutputDir = new char[c+1]; strncpy(ptrOutputDir, pstrOutputFilename, c); ptrOutputDir[c] = '\0'; CreateNewDir(ptrOutputDir); break; }

And now all run right.

Thanks,

Ruben

Near-infinite compression time on some textures.

The Godot engine is using this, and there's some code in EtcBlock4x4Encoding_RGB8.cpp and EtcBlock4x4Encoding_RGB8A1.cpp that has multiple loops with very large values.

iMaxRed2 = 252645152, iMaxGreen2 = 252645152, iMaxBlue2 = 252645152

            for (int iRed2 = iMinRed2; iRed2 <= iMaxRed2; iRed2++)
            {
                for (int iGreen2 = iMinGreen2; iGreen2 <= iMaxGreen2; iGreen2++)
                {
                    for (int iBlue2 = iMinBlue2; iBlue2 <= iMaxBlue2; iBlue2++)
                    {

After several minutes iRed2 was only at 15/252645152. I've attached the file I was trying to import:

water_normals2

Importing just the red and green channels is fine, but importing all of them results in a near-infinite loop.

This code looks suspect, but I'm not sure if that's by design or not:

		int iMaxRed2 = iColor2Red + (int)a_uiRadius;
		if (iMaxRed2 > 15)
		{
			iMinRed2 = 15;
		}

Several if statements check the max value, then set the min value. Should they be setting the max value to 15 instead?

Which format?

The source seems formatted a bit unconventional. Is there a clang format present?

how to improve compression speed

I test a 1024*1024 png image, the project run fine , the compresion formate is RGB8, file formate is pkm, But it cost 8 senconds.
But now i have 100000 image to do like this, so how to improve the speed. I use the "mali Texture compresion tool v4.3.0 " with fast mode , it only cost 1.7 sconds.

Default number of jobs is not correct

Documentation states that number of jobs by default is 1. However, when checking the source code, there is a macro definition called MIN_JOBS setting that value to 8.

So when etc2comp is run with parameters -jobs x (x > 0 and x < 9) etc2 spawns 11 threads. And when x > 8, thread count increases linear as expected.

EtcTool encoded sRGB(A) textures don't match source PNG.

The resulting images from EtcTool are too bright. This likely indicates some sRGB conversion on import, without a conversion back. Typically want to convert to high-precision linear color, build mips, and then convert each mip back to sRGB before the encode, or convert the endpoints ideally after the encode (so that you're fitting to linear and potentially premul colors). PVRTexToolCLI had a similar bug up until v4.24.

I also applied the min job change to 1, rectangular mip fixes, and max color bounding fixed listed as issues elsewhere on this bitrotting project. Timings for EtcTool (supposed to be so fast) are 900s at q70 vs. 44s for -qetcfast with PVRTexToolCLI using etcpack for the 100MB corpus of image that I'm using. I'm assuming 70 is pretty high quality, but this really isn't the fastest encoder in town or we're not using the magic setting for Etc2 textures. I'm assuming this project was abandoned once Android got ASTC support.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.