Currently GeneratorPipeline attempts to generate cubes in the order they are requested. When player is moving slowly it's close to nearest-to-farthest, but when player is moving faster than cubes can be generated it's not the case.

The order of generation should be independent of stages, so I would propose putting all "queued" cubes into one ArrayList and sorting it every few ticks (only if any cubes gave been added). Any kind of self-sorting structure won't work here because the order changes as players move around.

Sometimes generating a single cube takes up to a few hundred milliseconds, while all other cubes generate with normal speed. This causes some performance issues.

Mob spawning has been temporarily disabled to avoid crashes.

Currently, RequiredCube stores all Dependents in a single collection. Whenever its wrapped cube advances a stage all Dependents receive the update, regardless of whether they actually need it. If RequiredCube had individual Collections per GeneratorStage the amount of unnecessary updates could be reduced.

This may be a fix for #7.

Currently GeneratorPipeline is time-based, which makes it hard to reproduce certain bugs. Adding a special debug-only mode that would test for count instead of time should make debugging easier (and would stop debugger from interfering with generator pipeline)

Some empty columns that appear outside of the view radius do not get unloaded

The current system works, but the code is hard to understand and tends to result in long hard to understand stacktraces when it breaks. It's also a bit slow.

What can be done?

There are 2 ways to fix it:

1. Simplify the dependency system. While this would certainly make it less general, the code would be more readable
2. Remove it. The only reason it exists is that generation pipeline needed it to work correctly. And there are 2 reasons the pipeline exists:

• chunk generation is slow with cubic chunks (this can be fixed without generation pipeline)
• with previous implementations of some parts of generators, there have been complicated dependencies between different parts. Specifically, surface block generator needed cube above to be loaded and be in stage before caves have been generated. As this part has been reimplemented to use density array used to generate terrain shape instead of scanning top-down - it's no longer needed.

How it can be done?

There are a few different world types and each of then works a bit differently. VanillaCubic is very special here. Generating a single cube from vanilla height range should call vanilla generator and generate the whole 16x256x16 chunk. In case normal chunk unloading will be readded - the chunk should still be unloaded by chunkGC and not automatically. VanillaCubic also needs special handling from ChunkGC and ChunkProviderServer. Making it column generator might make it easier (no special handling from ChunkProviderServer)

CustomCubic and FlatCubic are simple enough to be handled with normal vanilla-like mechanisms (generalized for cubic chunks).

About lighting updates dependencies...

These are also a bit complicated. Because there are 16 light values, generating light for the whole chunk requires chunks in 17 block radius --> 2 chunks radius to be loaded. This means that if only cubes in player render distance are loaded, only cubes in (player render distance - 2 chunks) will be sent to player.

So why it works in vanilla?

I would say it eighter works by accident, or someone who wrote it tried to be too clever.
The short answer is: It doesn't.
The long answer is: It shouldn't work, but because of some interactions with other code it mostly works.
In vanilla chunks are only sent to client when lighting is fully generated, and it can only be updated when chunks in radius of 2 are all loaded. And they shouldn't be loaded, except...
Chunk#enqueueRelightChecks (possibly unintentionally) loads chunks in 1 chunk radius of all chunks loaded by PlayerChunkMap when checking if it can update light at chunk edge. This combined with player usually moving back and forth generating chunks a bit further makes it mostly work.
Because Cubic Chunks doesn't attempt to change the way Chunk#enqueueRelightChecks works (only enough to work with infinite height) - this behavior can be replicated once generation pipeline and dependency system are removed.

This is caused by hundreds of thousands of scheduled ticks for gravel blocks. In vanilla it's not that bad because chunks are bigger than cubes.

The tall grass block appears to be fully lit when it shouldn't be:

Zip of world: https://mega.nz/#!GU1lnbjQ!Fp5Z5YmM91WOl8p5oRs_gE9KDzMFXq19SCMFsKW7vGc
Stacktrace:

net.minecraft.util.ReportedException: Ticking memory connection
    at net.minecraft.network.NetworkSystem.networkTick(NetworkSystem.java:210) ~[NetworkSystem.class:?]
    at net.minecraft.server.MinecraftServer.updateTimeLightAndEntities(MinecraftServer.java:806) ~[MinecraftServer.class:?]
    at net.minecraft.server.MinecraftServer.tick(MinecraftServer.java:687) ~[MinecraftServer.class:?]
    at net.minecraft.server.integrated.IntegratedServer.tick(IntegratedServer.java:156) ~[IntegratedServer.class:?]
    at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:536) [MinecraftServer.class:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]
Caused by: java.lang.IllegalArgumentException: minY > maxY (65 > 64)
    at cubicchunks.world.cube.Cube$LightUpdateData.queueLightUpdate(Cube.java:531) ~[Cube$LightUpdateData.class:?]
    at cubicchunks.lighting.LightingManager.queueDiffuseUpdate(LightingManager.java:74) ~[LightingManager.class:?]
    at cubicchunks.lighting.LightingManager.columnSkylightUpdate(LightingManager.java:59) ~[LightingManager.class:?]
    at cubicchunks.server.chunkio.IONbtReader.readLightingInfo(IONbtReader.java:255) ~[IONbtReader.class:?]
    at cubicchunks.server.chunkio.IONbtReader.readCube(IONbtReader.java:115) ~[IONbtReader.class:?]
    at cubicchunks.server.chunkio.CubeIO.loadCubeAndAddToColumn(CubeIO.java:170) ~[CubeIO.class:?]
    at cubicchunks.server.ServerCubeCache.loadCube(ServerCubeCache.java:348) ~[ServerCubeCache.class:?]
    at cubicchunks.server.ServerCubeCache.loadCube(ServerCubeCache.java:388) ~[ServerCubeCache.class:?]
    at cubicchunks.world.dependency.DependencyManager.register(DependencyManager.java:122) ~[DependencyManager.class:?]
    at cubicchunks.worldgen.dependency.DependentCubeManager.register(DependentCubeManager.java:75) ~[DependentCubeManager.class:?]
    at cubicchunks.worldgen.WorldGenerator.generateCube(WorldGenerator.java:156) ~[WorldGenerator.class:?]
    at cubicchunks.worldgen.WorldGenerator.generateCube(WorldGenerator.java:172) ~[WorldGenerator.class:?]
    at cubicchunks.server.ServerCubeCache.loadCube(ServerCubeCache.java:369) ~[ServerCubeCache.class:?]
    at cubicchunks.server.ServerCubeCache.loadCube(ServerCubeCache.java:388) ~[ServerCubeCache.class:?]
    at cubicchunks.world.dependency.DependencyManager.register(DependencyManager.java:122) ~[DependencyManager.class:?]
    at cubicchunks.worldgen.dependency.DependentCubeManager.register(DependentCubeManager.java:75) ~[DependentCubeManager.class:?]
    at cubicchunks.worldgen.WorldGenerator.generateCube(WorldGenerator.java:156) ~[WorldGenerator.class:?]
    at cubicchunks.worldgen.WorldGenerator.generateCube(WorldGenerator.java:172) ~[WorldGenerator.class:?]
    at cubicchunks.server.ServerCubeCache.loadCube(ServerCubeCache.java:340) ~[ServerCubeCache.class:?]
    at cubicchunks.server.ServerCubeCache.loadCube(ServerCubeCache.java:384) ~[ServerCubeCache.class:?]
    at cubicchunks.server.PlayerCubeMapEntry.<init>(PlayerCubeMapEntry.java:71) ~[PlayerCubeMapEntry.class:?]
    at cubicchunks.server.PlayerCubeMap.getOrCreateCubeWatcher(PlayerCubeMap.java:304) ~[PlayerCubeMap.class:?]
    at cubicchunks.server.PlayerCubeMap.lambda$addPlayer$6(PlayerCubeMap.java:366) ~[PlayerCubeMap.class:?]
    at cubicchunks.visibility.CuboidalCubeSelector.forAllVisibleFrom(CuboidalCubeSelector.java:43) ~[CuboidalCubeSelector.class:?]
    at cubicchunks.server.PlayerCubeMap.addPlayer(PlayerCubeMap.java:362) ~[PlayerCubeMap.class:?]
    at net.minecraft.server.management.PlayerList.preparePlayer(PlayerList.java:310) ~[PlayerList.class:?]
    at net.minecraft.server.management.PlayerList.playerLoggedIn(PlayerList.java:389) ~[PlayerList.class:?]
    at net.minecraft.server.management.PlayerList.initializeConnectionToPlayer(PlayerList.java:171) ~[PlayerList.class:?]
    at net.minecraftforge.fml.common.network.handshake.NetworkDispatcher.completeServerSideConnection(NetworkDispatcher.java:263) ~[NetworkDispatcher.class:?]
    at net.minecraftforge.fml.common.network.handshake.NetworkDispatcher.access$100(NetworkDispatcher.java:73) ~[NetworkDispatcher.class:?]
    at net.minecraftforge.fml.common.network.handshake.NetworkDispatcher$1.update(NetworkDispatcher.java:212) ~[NetworkDispatcher$1.class:?]
    at net.minecraft.network.NetworkManager.processReceivedPackets(NetworkManager.java:308) ~[NetworkManager.class:?]
    at net.minecraft.network.NetworkSystem.networkTick(NetworkSystem.java:195) ~[NetworkSystem.class:?]
    ... 5 more

With render distance 32 updateMountedMovingPlayer takes 1-3 seconds.

Generation radius=12:
3 minutes and 55 seconds when generating chunks around player, vs 50 seconds for spawn chunks.

Commands to fix:

• CommandSetSpawnpoint (spawnpoint)
• CommandSetDefaultSpawnpoint (setworldspawn)
• CommandSpreadPlayers - too low priority and too much work. should world fine with terrain height 0-255
• CommandTeleport
• CommandSetBlock - fixing this one would make testing lighting much easier
• CommandFill - same as above
• CommandClone
• CommandCompare (testforblocks)
• CommandBlockData
• CommandTestForBlock

After decreasing render distance, all cubes that are outside of the new render distance but inside the old render distance are not sent to the client.

I compiled the repo then attempted to run it in mine craft but i crashed almost immediately

log: http://paste.ee/p/BUY3o#

It happened twice to me, once when I didn't expect it, second time when I tried to make it ctrash but set exception breakpoint to the wrong class:

net.minecraft.util.ReportedException: Exception ticking world
    at net.minecraft.server.MinecraftServer.updateTimeLightAndEntities(MinecraftServer.java:778) ~[MinecraftServer.class:?]
    at net.minecraft.server.MinecraftServer.tick(MinecraftServer.java:687) ~[MinecraftServer.class:?]
    at net.minecraft.server.integrated.IntegratedServer.tick(IntegratedServer.java:156) ~[IntegratedServer.class:?]
    at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:536) [MinecraftServer.class:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: java.lang.NullPointerException
    at it.unimi.dsi.fastutil.ints.Int2ObjectOpenHashMap$MapIterator.nextEntry(Int2ObjectOpenHashMap.java:511) ~[Int2ObjectOpenHashMap$MapIterator.class:?]
    at it.unimi.dsi.fastutil.ints.Int2ObjectOpenHashMap$ValueIterator.next(Int2ObjectOpenHashMap.java:706) ~[Int2ObjectOpenHashMap$ValueIterator.class:?]
    at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1042) ~[?:1.8.0_91]
    at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1042) ~[?:1.8.0_91]
    at cubicchunks.lighting.SkyLightUpdateCubeSelector.getCubesY(SkyLightUpdateCubeSelector.java:75) ~[SkyLightUpdateCubeSelector.class:?]
    at cubicchunks.lighting.LightingManager.columnSkylightUpdate(LightingManager.java:50) ~[LightingManager.class:?]
    at cubicchunks.world.column.Column.doOnBlockSetLightUpdates(Column.java:236) ~[Column.class:?]
    at cubicchunks.world.column.Column.setBlockState(Column.java:175) ~[Column.class:?]
    at net.minecraft.world.World.setBlockState(World.java:384) ~[World.class:?]
    at cubicchunks.worldgen.generator.custom.features.FeatureGenerator.setBlockOnly(FeatureGenerator.java:44) ~[FeatureGenerator.class:?]
    at cubicchunks.worldgen.generator.custom.features.trees.TreeGenerator.tryToPlaceDirtUnderTree(TreeGenerator.java:80) ~[TreeGenerator.class:?]
    at cubicchunks.worldgen.generator.custom.features.trees.SimpleTreeGenerator.canGenerateTree(SimpleTreeGenerator.java:84) ~[SimpleTreeGenerator.class:?]
    at cubicchunks.worldgen.generator.custom.features.trees.SimpleTreeGenerator.generateAt(SimpleTreeGenerator.java:58) ~[SimpleTreeGenerator.class:?]
    at cubicchunks.worldgen.generator.custom.features.SurfaceFeatureGenerator.generate(SurfaceFeatureGenerator.java:64) ~[SurfaceFeatureGenerator.class:?]
    at cubicchunks.worldgen.generator.custom.features.VariantFeatureGenerator.generate(VariantFeatureGenerator.java:57) ~[VariantFeatureGenerator.class:?]
    at cubicchunks.worldgen.generator.custom.features.MultiFeatureGenerator.generate(MultiFeatureGenerator.java:45) ~[MultiFeatureGenerator.class:?]
    at cubicchunks.worldgen.generator.custom.CustomPopulationProcessor.populate(CustomPopulationProcessor.java:63) ~[CustomPopulationProcessor.class:?]
    at cubicchunks.world.type.CustomCubicChunksWorldType$1.populate(CustomCubicChunksWorldType.java:73) ~[CustomCubicChunksWorldType$1.class:?]
    at cubicchunks.server.ServerCubeCache.lambda$getCube$0(ServerCubeCache.java:352) ~[ServerCubeCache.class:?]
    at cubicchunks.util.Box.forEachPoint(Box.java:44) ~[Box.class:?]
    at cubicchunks.server.ServerCubeCache.getCube(ServerCubeCache.java:347) ~[ServerCubeCache.class:?]
    at cubicchunks.server.PlayerCubeMapEntry.providePlayerCube(PlayerCubeMapEntry.java:133) ~[PlayerCubeMapEntry.class:?]
    at cubicchunks.server.PlayerCubeMap.tick(PlayerCubeMap.java:263) ~[PlayerCubeMap.class:?]
    at net.minecraft.world.WorldServer.tick(WorldServer.java:230) ~[WorldServer.class:?]
    at net.minecraft.server.MinecraftServer.updateTimeLightAndEntities(MinecraftServer.java:772) ~[MinecraftServer.class:?]
    ... 4 more

OpacityIndex is saved expecting all cubes to be there. If these cubes aren't saved properly - heightmap will be wrong causing failures when calculating skylight.

PlayerChunkMap is basically the vanilla equivalent of GeneratorPipeline, that also manages cubes for players. Information about which players load which cubes would be useful also for GeneratorPipeline. Currently they are separated, which causes some issues (like generating cubes in incorrect order).

What would make most sense is probably refactoring them into 2 different classes, one of them would deal with loading and generating chunks, the other with sending updates to client and keeping track of which chunks are loaded by which player as players move.

Doin gthat should also remove some duplicate work being done, which should improve performance.

It causes redstone clocks to stop working after reload, interrupted liquid flow doesn't continue etc...

As minecraft updates, it needs to be re-checked again. Last time most of it has been checked in 1.8, some of it in 1.9.

ChunkCache needs to be modified to use Cubes instead of Columns because it's used from multiple threads (Chunk Batchers)

Using the new dependency system should fix it.

• Terrain stage - terrain looks differently than vanilla terrain (incorrectly configured noise generator and biome-related code)
• Population stage - mostly unimplemented /*TODO: list all features to implement*/
  • Basic biome decorator
    • Make all of that work beyond vanilla heights
  • Biome-specific decorator
  • General populator
• Structure generation - only caves and ravines are implemented (will it be implemented?)
• Lava in caves
• Customization options
  • Advanced terrain shape
  • Basic population options
  • Ore configuration
  • Block replacer config
  • Allow setting values directly
  • Show resulting json config
• Merge Terrain Surface and Structures stages

When it happens on newly loaded chunks, it's usually reproducible with that world, but only with the same render distance and from the same (or close) player position

It may be a vanilla bug.

Example world where it is reproducible (at least on my computer) with render distance=8:
seed=8660360411258039337
player position: X=9765, Y=70, Z=245, facing: south-east (positive Z, positive Z)

Flying around changes only some surrounding chunks, but some chunks remain invisible. As long as player stays in the same position when joining the world, it's reproducible.

Screenshot:

#18

The cubes are no longer present in the ServerCubeCache.

The error is:

java.lang.NullPointerException
    at cubicchunks.world.dependency.RequiredCube.setTargetStage(RequiredCube.java:96) ~[RequiredCube.class:?]
    at cubicchunks.world.dependency.DependencyManager.register(DependencyManager.java:101) ~[DependencyManager.class:?]
    at cubicchunks.worldgen.dependency.DependentCubeManager.register(DependentCubeManager.java:75) ~[DependentCubeManager.class:?]
//and so on

(full stacktrace: http://pastebin.com/5D2pHYzD)

This causes holes in the world, because forge ignores these exceptions.

The reason is that some cubes that it uses are not loaded yet.

This may be vanilla bug.

Falling blocks also turn into items way to quickly under y:0

Using the dependency system we can define minimum distance that need to be maintained when working on several cubes at once. For example, if cube A is currently being processed in the lighting stage, no cube within a radius of 2 around it must be altered from the outside. Leveraging this information workers could avoid each other without changing everything to be inherently thread-safe.

Currently rain uses light heightmap which ignores transparent blocks.

With VanillaCubic world type some cubes are not unloaded (it's not implemented yet)

It may be caused by not removing cubes that depend on unloaded cube.

Code used to set up GeneratorPipeline:

                GeneratorStage terrain = new IndependentGeneratorStage("terrain");
                GeneratorStage surface = new GeneratorStage("surface") {
                        @Override public Dependency getDependency(Cube cube) {
                                //cube above must exist and can't be before SURFACE stage.
                                //also in the next stage need to make sure that we don't generate structures
                                //when biome blocks aren't placed in cube below
                                return new RegionDependency(this, 0, 0, 0, 1, 0, 0);
                        }
                };
                GeneratorStage features = new GeneratorStage("features") {
                        @Override public Dependency getDependency(Cube cube) {
                                return new RegionDependency(this, 0, 0, -1, 0, 0, 0);
                        }
                };
                GeneratorStage population = new GeneratorStage("population") {
                        @Override public Dependency getDependency(Cube cube) {
                                return new RegionDependency(this, 0, 1, 0, 1, 0, 1);
                        }
                };
                //everything else as usual

And LIGHTING stage:

        public static GeneratorStage LIGHTING = new GeneratorStage("lighting") {
                @Override public Dependency getDependency(Cube cube) {
                        return new RegionDependency(LIGHTING, 2);
                }
        };

It crashes if I remove cubesExist checks from stage processors, and it doesn't finish generation if I leave them.

Currently I identified the following things to configure:

• Max amount of cubes generated per tick (default 49*16)
• Max time spent generating cubes per tick (default 50 milliseconds)
• Vertical cube load distance (options: default=same as horizontal, and values 2-32)
  • Make this option available in video settings and server.properties
• ​(maybe) Max client skylight update depth (default: 64)
• Enable/disable ChunkGC (when normal chunk unloading is readded) not really possible
• Min/Max world height
  • Dimension height and depth stored in NBT
  • Configurable in world type customization options (default value for new dimensions and separate values for dimension IDs)
  • When cubic chunks toggle switch is added - Allow to set default values for new dimensions and separate values for dimension IDs
    (list will be updated as I find more options)