kerlomz / captcha_trainer Goto Github PK

使用的编译版GUI
当我使用默认的Channel配置值 3 时，Start Training会出现
ValueError: zero-size array to reduction operation maximum which has no identity
而将Channel改为1时则正常进行。
请问这是什么问题？

INFO:tensorflow:Step: 400 Time: 0.846 sec/batch, Cost = 1.53476572, BatchSize: 64, Shape[1]: 18
INFO:tensorflow:Step: 500 Time: 0.792 sec/batch, Cost = 1.08882499, BatchSize: 64, Shape[1]: 18
2019-12-23 15:55:03.028956: W tensorflow/core/common_runtime/bfc_allocator.cc:314] Allocator (GPU_0_bfc) ran out of memory trying to allocate 615.23MiB (rounded to 645120000). Current allocation summary follows.
2019-12-23 15:55:03.035027: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (256): Total Chunks: 128, Chunks in use: 127. 32.0KiB allocated for chunks. 31.8KiB in use in bin. 15.6KiB client-requested in use in bin.
2019-12-23 15:55:03.042178: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (512): Total Chunks: 43, Chunks in use: 43. 21.5KiB allocated for chunks. 21.5KiB in use in bin. 17.8KiB client-requested in use in bin.
2019-12-23 15:55:03.047903: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.3KiB allocated for chunks. 1.3KiB in use in bin. 1.0KiB client-requested in use in bin.
2019-12-23 15:55:03.053329: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2048): Total Chunks: 4, Chunks in use: 4. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 13.5KiB client-requested in use in bin.
2019-12-23 15:55:03.060581: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4096): Total Chunks: 5, Chunks in use: 4. 31.8KiB allocated for chunks. 24.0KiB in use in bin. 24.0KiB client-requested in use in bin.
2019-12-23 15:55:03.068945: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-12-23 15:55:03.076905: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16384): Total Chunks: 9, Chunks in use: 8. 175.0KiB allocated for chunks. 157.0KiB in use in bin. 145.5KiB client-requested in use in bin.
2019-12-23 15:55:03.084378: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (32768): Total Chunks: 5, Chunks in use: 4. 160.0KiB allocated for chunks. 128.0KiB in use in bin. 128.0KiB client-requested in use in bin.
2019-12-23 15:55:03.091372: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-12-23 15:55:03.098174: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (131072): Total Chunks: 17, Chunks in use: 16. 3.11MiB allocated for chunks. 2.90MiB in use in bin. 2.77MiB client-requested in use in bin.
2019-12-23 15:55:03.105264: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-12-23 15:55:03.112143: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-12-23 15:55:03.118394: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-12-23 15:55:03.125309: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-12-23 15:55:03.131695: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4194304): Total Chunks: 1, Chunks in use: 0. 7.61MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-12-23 15:55:03.139277: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8388608): Total Chunks: 1, Chunks in use: 0. 8.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-12-23 15:55:03.145077: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-12-23 15:55:03.151896: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-12-23 15:55:03.159025: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-12-23 15:55:03.165895: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (134217728): Total Chunks: 1, Chunks in use: 0. 128.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-12-23 15:55:03.172894: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (268435456): Total Chunks: 4, Chunks in use: 1. 1.96GiB allocated for chunks. 615.23MiB in use in bin. 615.23MiB client-requested in use in bin.
2019-12-23 15:55:03.179905: I tensorflow/core/common_runtime/bfc_allocator.cc:780] Bin for 615.23MiB was 256.00MiB, Chunk State:
2019-12-23 15:55:03.183511: I tensorflow/core/common_runtime/bfc_allocator.cc:786] Size: 317.34MiB | Requested Size: 247.61MiB | in_use: 0 | bin_num: 20, prev: Size: 615.23MiB | Requested Size: 615.23MiB | in_use: 1 | bin_num: -1, next: Size: 144.0KiB | Requested Size: 144.0KiB | in_use: 1 | bin_num: -1
2019-12-23 15:55:03.193406: I tensorflow/core/common_runtime/bfc_allocator.cc:786] Size: 512.00MiB | Requested Size: 307.62MiB | in_use: 0 | bin_num: 20
2019-12-23 15:55:03.198517: I tensorflow/core/common_runtime/bfc_allocator.cc:786] Size: 558.68MiB | Requested Size: 307.62MiB | in_use: 0 | bin_num: 20, prev: Size: 144.0KiB | Requested Size: 144.0KiB | in_use: 1 | bin_num: -1
2019-12-23 15:55:03.205800: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 1048576
2019-12-23 15:55:03.208822: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703600000 next 1 of size 256
2019-12-23 15:55:03.213355: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703600100 next 2 of size 256
2019-12-23 15:55:03.216876: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703600200 next 3 of size 227328
2019-12-23 15:55:03.220694: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703637A00 next 4 of size 256
2019-12-23 15:55:03.224816: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703637B00 next 5 of size 256
2019-12-23 15:55:03.228228: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703637C00 next 6 of size 256
2019-12-23 15:55:03.232721: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703637D00 next 7 of size 256
2019-12-23 15:55:03.236215: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703637E00 next 8 of size 256
2019-12-23 15:55:03.240207: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703637F00 next 9 of size 256
2019-12-23 15:55:03.243924: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703638000 next 10 of size 18432
2019-12-23 15:55:03.247537: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363C800 next 11 of size 512
2019-12-23 15:55:03.251887: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363CA00 next 12 of size 512
2019-12-23 15:55:03.255408: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363CC00 next 13 of size 512
2019-12-23 15:55:03.259702: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363CE00 next 14 of size 512
2019-12-23 15:55:03.263116: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363D000 next 15 of size 512
2019-12-23 15:55:03.267501: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363D200 next 16 of size 256
2019-12-23 15:55:03.270991: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363D300 next 17 of size 256
2019-12-23 15:55:03.274735: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363D400 next 18 of size 3584
2019-12-23 15:55:03.278855: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363E200 next 19 of size 512
2019-12-23 15:55:03.282935: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363E400 next 20 of size 256
2019-12-23 15:55:03.287390: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070363E500 next 21 of size 147456
2019-12-23 15:55:03.291455: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703662500 next 22 of size 256
2019-12-23 15:55:03.295633: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703662600 next 23 of size 256
2019-12-23 15:55:03.299245: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703662700 next 24 of size 256
2019-12-23 15:55:03.303536: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703662800 next 25 of size 6144
2019-12-23 15:55:03.307109: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703664000 next 26 of size 256
2019-12-23 15:55:03.310887: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703664100 next 27 of size 256
2019-12-23 15:55:03.315193: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703664200 next 28 of size 204800
2019-12-23 15:55:03.318711: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703696200 next 29 of size 256
2019-12-23 15:55:03.323267: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703696300 next 30 of size 256
2019-12-23 15:55:03.326718: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703696400 next 31 of size 18944
2019-12-23 15:55:03.331132: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070369AE00 next 32 of size 256
2019-12-23 15:55:03.334594: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070369AF00 next 33 of size 256
2019-12-23 15:55:03.338385: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070369B000 next 34 of size 256
2019-12-23 15:55:03.342615: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070369B100 next 35 of size 147456
2019-12-23 15:55:03.346250: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036BF100 next 36 of size 256
2019-12-23 15:55:03.350587: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036BF200 next 37 of size 256
2019-12-23 15:55:03.354726: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036BF300 next 38 of size 32768
2019-12-23 15:55:03.358855: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C7300 next 39 of size 512
2019-12-23 15:55:03.362387: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C7500 next 40 of size 512
2019-12-23 15:55:03.365897: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C7700 next 41 of size 512
2019-12-23 15:55:03.370651: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C7900 next 42 of size 512
2019-12-23 15:55:03.374030: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C7B00 next 43 of size 512
2019-12-23 15:55:03.378189: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C7D00 next 44 of size 256
2019-12-23 15:55:03.381647: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C7E00 next 45 of size 256
2019-12-23 15:55:03.384917: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C7F00 next 46 of size 256
2019-12-23 15:55:03.389817: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8000 next 47 of size 256
2019-12-23 15:55:03.393262: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8100 next 48 of size 256
2019-12-23 15:55:03.397372: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8200 next 49 of size 256
2019-12-23 15:55:03.400720: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8300 next 50 of size 256
2019-12-23 15:55:03.404964: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8400 next 51 of size 256
2019-12-23 15:55:03.409168: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8500 next 52 of size 256
2019-12-23 15:55:03.412576: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8600 next 53 of size 256
2019-12-23 15:55:03.417357: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8700 next 54 of size 256
2019-12-23 15:55:03.420692: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8800 next 55 of size 256
2019-12-23 15:55:03.426379: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8900 next 56 of size 256
2019-12-23 15:55:03.429848: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8A00 next 57 of size 256
2019-12-23 15:55:03.435081: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8B00 next 58 of size 256
2019-12-23 15:55:03.438870: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8C00 next 59 of size 512
2019-12-23 15:55:03.443106: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C8E00 next 60 of size 512
2019-12-23 15:55:03.446523: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9000 next 61 of size 512
2019-12-23 15:55:03.451070: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9200 next 62 of size 512
2019-12-23 15:55:03.454620: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9400 next 63 of size 512
2019-12-23 15:55:03.458847: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9600 next 64 of size 256
2019-12-23 15:55:03.462621: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9700 next 65 of size 256
2019-12-23 15:55:03.466184: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9800 next 66 of size 256
2019-12-23 15:55:03.470599: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9900 next 67 of size 256
2019-12-23 15:55:03.474181: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9A00 next 68 of size 256
2019-12-23 15:55:03.478221: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9B00 next 69 of size 256
2019-12-23 15:55:03.482325: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9C00 next 70 of size 256
2019-12-23 15:55:03.486454: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9D00 next 71 of size 256
2019-12-23 15:55:03.490066: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9E00 next 72 of size 256
2019-12-23 15:55:03.493746: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036C9F00 next 73 of size 256
2019-12-23 15:55:03.498162: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036CA000 next 74 of size 1280
2019-12-23 15:55:03.501579: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036CA500 next 75 of size 256
2019-12-23 15:55:03.505785: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007036CA600 next 18446744073709551615 of size 219648
2019-12-23 15:55:03.510010: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 2097152
2019-12-23 15:55:03.513823: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703800000 next 77 of size 204800
2019-12-23 15:55:03.517277: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703832000 next 78 of size 227328
2019-12-23 15:55:03.521818: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703869800 next 79 of size 512
2019-12-23 15:55:03.525268: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703869A00 next 80 of size 147456
2019-12-23 15:55:03.529328: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070388DA00 next 81 of size 256
2019-12-23 15:55:03.533531: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070388DB00 next 82 of size 256
2019-12-23 15:55:03.537254: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070388DC00 next 83 of size 512
2019-12-23 15:55:03.541323: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070388DE00 next 84 of size 6144
2019-12-23 15:55:03.544864: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070388F600 next 85 of size 18944
2019-12-23 15:55:03.549147: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703894000 next 86 of size 512
2019-12-23 15:55:03.552947: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703894200 next 87 of size 3584
2019-12-23 15:55:03.556378: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703895000 next 88 of size 512
2019-12-23 15:55:03.561158: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703895200 next 89 of size 256
2019-12-23 15:55:03.564551: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703895300 next 90 of size 18432
2019-12-23 15:55:03.568872: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703899B00 next 91 of size 512
2019-12-23 15:55:03.572483: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703899D00 next 92 of size 512
2019-12-23 15:55:03.576986: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703899F00 next 93 of size 512
2019-12-23 15:55:03.580459: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070389A100 next 94 of size 512
2019-12-23 15:55:03.583930: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070389A300 next 95 of size 32768
2019-12-23 15:55:03.588643: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A2300 next 96 of size 512
2019-12-23 15:55:03.592075: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A2500 next 97 of size 256
2019-12-23 15:55:03.596119: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A2600 next 98 of size 512
2019-12-23 15:55:03.599701: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A2800 next 99 of size 512
2019-12-23 15:55:03.602993: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A2A00 next 100 of size 256
2019-12-23 15:55:03.607754: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A2B00 next 101 of size 512
2019-12-23 15:55:03.611233: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A2D00 next 102 of size 256
2019-12-23 15:55:03.615535: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A2E00 next 103 of size 256
2019-12-23 15:55:03.619511: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A2F00 next 104 of size 256
2019-12-23 15:55:03.623603: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3000 next 105 of size 256
2019-12-23 15:55:03.627092: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3100 next 106 of size 512
2019-12-23 15:55:03.630555: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3300 next 107 of size 256
2019-12-23 15:55:03.634828: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3400 next 108 of size 256
2019-12-23 15:55:03.638848: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3500 next 109 of size 512
2019-12-23 15:55:03.642937: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3700 next 110 of size 512
2019-12-23 15:55:03.646380: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3900 next 111 of size 256
2019-12-23 15:55:03.650475: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3A00 next 112 of size 256
2019-12-23 15:55:03.654036: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3B00 next 113 of size 512
2019-12-23 15:55:03.657806: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3D00 next 114 of size 256
2019-12-23 15:55:03.662041: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3E00 next 115 of size 256
2019-12-23 15:55:03.665886: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A3F00 next 116 of size 256
2019-12-23 15:55:03.670073: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4000 next 117 of size 512
2019-12-23 15:55:03.673576: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4200 next 118 of size 256
2019-12-23 15:55:03.676944: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4300 next 119 of size 256
2019-12-23 15:55:03.681189: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4400 next 120 of size 512
2019-12-23 15:55:03.684591: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4600 next 121 of size 512
2019-12-23 15:55:03.688935: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4800 next 122 of size 256
2019-12-23 15:55:03.692329: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4900 next 123 of size 256
2019-12-23 15:55:03.695737: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4A00 next 124 of size 512
2019-12-23 15:55:03.700272: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4C00 next 125 of size 512
2019-12-23 15:55:03.703679: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4E00 next 126 of size 256
2019-12-23 15:55:03.707751: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A4F00 next 127 of size 256
2019-12-23 15:55:03.711064: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5000 next 128 of size 256
2019-12-23 15:55:03.715780: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5100 next 129 of size 256
2019-12-23 15:55:03.719225: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5200 next 130 of size 512
2019-12-23 15:55:03.722646: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5400 next 131 of size 256
2019-12-23 15:55:03.726832: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5500 next 132 of size 256
2019-12-23 15:55:03.730208: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5600 next 133 of size 256
2019-12-23 15:55:03.734647: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5700 next 134 of size 512
2019-12-23 15:55:03.738104: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5900 next 135 of size 256
2019-12-23 15:55:03.742371: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5A00 next 136 of size 256
2019-12-23 15:55:03.746156: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5B00 next 137 of size 256
2019-12-23 15:55:03.749717: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5C00 next 138 of size 256
2019-12-23 15:55:03.753928: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5D00 next 139 of size 256
2019-12-23 15:55:03.757406: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A5E00 next 140 of size 512
2019-12-23 15:55:03.761853: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6000 next 141 of size 256
2019-12-23 15:55:03.765341: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6100 next 142 of size 256
2019-12-23 15:55:03.769651: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6200 next 143 of size 256
2019-12-23 15:55:03.773187: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6300 next 144 of size 256
2019-12-23 15:55:03.776629: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6400 next 145 of size 256
2019-12-23 15:55:03.781382: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6500 next 146 of size 256
2019-12-23 15:55:03.784928: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6600 next 147 of size 256
2019-12-23 15:55:03.789009: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6700 next 148 of size 256
2019-12-23 15:55:03.793127: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6800 next 149 of size 256
2019-12-23 15:55:03.797284: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6900 next 150 of size 256
2019-12-23 15:55:03.800799: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6A00 next 151 of size 256
2019-12-23 15:55:03.804969: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6B00 next 152 of size 256
2019-12-23 15:55:03.808447: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6C00 next 153 of size 256
2019-12-23 15:55:03.812213: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6D00 next 154 of size 256
2019-12-23 15:55:03.816385: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6E00 next 155 of size 256
2019-12-23 15:55:03.819981: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A6F00 next 156 of size 256
2019-12-23 15:55:03.824627: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7000 next 157 of size 256
2019-12-23 15:55:03.828110: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7100 next 158 of size 256
2019-12-23 15:55:03.832317: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7200 next 159 of size 256
2019-12-23 15:55:03.835825: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7300 next 166 of size 256
2019-12-23 15:55:03.839712: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7400 next 177 of size 256
2019-12-23 15:55:03.843962: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7500 next 172 of size 256
2019-12-23 15:55:03.847500: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7600 next 185 of size 256
2019-12-23 15:55:03.851658: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7700 next 174 of size 256
2019-12-23 15:55:03.855511: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7800 next 161 of size 256
2019-12-23 15:55:03.859561: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7900 next 216 of size 256
2019-12-23 15:55:03.863041: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7A00 next 210 of size 256
2019-12-23 15:55:03.866516: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7B00 next 228 of size 256
2019-12-23 15:55:03.871179: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7C00 next 232 of size 256
2019-12-23 15:55:03.874636: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7D00 next 233 of size 256
2019-12-23 15:55:03.878729: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7E00 next 226 of size 256
2019-12-23 15:55:03.882381: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A7F00 next 229 of size 256
2019-12-23 15:55:03.886829: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8000 next 230 of size 256
2019-12-23 15:55:03.890253: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8100 next 214 of size 512
2019-12-23 15:55:03.893694: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8300 next 163 of size 512
2019-12-23 15:55:03.897987: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8500 next 164 of size 256
2019-12-23 15:55:03.901467: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8600 next 165 of size 256
2019-12-23 15:55:03.905525: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8700 next 181 of size 256
2019-12-23 15:55:03.909268: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8800 next 179 of size 256
2019-12-23 15:55:03.913465: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8900 next 189 of size 256
2019-12-23 15:55:03.917181: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8A00 next 223 of size 256
2019-12-23 15:55:03.920685: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8B00 next 220 of size 256
2019-12-23 15:55:03.925049: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8C00 next 224 of size 256
2019-12-23 15:55:03.928529: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8D00 next 195 of size 256
2019-12-23 15:55:03.932797: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8E00 next 257 of size 256
2019-12-23 15:55:03.936302: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A8F00 next 171 of size 256
2019-12-23 15:55:03.940868: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 00000007038A9000 next 266 of size 256
2019-12-23 15:55:03.944663: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A9100 next 175 of size 256
2019-12-23 15:55:03.947922: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038A9200 next 264 of size 256
2019-12-23 15:55:03.952551: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 00000007038A9300 next 208 of size 7936
2019-12-23 15:55:03.956016: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038AB200 next 215 of size 6144
2019-12-23 15:55:03.960264: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038ACA00 next 231 of size 6144
2019-12-23 15:55:03.963889: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038AE200 next 209 of size 512
2019-12-23 15:55:03.968287: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038AE400 next 218 of size 3584
2019-12-23 15:55:03.971737: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038AF200 next 217 of size 3584
2019-12-23 15:55:03.975156: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 00000007038B0000 next 212 of size 18432
2019-12-23 15:55:03.979327: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038B4800 next 206 of size 18432
2019-12-23 15:55:03.982811: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038B9000 next 213 of size 18432
2019-12-23 15:55:03.987578: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 00000007038BD800 next 211 of size 32768
2019-12-23 15:55:03.991109: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038C5800 next 198 of size 32768
2019-12-23 15:55:03.995759: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038CD800 next 204 of size 32768
2019-12-23 15:55:03.999416: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038D5800 next 168 of size 18944
2019-12-23 15:55:04.002934: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038DA200 next 205 of size 30208
2019-12-23 15:55:04.007278: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007038E1800 next 201 of size 147456
2019-12-23 15:55:04.010826: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703905800 next 199 of size 147456
2019-12-23 15:55:04.015683: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0000000703929800 next 221 of size 213504
2019-12-23 15:55:04.019174: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070395DA00 next 227 of size 196608
2019-12-23 15:55:04.023370: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000070398DA00 next 225 of size 227328
2019-12-23 15:55:04.026986: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 00000007039C5200 next 18446744073709551615 of size 241152
2019-12-23 15:55:04.030930: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 8388608
2019-12-23 15:55:04.035140: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703A00000 next 176 of size 204800
2019-12-23 15:55:04.038806: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000703A32000 next 178 of size 204800
2019-12-23 15:55:04.043366: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0000000703A64000 next 18446744073709551615 of size 7979008
2019-12-23 15:55:04.047441: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 8388608
2019-12-23 15:55:04.051363: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0000000704200000 next 18446744073709551615 of size 8388608
2019-12-23 15:55:04.055332: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 134217728
2019-12-23 15:55:04.058469: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 0000000704A00000 next 18446744073709551615 of size 134217728
2019-12-23 15:55:04.063751: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 536870912
2019-12-23 15:55:04.066880: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 000000070EE00000 next 18446744073709551615 of size 536870912
2019-12-23 15:55:04.072355: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 1563846144
2019-12-23 15:55:04.075439: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 000000072EE00000 next 246 of size 645120000
2019-12-23 15:55:04.079768: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 000000075553C000 next 219 of size 332759040
2019-12-23 15:55:04.083344: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0000000769294000 next 222 of size 147456
2019-12-23 15:55:04.088033: I tensorflow/core/common_runtime/bfc_allocator.cc:800] Free at 00000007692B8000 next 18446744073709551615 of size 585819648
2019-12-23 15:55:04.092059: I tensorflow/core/common_runtime/bfc_allocator.cc:809] Summary of in-use Chunks by size:
2019-12-23 15:55:04.096021: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 127 Chunks of size 256 totalling 31.8KiB
2019-12-23 15:55:04.099440: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 43 Chunks of size 512 totalling 21.5KiB
2019-12-23 15:55:04.102860: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 1280 totalling 1.3KiB
2019-12-23 15:55:04.107029: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 3584 totalling 14.0KiB
2019-12-23 15:55:04.110885: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 6144 totalling 24.0KiB
2019-12-23 15:55:04.114834: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 18432 totalling 72.0KiB
2019-12-23 15:55:04.118602: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 3 Chunks of size 18944 totalling 55.5KiB
2019-12-23 15:55:04.122083: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 30208 totalling 29.5KiB
2019-12-23 15:55:04.126229: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 32768 totalling 128.0KiB
2019-12-23 15:55:04.129866: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 6 Chunks of size 147456 totalling 864.0KiB
2019-12-23 15:55:04.134333: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 196608 totalling 192.0KiB
2019-12-23 15:55:04.138252: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 204800 totalling 800.0KiB
2019-12-23 15:55:04.142353: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 219648 totalling 214.5KiB
2019-12-23 15:55:04.145692: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 3 Chunks of size 227328 totalling 666.0KiB
2019-12-23 15:55:04.149094: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 241152 totalling 235.5KiB
2019-12-23 15:55:04.153582: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 645120000 totalling 615.23MiB
2019-12-23 15:55:04.157423: I tensorflow/core/common_runtime/bfc_allocator.cc:816] Sum Total of in-use chunks: 618.50MiB
2019-12-23 15:55:04.161201: I tensorflow/core/common_runtime/bfc_allocator.cc:818] total_region_allocated_bytes_: 2254857728 memory_limit_: 2254857830 available bytes: 102 curr_region_allocation_bytes_: 2147483648
2019-12-23 15:55:04.169402: I tensorflow/core/common_runtime/bfc_allocator.cc:824] Stats:
Limit: 2254857830
InUse: 648549888
MaxInUse: 2040417280
NumAllocs: 133567
MaxAllocSize: 1083703296

2019-12-23 15:55:04.179973: W tensorflow/core/common_runtime/bfc_allocator.cc:319] _____________________________****************************_______________________________________
2019-12-23 15:55:04.185661: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at transpose_op.cc:199 : Resource exhausted: OOM when allocating tensor with shape[300,140,60,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "C:\Users\super\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\client\session.py", line 1356, in _do_call
return fn(*args)
File "C:\Users\super\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\client\session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "C:\Users\super\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\client\session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: OOM when allocating tensor with shape[300,140,60,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node CNNX/batch_normalization_1/FusedBatchNorm-0-0-TransposeNCHWToNHWC-LayoutOptimizer}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[dense_decoded/_281]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

(1) Resource exhausted: OOM when allocating tensor with shape[300,140,60,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node CNNX/batch_normalization_1/FusedBatchNorm-0-0-TransposeNCHWToNHWC-LayoutOptimizer}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "app.py", line 1403, in training_task
self.current_task.train_process()
File "C:\Users\super\Desktop\captcha_trainer-master\trains.py", line 196, in train_process
feed_dict=val_feed
File "C:\Users\super\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\client\session.py", line 950, in run
run_metadata_ptr)
File "C:\Users\super\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\client\session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\super\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _do_run
run_metadata)
File "C:\Users\super\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow\python\client\session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: OOM when allocating tensor with shape[300,140,60,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node CNNX/batch_normalization_1/FusedBatchNorm-0-0-TransposeNCHWToNHWC-LayoutOptimizer}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

     [[dense_decoded/_281]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

(1) Resource exhausted: OOM when allocating tensor with shape[300,140,60,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node CNNX/batch_normalization_1/FusedBatchNorm-0-0-TransposeNCHWToNHWC-LayoutOptimizer}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

大佬，没看见model_demo.yaml和config_demo.yaml两个文件啊

Traceback (most recent call last):
sdk = muggle_ocr.SDK(model_type=muggle_ocr.ModelType.Captcha, conf_path='myyaml.yaml')
File "/project_path/venv/lib/python3.7/site-packages/muggle_ocr/sdk.py", line 1036, in init
self.predict(init_img)
File "/project_path/venv/lib/python3.7/site-packages/muggle_ocr/sdk.py", line 1045, in predict
result = self.interface.predict_batch(image_batch, None)
File "/project_path/venv/lib/python3.7/site-packages/muggle_ocr/sdk.py", line 826, in predict_batch
output_split
File "/project_path/venv/lib/python3.7/site-packages/muggle_ocr/sdk.py", line 850, in predict_func
expression += self.decode_maps(model.category)[i]
KeyError: 13

保持默认的ALPHANUMERIC可以正常执行

Can plot the accuracy on both train/test set to the tensorboard as well? Thanks.

2019-02-20 15:44:47.246068: W tensorflow/core/util/ctc/ctc_loss_calculator.cc:144] No valid path found.

Could not find a version that satisfies the requirement opencv-python==3.3.0.10 (from -r requirements.txt (line 3)) (from versions: 3.4.2.16, 3.4.2.17, 3.4.3.18, 3.4.4.19, 3.4.5.20, 4.0.0.21)
No matching distribution found for opencv-python==3.3.0.10 (from -r requirements.txt (line 3))

WARNING:tensorflow:ParseError: 1:24 : 'model_checkpoint_path: None': Expected string but found: 'None'
WARNING:tensorflow:./projects/demo-CNNX-NoRecurrent-H64-CrossEntropy-C3/model/checkpoint: Checkpoint ignored
INFO:tensorflow:None
WARNING:tensorflow:ParseError: 1:24 : 'model_checkpoint_path: None': Expected string but found: 'None'
WARNING:tensorflow:./projects/demo-CNNX-NoRecurrent-H64-CrossEntropy-C3/model/checkpoint: Checkpoint ignored
Traceback (most recent call last):
File "/Users/da/code_world/git/captcha_trainer/app.py", line 1332, in training_task
self.current_task.train_process()
File "/Users/da/code_world/git/captcha_trainer/trains.py", line 258, in train_process
self.compile_graph(accuracy)
File "/Users/da/code_world/git/captcha_trainer/trains.py", line 76, in compile_graph
saver.restore(predict_sess, tf.train.latest_checkpoint(self.model_conf.model_root_path))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1274, in restore
raise ValueError("Can't load save_path when it is None.")
ValueError: Can't load save_path when it is None.

mac环境 python3.6
tensorflow 1.14.0
tensorflow-estimator 1.14.0
使用 tk图形界面或者 trains.py启动报同样的错误

when using cpu,what should change?

请问如何准备自己的训练集与测试集，是否需要将验证码中的每个字符单独切割出来？
还是说每张图片可以整体输入，是不是需要将每个图片用验证码字符命名？
请问能否给一个简单的例子？
谢谢

我的图片输入 resize: [200,60] size [200,60]

跑模型的时候提示

x.get_shape() [None, None, 4, 64]

INFO:tensorflow:CNN Output: (?, ?, 256)

怎么可以改成只识别4位验证码的模型

这个可以做OCR识别吗？识别后返回每个字的xy坐标。

大佬，请教一下为什么不设置指数衰减学习率呢？

INFO:tensorflow:Start training...
2019-06-05 10:24:19.195228: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at ctc_loss_op.cc:168 : Invalid argument: Labels length is zero in batch 0
Traceback (most recent call last):
File "/data1/gjl/venv3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
return fn(*args)
File "/data1/gjl/venv3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/data1/gjl/venv3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Labels length is zero in batch 0
[[{{node CTCLoss}} = CTCLoss[ctc_merge_repeated=true, ignore_longer_outputs_than_inputs=false, preprocess_collapse_repeated=false, _device="/job:localhost/replica:0/task:0/device:CPU:0"](output/predict/_89, _arg_labels/indices_0_1, _arg_labels/values_0_3, seq_len/_91)]]
[[{{node gradients/Mean_grad/Shape/_104}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_5999_gradients/Mean_grad/Shape", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "trains.py", line 228, in
tf.app.run()
File "/data1/gjl/venv3.6/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "trains.py", line 221, in main
train_process()
File "trains.py", line 146, in train_process
feed_dict=feed
File "/data1/gjl/venv3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/data1/gjl/venv3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/data1/gjl/venv3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/data1/gjl/venv3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Labels length is zero in batch 0
[[node CTCLoss (defined at /data1/gjl/captcha_trainer/framework.py:119) = CTCLoss[ctc_merge_repeated=true, ignore_longer_outputs_than_inputs=false, preprocess_collapse_repeated=false, _device="/job:localhost/replica:0/task:0/device:CPU:0"](output/predict/_89, _arg_labels/indices_0_1, _arg_labels/values_0_3, seq_len/_91)]]
[[{{node gradients/Mean_grad/Shape/_104}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_5999_gradients/Mean_grad/Shape", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

Caused by op 'CTCLoss', defined at:
File "trains.py", line 228, in
tf.app.run()
File "/data1/gjl/venv3.6/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "trains.py", line 221, in main
train_process()
File "trains.py", line 49, in train_process
model.build_graph()
File "/data1/gjl/captcha_trainer/framework.py", line 33, in build_graph
self._build_train_op()
File "/data1/gjl/captcha_trainer/framework.py", line 119, in _build_train_op
time_major=CTC_LOSS_TIME_MAJOR
File "/data1/gjl/venv3.6/lib/python3.6/site-packages/tensorflow/python/ops/ctc_ops.py", line 158, in ctc_loss
ignore_longer_outputs_than_inputs=ignore_longer_outputs_than_inputs)
File "/data1/gjl/venv3.6/lib/python3.6/site-packages/tensorflow/python/ops/gen_ctc_ops.py", line 286, in ctc_loss
name=name)
File "/data1/gjl/venv3.6/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/data1/gjl/venv3.6/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/data1/gjl/venv3.6/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
op_def=op_def)
File "/data1/gjl/venv3.6/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in init
self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Labels length is zero in batch 0
[[node CTCLoss (defined at /data1/gjl/captcha_trainer/framework.py:119) = CTCLoss[ctc_merge_repeated=true, ignore_longer_outputs_than_inputs=false, preprocess_collapse_repeated=false, _device="/job:localhost/replica:0/task:0/device:CPU:0"](output/predict/_89, _arg_labels/indices_0_1, _arg_labels/values_0_3, seq_len/_91)]]
[[{{node gradients/Mean_grad/Shape/_104}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_5999_gradients/Mean_grad/Shape", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

在运行tutorial.py之后只是生成了model.yaml，而没有config.yaml

win7，app.exe双击打开后，cmd闪退

训练提示cuda版本不匹配 我本机已经配置了tf环境

WARNING:tensorflow:ParseError: 1:24 : 'model_checkpoint_path: None': Expected string but found: 'None'
WARNING:tensorflow:./projects/demo-CNNX-NoRecurrent-H64-CrossEntropy-C3/model/checkpoint: Checkpoint ignored
INFO:tensorflow:None
WARNING:tensorflow:ParseError: 1:24 : 'model_checkpoint_path: None': Expected string but found: 'None'
WARNING:tensorflow:./projects/demo-CNNX-NoRecurrent-H64-CrossEntropy-C3/model/checkpoint: Checkpoint ignored
Traceback (most recent call last):
File "/Users/da/code_world/git/captcha_trainer/app.py", line 1332, in training_task
self.current_task.train_process()
File "/Users/da/code_world/git/captcha_trainer/trains.py", line 258, in train_process
self.compile_graph(accuracy)
File "/Users/da/code_world/git/captcha_trainer/trains.py", line 76, in compile_graph
saver.restore(predict_sess, tf.train.latest_checkpoint(self.model_conf.model_root_path))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1274, in restore
raise ValueError("Can't load save_path when it is None.")
ValueError: Can't load save_path when it is None.

我是个新手，使用本项目在自己生成的一套验证码上达到了 93% 的识别率。结果似乎表明，当答案中包含连续的相同字符（例如 ABCCDE）时，重复的字符特别容易被漏识别（即识别成 ABCDE）。请问是否有什么建议，可能是哪个地方出了问题？

最近在kaggle跑这个，但是kaggle上默认是cuda10，tf2.0，tf降到1.6不兼容cuda10，通过简单粗暴的方法将当前代码转到tf2.0上
import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()
结果还是一堆报错，跑不通，求救。。。

你好，请问该项目是是严格只支持python 3.7吗，还是python 3.5 和3.6也可以支持？
谢谢!

如何选择或者输入??

请问，我预测的时候输出的都是rename是怎么回事啊，求指点

学习4位的数字英文，准确率在80左右就很难提高了，LearningRate特别低怎么解决？

docker compile cpp, docker dev environment, etc.

如图，请问这个是什么原因？

1. 模型的图片输入shape不应该是[batch_size, image_width, image_height, image_channel]吗？但是 core.py 里面的采用ctc loss function 时，模型输入shape却是[None, image_height, image_channel]？是我配置错了吗？
2. 数据从 tfrecords 里面读取出来是一个长串的字节码吧，读取时也没有做任何处理就直接输入模型，不用reshape图片原来大小？或者转换成图片向量？
3. 现在无法执行训练了，一直报错：(0) Invalid argument: Not enough time for target transition sequence (required: 6, available: 5)11You can turn this error into a warning by using the flag ignore_longer_outputs_than_inputs

感谢分享

因为好多玩家的电脑可能没有GPU，没有GPU就要自己去安装一大堆软件包才能开始训练，太麻烦了，谢谢！

你好，kerlomz，我使用您的代码跑的时候，framework.py中的类GraphOCR中的 self.dense_decoded = tf.sparse.to_dense(self.decoded[0], default_value=-1, name="dense_decoded")这行代码报错，AttributeError: module 'tensorflow.sparse' has no attribute 'to_dense'。请问您确定在TF_GPU 1.10.0这个版本可以运行您的程序？
谢谢~

感谢分享，现在这项目已经跑了28000步了，acc大概在80左右，现在是要等他达到配置的准确率才会停吗？

没有测试过MacOS还是测试不通过？
现在在这个平台搭建一个用于处理验证码

哪里有问题吗

W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 538.33MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.

W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.02GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.

E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 378.60M (396990720 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY

E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 378.60M (396990720 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY

INFO:tensorflow:Step: 9200 Time: 13.979 sec/batch, Cost = 0.00025, 160x60-BatchSize: 128
INFO:tensorflow:Step: 9300 Time: 13.981 sec/batch, Cost = 0.00025, 160x60-BatchSize: 128
INFO:tensorflow:Step: 9400 Time: 13.941 sec/batch, Cost = 0.00025, 160x60-BatchSize: 128
INFO:tensorflow:Step: 9500 Time: 14.033 sec/batch, Cost = 0.00024, 160x60-BatchSize: 128
INFO:tensorflow:Step: 9600 Time: 14.009 sec/batch, Cost = 0.00024, 160x60-BatchSize: 128
INFO:tensorflow:Step: 9700 Time: 13.956 sec/batch, Cost = 0.00024, 160x60-BatchSize: 128
INFO:tensorflow:Step: 9800 Time: 14.029 sec/batch, Cost = 0.00023, 160x60-BatchSize: 128
INFO:tensorflow:Step: 9900 Time: 14.055 sec/batch, Cost = 0.00023, 160x60-BatchSize: 128
INFO:tensorflow:Step: 10000 Time: 14.234 sec/batch, Cost = 0.00023, 160x60-BatchSize: 128
INFO:tensorflow:0 7 7 [13, 11, 26, 30, 13, 18, 11] [13, 11, 26, 30, 13, 18, 11] --> ['c', 'a', 'p', 't', 'c', 'h', 'a'] ['c', 'a', 'p', 't', 'c', 'h', 'a']
INFO:tensorflow:1 7 7 [13, 11, 26, 30, 13, 18, 11] [13, 11, 26, 30, 13, 18, 11] --> ['c', 'a', 'p', 't', 'c', 'h', 'a'] ['c', 'a', 'p', 't', 'c', 'h', 'a']
INFO:tensorflow:2 7 7 [13, 11, 26, 30, 13, 18, 11] [13, 11, 26, 30, 13, 18, 11] --> ['c', 'a', 'p', 't', 'c', 'h', 'a'] ['c', 'a', 'p', 't', 'c', 'h', 'a']
INFO:tensorflow:3 7 7 [13, 11, 26, 30, 13, 18, 11] [13, 11, 26, 30, 13, 18, 11] --> ['c', 'a', 'p', 't', 'c', 'h', 'a'] ['c', 'a', 'p', 't', 'c', 'h', 'a']
INFO:tensorflow:4 7 7 [13, 11, 26, 30, 13, 18, 11] [13, 11, 26, 30, 13, 18, 11] --> ['c', 'a', 'p', 't', 'c', 'h', 'a'] ['c', 'a', 'p', 't', 'c', 'h', 'a']
ERROR:tensorflow:[]
INFO:tensorflow:Epoch: 1358, Step: 10000, Accuracy = 1.0000, Cost = 0.00023, Time = 15.367 sec/batch, LearningRate: 0.0009800000116229057
INFO:tensorflow:CNN Output: (?, ?, 256)
INFO:tensorflow:./model/myfirsttest.model-10000
INFO:tensorflow:Restoring parameters from ./model/myfirsttest.model-10000
INFO:tensorflow:Froze 40 variables.
Converted 40 variables to const ops.
INFO:tensorflow:Total Time: 33.93611812591553 sec.
INFO:tensorflow:Training completed.

执行 app.py ，会要求设置 sample source
这个应该就是要设置样本的路径吧，也就是作者说的喂数据
SourcePath:
Training: {SourceTrainPath}
Validation: {SourceValidationPath}
但这个文件/目录格式是怎么样的，不太懂。
比如我有图片+正确的码，怎么存放，才可以作为样本，从而让程序跑起来呢

每隔500次系统卡顿一次,采用cpu训练

2019-04-05 16:54:36.524458: W tensorflow/core/framework/allocator.cc:124] Allocation of 614400000 exceeds 10% of system memory.
2019-04-05 16:55:43.043568: W tensorflow/core/framework/allocator.cc:124] Allocation of 614400000 exceeds 10% of system memory.
2019-04-05 16:55:50.748391: W tensorflow/core/framework/allocator.cc:124] Allocation of 614400000 exceeds 10% of system memory.
0 4 4 [1, 5, 2, 2] [1, 5, 2, 2]
1 4 4 [5, 6, 4, 5] [5, 6, 4, 5]
2 4 4 [9, 9, 10, 7] [9, 9, 10, 7]
3 4 4 [6, 2, 4, 1] [6, 2, 4, 1]
4 4 4 [1, 6, 10, 2] [1, 6, 10, 2]
Epoch: 3, Step: 1500, Accuracy = 0.973, Cost = 0.1232, Time = 89.143, LearningRate: 0.009999999776482582, LastBatchError: 0.006666666828095913

后续新的训练集怎么打包进去，和原有的训练集合起来重新打包训练吗

你好，我在TensorFlow这是个新手，我按照您给的教程，TestSetNum设置为10w之后，运行了trains.py，我看到在进行到1500步的时候，停下来提示训练完成，然后我按照部署教程部署，post之后永远是结果8，换图片都是这样。
请问训练用的验证码图片放在哪？
我的操作是否有误。

如上，多谢

Windows 10, Python 3.6.7 64-bit, CUDA 9.0, tensorflow-gpu 1.11, CNN 分支(因为 LSTM 对于我的训练数据而言似乎效果不好)
内存 8GB, 显存 6GB.
训练数据集是 5 万张 240x60 大小的 JPG 图片.

一般是执行 python .\trains.py 一段时间后就报错退出.报错信息有这三种情况.
1

Traceback (most recent call last):
  File ".\trains.py", line 185, in <module>
    train_process()
  File ".\trains.py", line 79, in train_process
    batch_x, batch_y = get_next_batch(64, False)  # 64
  File ".\trains.py", line 115, in get_next_batch
    batch_x = np.zeros([batch_size, IMAGE_HEIGHT * IMAGE_WIDTH])
MemoryError

2
几屏幕的 CUDA 和 tensorflow 的报错信息,大概说的是内存分配失败 (原文是 Memory, 指的是内存还是显存?), 就像这样:

2018-11-04 01:54:01.061494: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.00G (2147483648 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory

3
没有报错信息,执行到这里之后就直接退出了:

Session Initializing...
Loading history archive...
Initialized.
---------------------------------------------------------------------------------

但是我在任务管理器中看到,在报错退出之前,内存和显存的占用率都没有占满.这是怎么回事?应该怎么修复呢?