[act] reduce memory footprint about robrank HOT 5 CLOSED

before optimization: 2080Ti
cub:rres18:ptripletN memory consumption: 7165MB each GPU
cub:rres18p:ptripletN: 9101MB.

after optimization:
cub:rres18p:ptripletN: 7181MB each GPU.

from robrank.

before optimization: 2080Ti*3 (distributed data parallel)
mnist:rc2f2:ptripletN: 2067MB each GPU.
mnist:rc2f2d:ptripletN: 2067MB each GPU. (my code is already memory-efficient enough)
mnist:rc2f2p:ptripletN: 2181MB each GPU.

after optimization: 2067MB each GPU.

from robrank.

• performance sanity: mnist:rc2f2p:ptripletN after memory footprint optimization.
• training speed: likely faster.

from robrank.

after optimization:
cub:rres50:ptripletN 13569 MB
cub:rres50d:ptripletN 13569 MB
cub:rres50p:ptripletN 16483 MB (much better; before ~23000MB)

from robrank.

looks good.

from robrank.