Linux 4.15> does not allow mmap with a large address about palloc HOT 3 OPEN

Thanks for the update.

We are currently testing a new detector that uses the safer pagemap interface instead of using /dev/mem. The new detector can be found in the repository: mc-mapping-pagemap.c.

The following is the result of the new detector on the Nehalem platform we used in the original PALLOC paper, which clearly shows bit 12,13,19,20 are used for the mapping. We plan to remove the old /dev/mem version in the future.

$ sudo chrt -f 1 ./mc-mapping-pagemap -p 0.7 -n 3
mem_size (MB): 2756
allocation complete.
worker thread begins
worker thread begins
worker thread begins
Bit6: 299.67 MB/s, 213.57 ns
Bit7: 307.61 MB/s, 208.05 ns
Bit8: 295.97 MB/s, 216.24 ns
Bit9: 297.84 MB/s, 214.88 ns
Bit10: 300.66 MB/s, 212.86 ns
Bit11: 245.89 MB/s, 260.28 ns
Bit12: 792.58 MB/s, 80.75 ns
Bit13: 789.23 MB/s, 81.09 ns
Bit14: 296.21 MB/s, 216.06 ns
Bit15: 294.19 MB/s, 217.55 ns
Bit16: 240.98 MB/s, 265.58 ns
Bit17: 294.20 MB/s, 217.54 ns
Bit18: 294.07 MB/s, 217.64 ns
Bit19: 789.05 MB/s, 81.11 ns
Bit20: 789.15 MB/s, 81.10 ns
Bit21: 294.17 MB/s, 217.56 ns
Bit22: 240.98 MB/s, 265.58 ns
Bit23: 294.10 MB/s, 217.62 ns

from palloc.

Hi, thanks for the information.

the new detector does not seem working well in my environment. I can provide further information if it would help improving the detector. The machine has only 1 DIMM and I set g_cache_num_ways to match the actual associativity. Note that the machine is different from the one I used in #9.

I guess memory accesses generated by mc-mapping-pagemap are served by the cache as their latency is reasonably shorter than the one measured by Intel Memory Latency Checker (65.1 ns).

$ cat /proc/cpuinfo | grep "model name" | uniq
model name	: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz

$ cat /sys/devices/system/cpu/cpu0/cache/index3/ways_of_associativity 
20

$  sudo chrt -f 1 ./mc-mapping-pagemap -p 0.7 -n 3
mem_size (MB): 5518
allocation complete.
worker thread begins
worker thread begins
worker thread begins
Bit6: 1228.09 MB/s, 52.11 ns
Bit7: 1441.79 MB/s, 44.39 ns
Bit8: 1619.70 MB/s, 39.51 ns
Bit9: 1660.54 MB/s, 38.54 ns
Bit10: 1663.63 MB/s, 38.47 ns
Bit11: 1647.35 MB/s, 38.85 ns
Bit12: 1726.24 MB/s, 37.07 ns
Bit13: 1709.34 MB/s, 37.44 ns
Bit14: 1902.05 MB/s, 33.65 ns
Bit15: 1876.14 MB/s, 34.11 ns
Bit16: 1936.24 MB/s, 33.05 ns
Bit17: 1901.17 MB/s, 33.66 ns
Bit18: 1714.88 MB/s, 37.32 ns
Bit19: 1699.40 MB/s, 37.66 ns
Bit20: 1540.65 MB/s, 41.54 ns
Bit21: 1557.51 MB/s, 41.09 ns 
Bit22: 1924.62 MB/s, 33.25 ns
Bit23: 1703.65 MB/s, 37.57 ns

$ sudo ./mlc
Intel(R) Memory Latency Checker - v3.1a
Measuring idle latencies (in ns)...
	Memory node
Socket	     0	
     0	  65.1

from palloc.

For what it's worth, here are my results on an i7-6700HQ with two R2x8 DDR4 DIMMs (each with 16 banks and 4 bank groups):

$ sudo chrt -f 1 ./mc-mapping-pagemap -p 0.7 -n 3
mem_size (MB): 11136
allocation complete.
worker thread begins
worker thread begins
worker thread begins
Bit6: 4671.85 MB/s, 13.70 ns
Bit7: 4818.67 MB/s, 13.28 ns
Bit8: 4750.99 MB/s, 13.47 ns
Bit9: 4751.69 MB/s, 13.47 ns
Bit10: 4735.04 MB/s, 13.52 ns
Bit11: 4818.97 MB/s, 13.28 ns
Bit12: 4729.72 MB/s, 13.53 ns
Bit13: 4817.95 MB/s, 13.28 ns
Bit14: 4728.85 MB/s, 13.53 ns
Bit15: 4817.50 MB/s, 13.28 ns
Bit16: 1966.58 MB/s, 32.54 ns
Bit17: 4095.22 MB/s, 15.63 ns
Bit18: 3948.74 MB/s, 16.21 ns
Bit19: 3857.12 MB/s, 16.59 ns
Bit20: 3908.53 MB/s, 16.37 ns
Bit21: 1059.46 MB/s, 60.41 ns
Bit22: 918.14 MB/s, 69.71 ns
Bit23: 525.82 MB/s, 121.72 ns

I'm not sure how to interpret the results: the latencies for almost all bits seem way to low to reflect misses to main memory. Note that Skylake has a hashed L3, so you can't rely on simple ways/sets logic to force misses by overloading a way: you would need to know the hashing function (it has been reverse engineered elsewhere).

from palloc.