To analyze: 464.h264ref OOBs under ASan on Windows about address-sanitizer HOT 14 CLOSED

FTR, the stack looks similar to the global buffer overflow mentioned on the 
FoundBugs wiki page; but the top frames are slightly different.

Also, the report in this issue is marked as heap overflow, not global.

This is by no means an asan bug or feature request. (or is it?)
Please don't pollute our bug tracker. 

asan prints "heap-buffer-overflow" because it checks the shadow bytes, and they 
clearly indicate that this is a heap bug, has nothing to do with globals. 
Shadow byte and word:
  0x205cc9bc: fa
  0x205cc9bc: fa fa fa fa

> This is by no means an asan bug or feature request. (or is it?)
It is a task, related to asan.
Reopening until we/I have understanding what's going on.

Re-ran with `set ASAN_OPTIONS="redzone=256"` on Windows -> got the same stack 
and right-OOB report:
0x02b44e64 is located 260 bytes to the right of 3168-byte region 
[0x02b44100,0x02b44d60)

Re-ran with `set ASAN_OPTIONS="redzone=512"` and got a left-OOB report:
0x02d96f64 is located 668 bytes to the left of 1584-byte region 
[0x02d97200,0x02d97830)
allocated by thread T0 here:
  #0 0x5b6e85  calloc c:\src\llvm\projects\compiler-rt\lib\asan\asan_malloc_win.cc:52
  #1 0x4d57ec  get_mem2D+0x0x0000008c
  #2 0x4c8224  alloc_colocated+0x0x00000114
  #3 0x50f304  GenerateSequenceParameterSet+0x0x00000914
  #4 0x50df93  GenerateParameterSets+0x0x00000023
  #5 0x46de87  main+0x0x00000327

FTR, the "fa fa fa fa" shadow bytes are kAsanHeapLeftRedzoneMagic.
Not clear why the first two reports say "right-OOB"

with redzone=2048 the allocation stack is different:
0x04ea3564 is located 668 bytes to the left of 260-byte region 
[0x04ea3800,0x04ea3904)
allocated by thread T0 here:
  #0 0x5b6e85  calloc c:\src\llvm\projects\compiler-rt\lib\asan\asan_malloc_win.cc:52
  #1 0x473d7f  get_mem_DCcoeff+0x0x0000031f
  #2 0x470fa0  init_img+0x0x00000730
  #3 0x46de8c  main+0x0x0000032c
  #4 0x5c509d  __tmainCRTStartup f:\dd\vctools\crt_bld\self_x86\crt\src\crt0.c:266
  #5 0x76943677  BaseThreadInitThunk+0x0x00000012
  #6 0x77009f42  RtlInitializeExceptionChain+0x0x00000063

Can't reproduce on linux, even with redzone=512
Ok, there is a chance this is an asan/win bug, reopening.

Iterestingly, this doesn't repro with -O0 on Windows too...

FTR, clang -O2 (without asan) also fails this test:
*** Miscompare of foreman_train_baseline_encodelog.out; for details see
    C:/src/spec2006/benchspec/CPU2006/464.h264ref/run/run_base_train_none.0001/foreman_train_baseline_encodelog.out.mis
0015:  0000(IDR)   21952 0 28  37.383  41.260  42.850        0       0     FRM  
  99
       0000(IDR)    6808 0 28  -1.#IO  40.796  42.637        0       0     FRM    99
                       ^
0016:  0001(P)      3024 0 28  36.868  41.036  42.720        0       0     FRM  
   0
       0001(P)      5360 0 28  -1.#IO  40.027  41.179        0       0     FRM    99
                       ^
0017:  0002(P)      4120 0 28  36.875  40.936  42.683        0       0     FRM  
   2
       0002(P)      5424 0 28  -1.#IO  40.024  40.980        0       0     FRM    99
                       ^
0018:  0003(P)      4552 0 28  36.851  40.950  42.670        0       0     FRM  
   2
       0003(P)      5520 0 28  -1.#IO  40.026  40.936        0       0     FRM    99
                       ^
0019:  0004(P)      4952 0 28  36.848  40.815  42.407        0       0     FRM  
   2
       0004(P)      5408 0 28  -1.#IO  39.977  41.178        0       0     FRM    99
                       ^
0020:  0005(P)      4448 0 28  36.775  40.691  42.386        0       0     FRM  
   1
       0005(P)      5504 0 28  -1.#IO  39.938  41.115        0       0     FRM    99
                       ^
0021:  0006(P)      3944 0 28  36.689  40.740  42.206        0       0     FRM  
   1
       0006(P)      5480 0 28  -1.#IO  39.815  41.027        0       0     FRM    99
                       ^
0022:  0007(P)      4040 0 28  36.698  40.821  42.262        0       0     FRM  
   0
       0007(P)      5552 0 28  -1.#IO  39.820  41.052        0       0     FRM    99
                       ^
0023:  0008(P)      4104 0 28  36.751  40.824  42.114        0       0     FRM  
   0
       0008(P)      5528 0 28  -1.#IO  39.863  40.818        0       0     FRM    99
                       ^
0024:  0009(P)      4576 0 28  36.724  40.773  41.954        0       0     FRM  
   1
       0009(P)      5640 0 28  -1.#IO  40.015  40.742        0       0     FRM    99

and it passes with clang -O3...

Now that we have debug info and fixed quite a few ABI issues in Clang, I think 
it's time to revisit.

Here's a fresh report with a symbolized stacktrace:
=================================================================
==5056==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x00c3e990 at 
pc 0xf5efc7 sp 0xc3e8d4
READ of size 4 at 0x00c3e990 thread T0
    #0 0xf5efc6 in SATD  mv-search.c:1093
    #1 0xf6209f in SubPelBlockMotionSearch  mv-search.c:1398
    #2 0xf78e79 in BlockMotionSearch  mv-search.c:2672
    #3 0xf80ab2 in PartitionMotionSearch  mv-search.c:3272
    #4 0xfd0aef in encode_one_macroblock  rdopt.c:3096
    #5 0xff319c in encode_one_slice  slice.c:253
    #6 0xed074e in code_a_picture  image.c:236
    #7 0xed7683 in frame_picture  image.c:798
    #8 0xed1ca7 in encode_one_frame  image.c:409
    #9 0xef12c9 in main  lencod.c:413

Address 0x00c3e990 is located in stack of thread T0 at offset 80 in frame
    #0 0xf5e6ef in SATD  mv-search.c:1018

  This frame has 1 object(s):
    [16, 80) 'd' <== Memory access at offset 80 overflows this variable
HINT: this may be a false positive if your program uses some custom stack 
unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow  mv-search.c:1093 SATD
Shadow bytes around the buggy address:
  0x20187ce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x20187cf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x20187d00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x20187d10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x20187d20: 00 00 00 00 00 00 00 00 f1 f1 00 00 00 00 00 00
=>0x20187d30: 00 00[f3]f3 f3 f3 00 00 00 00 00 00 00 00 00 00
  0x20187d40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x20187d50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x20187d60: 00 00 00 00 f1 f1 00 00 00 00 00 00 00 00 00 00
  0x20187d70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x20187d80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Aha, this is an obvious bug which is already documented in 
FoundBugs#Spec_CPU_2006

ASan WAI

Adding Project:AddressSanitizer as part of GitHub migration.