rrze-hpc / osaca Goto Github PK
View Code? Open in Web Editor NEWOpen Source Architecture Code Analyzer
License: GNU Affero General Public License v3.0
Open Source Architecture Code Analyzer
License: GNU Affero General Public License v3.0
The following benchmark leads to a TypeError on ICL. It works on other targets.
echo 'movb $0x1,0x47(%rsp)' | osaca --arch ICL -
Output:
Traceback (most recent call last):
File "/usr/local/bin/osaca", line 11, in <module>
load_entry_point('osaca==0.3.14', 'console_scripts', 'osaca')()
File "/usr/local/lib/python3.6/dist-packages/osaca/osaca.py", line 387, in main
run(args, output_file=args.out)
File "/usr/local/lib/python3.6/dist-packages/osaca/osaca.py", line 339, in run
inspect(args, output_file=output_file)
File "/usr/local/lib/python3.6/dist-packages/osaca/osaca.py", line 293, in inspect
semantics.add_semantics(kernel)
File "/usr/local/lib/python3.6/dist-packages/osaca/semantics/arch_semantics.py", line 31, in add_semantics
self.assign_tp_lt(instruction_form)
File "/usr/local/lib/python3.6/dist-packages/osaca/semantics/arch_semantics.py", line 283, in assign_tp_lt
else 0
TypeError: unsupported operand type(s) for +=: 'NoneType' and 'int'
OSACA is currently checking if a binary is an elf64 binary in osaca.check_elffile()
. Is that really necessary? What are the elf64 specifics that we rely on or could this restriction be lifted? (note that my mac produces Mach-O binaries and objdump is able to decode those)
I would like to run OSACA standalone, i.e., without first installing it - so I can keep my distribution python clean and so I can try changes quickly. I assume during development that's what you do? Is it possibly/easy?
E.g. nopw %cs:(%rax,%rax)
.
Was trying OSACA for some code snippets from MKL lib, but its crashing on me.
Traceback (most recent call last):
File "/home/jd/.local/lib/python3.6/site-packages/osaca/osaca.py", line 270, in inspect
parsed_code = parser.parse_file(code)
File "/home/jd/.local/lib/python3.6/site-packages/osaca/parser/base_parser.py", line 57, in parse_file
asm_instructions.append(self.parse_line(line, i + 1 + start_line))
File "/home/jd/.local/lib/python3.6/site-packages/osaca/parser/parser_x86att.py", line 258, in parse_line
'Could not parse instruction on line {}: {!r}'.format(line_number, line)
ValueError: Could not parse instruction on line 6: 'vbroadcastsd -0x10(%rbp),%ymm3'
Here the asm fragment:
# OSACA-BEGIN
.L19:
vfmadd231pd %ymm0,%ymm3,%ymm5
vfmadd231pd %ymm1,%ymm3,%ymm9
vfmadd231pd %ymm2,%ymm3,%ymm13
vbroadcastsd -0x10(%rbp),%ymm3
vfmadd231pd %ymm0,%ymm3,%ymm6
vfmadd231pd %ymm1,%ymm3,%ymm10
vfmadd231pd %ymm2,%ymm3,%ymm14
vbroadcastsd -0x8(%rbp),%ymm3
sub $0xffffffffffffff80,%rbp
vfmadd231pd %ymm0,%ymm3,%ymm7
vmovupd 0x100(%rbx),%ymm0
vfmadd231pd %ymm1,%ymm3,%ymm11
vmovupd 0x120(%rbx),%ymm1
vfmadd231pd %ymm2,%ymm3,%ymm15
vmovupd 0x140(%rbx),%ymm2
sub $0xfffffffffffffe80,%rbx
vbroadcastsd -0x80(%rbp),%ymm3
sub $0x1,%rax
jne .L19
# OSACA-END
Installing via pip fails (on Windows and Mac):
c:\Python37>scripts\pip install osaca
Collecting osaca
Using cached https://files.pythonhosted.org/packages/88/a5/2a81436fc6321f057e6d6a70369d0847d6c6ba48b9d70cbc16920846b288/osaca-0.2.0-py3-none-any.whl
Collecting pandas (from osaca)
Using cached https://files.pythonhosted.org/packages/26/fc/d0509d445d2724fbc5f9c9a6fc9ce7da794873469739b6c94afc166ac2a2/pandas-0.23.4-cp37-cp37m-win32.whl
Collecting kerncraft (from osaca)
Using cached https://files.pythonhosted.org/packages/0a/d0/7bead12116240d698e58ac54349f0a45c645fe3d9b8cc88ab0121b69da00/kerncraft-0.6.10.tar.gz
Collecting numpy (from osaca)
Using cached https://files.pythonhosted.org/packages/2d/d4/ddfa53d27e2cdfef261b744df6e4b68134fc69b0d45ad8211e560178c852/numpy-1.15.1-cp37-none-win32.whl
Collecting pytz>=2011k (from pandas->osaca)
Using cached https://files.pythonhosted.org/packages/30/4e/27c34b62430286c6d59177a0842ed90dc789ce5d1ed740887653b898779a/pytz-2018.5-py2.py3-none-any.whl
Collecting python-dateutil>=2.5.0 (from pandas->osaca)
Using cached https://files.pythonhosted.org/packages/cf/f5/af2b09c957ace60dcfac112b669c45c8c97e32f94aa8b56da4c6d1682825/python_dateutil-2.7.3-py2.py3-none-any.whl
Collecting ruamel.yaml<0.14.0,>=0.13.4 (from kerncraft->osaca)
Using cached https://files.pythonhosted.org/packages/6d/00/061d150591d34065748857580735c4b42879a7dd3ebb75dde23671b60c26/ruamel.yaml-0.13.14.tar.gz
Complete output from command python setup.py egg_info:
Warning: 'keywords' should be a list, got type 'NoneType'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Egon\AppData\Local\Temp\pip-install-rn5n_nk5\ruamel.yaml\setup.py", line 854, in <module>
main()
File "C:\Users\Egon\AppData\Local\Temp\pip-install-rn5n_nk5\ruamel.yaml\setup.py", line 843, in main
setup(**kw)
File "c:\python37\lib\site-packages\setuptools\__init__.py", line 129, in setup
return distutils.core.setup(**attrs)
File "c:\python37\lib\distutils\core.py", line 108, in setup
_setup_distribution = dist = klass(attrs)
File "c:\python37\lib\site-packages\setuptools\dist.py", line 372, in __init__
_Distribution.__init__(self, attrs)
File "c:\python37\lib\distutils\dist.py", line 267, in __init__
getattr(self.metadata, "set_" + key)(val)
File "c:\python37\lib\distutils\dist.py", line 1203, in set_keywords
self.keywords = _ensure_list(value, 'keywords')
File "c:\python37\lib\distutils\dist.py", line 40, in _ensure_list
value = list(value)
TypeError: 'NoneType' object is not iterable
Either use relative imports (e.g., import .param
) or absolute imports (e.g., from osaca import param
). The first will break script usage and the second will require the package to be installed.
See https://stackoverflow.com/a/14132912/2754040 for an explanation.
This can be checked using tox, currently it fails with:
$ tox
GLOB sdist-make: ~/Development/OSACA/setup.py
py35 inst-nodeps: ~/Development/OSACA/.tox/dist/osaca-0.1.zip
py35 installed: kerncraft==0.5.0,mpmath==1.0.0,numpy==1.13.3,osaca==0.1,pandas==0.20.3,pycachesim==0.1.5,pylru==1.0.9,python-dateutil==2.6.1,pytz==2017.2,ruamel.yaml==0.13.14,six==1.11.0,sympy==1.1.1
py35 runtests: PYTHONHASHSEED='2543723380'
py35 runtests: commands[0] | osaca
Traceback (most recent call last):
File "~/Development/OSACA/.tox/py35/bin/osaca", line 11, in <module>
load_entry_point('osaca==0.1', 'console_scripts', 'osaca')()
File "~/Development/OSACA/.tox/py35/lib/python3.5/site-packages/pkg_resources/__init__.py", line 570, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "~/Development/OSACA/.tox/py35/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2751, in load_entry_point
return ep.load()
File "~/Development/OSACA/.tox/py35/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2405, in load
return self.resolve()
File "~/Development/OSACA/.tox/py35/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2411, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "~/Development/OSACA/.tox/py35/lib/python3.5/site-packages/osaca/osaca.py", line 9, in <module>
from param import Register, MemAddr, Parameter
ImportError: No module named 'param'
ERROR: InvocationError: '~/Development/OSACA/.tox/py35/bin/osaca'
______________________________________________________________ summary ______________________________________________________________
ERROR: py35: commands failed
$
Also, there are style guides on imports in PEP8: https://www.python.org/dev/peps/pep-0008/#imports :)
working on a Windows 10 pc with Clang 7.0.1 installed the example taxCals.c does not work:
the output given is
Blocks found in assembly file:
block | OPs | pck. | AVX || Registers | ZMM | YMM | XMM | GP ||ptr.inc|
-----------+-----+------+-----++-----------+----------+----------+----------+---------++-------|
0 .LBB4_1 | 31 | 0 | 0 || 19 ( 4) | 0 ( 0) | 0 ( 0) | 6 ( 2) | 13 ( 2) || None |
Choose block to be marked [0]: 0
Traceback (most recent call last):
File "...\virtual_env\Scripts\osaca-script.py", line 11, in <module>
load_entry_point('osaca==0.2.1', 'console_scripts', 'osaca')()
File "...\virtual_env\lib\site-packages\osaca\osaca.py", line 814, in main
osaca = OSACA(args.arch, assembly)
File "...\virtual_env\lib\site-packages\osaca\osaca.py", line 381, in __init__
assembly = extract_marked_section(assembly)
File "...\virtual_env\lib\site-packages\osaca\osaca.py", line 294, in extract_marked_section
raise ValueError("Could not find start and end markers.")
ValueError: Could not find start and end markers.
The following benchmark leads to a KeyError on HSW. It works on other targets.
echo 'xor 0x12(%rsi),%al' | osaca --arch HSW -
Output:
Traceback (most recent call last):
File "/usr/local/bin/osaca", line 11, in <module>
load_entry_point('osaca==0.3.14', 'console_scripts', 'osaca')()
File "/usr/local/lib/python3.6/dist-packages/osaca/osaca.py", line 387, in main
run(args, output_file=args.out)
File "/usr/local/lib/python3.6/dist-packages/osaca/osaca.py", line 339, in run
inspect(args, output_file=output_file)
File "/usr/local/lib/python3.6/dist-packages/osaca/osaca.py", line 293, in inspect
semantics.add_semantics(kernel)
File "/usr/local/lib/python3.6/dist-packages/osaca/semantics/arch_semantics.py", line 31, in add_semantics
self.assign_tp_lt(instruction_form)
File "/usr/local/lib/python3.6/dist-packages/osaca/semantics/arch_semantics.py", line 215, in assign_tp_lt
operands.index(self._create_reg_wildcard())
File "/usr/local/lib/python3.6/dist-packages/osaca/parser/parser_x86att.py", line 445, in get_reg_type
if self.is_gpr(register):
File "/usr/local/lib/python3.6/dist-packages/osaca/parser/parser_x86att.py", line 429, in is_gpr
if self.is_basic_gpr(register):
File "/usr/local/lib/python3.6/dist-packages/osaca/parser/parser_x86att.py", line 419, in is_basic_gpr
any(char.isdigit() for char in register['name'])
File "/home/andreas/.local/lib/python3.6/site-packages/ruamel/yaml/comments.py", line 753, in __getitem__
return ordereddict.__getitem__(self, key)
KeyError: 'name'
Apologies if this is specific to the Compiler Explorer integration, but the simple input:
square(int): // @square(int)
sub sp, sp, #16 // =16
str w0, [sp, #12]
ldr w8, [sp, #12]
ldr w9, [sp, #12]
mul w0, w8, w9
add sp, sp, #16 // =16
ret
Yields this error:
*-*-*-*-*-*-*-*-*-*-
8: .file 1 "/home/ce" "./example.cpp"
*-*-*-*-*-*-*-*-*-*-
Traceback (most recent call last):
File "/opt/compiler-explorer/osaca-0.3.10/lib/python3.8/site-packages/osaca/osaca.py", line 262, in inspect
parsed_code = parser.parse_file(code)
File "/opt/compiler-explorer/osaca-0.3.10/lib/python3.8/site-packages/osaca/parser/base_parser.py", line 54, in parse_file
asm_instructions.append(self.parse_line(line, i + 1 + start_line))
File "/opt/compiler-explorer/osaca-0.3.10/lib/python3.8/site-packages/osaca/parser/parser_AArch64.py", line 286, in parse_line
instruction_form[self.INSTRUCTION_ID] = result[self.INSTRUCTION_ID]
TypeError: 'NoneType' object is not subscriptable
which seems to be an unrecognized instruction? c.f. compiler-explorer/compiler-explorer#2319
osaca --arch ivb --iaca osaca/taxCalc/ivb-iaca
returns 0 if file does not exists
I would like to use the OSACA.inspect_with_iaca()
or a similar high-level function as an API, but this is currently not very usable due to two reasons:
the result is a string, which is directly output to stdout.
I would need a machine readable output, which is separate from the output formatting.
running this function multiple times multiplies the output length
there should not be an internal state that breaks if this is run several times. either generate it always on-the-fly or overwrite the internal state on regeneration.
Either implement own table generation code and separate it from everything else or make use of already existing packages.
The intertwinedness of computation, result formatting and outputting is currently not very helpful.
When OSACA tries to understand an instruction it doesn't recognize, it would be great to report an error instead of:
File "/opt/compiler-explorer/osaca-0.3.10/lib/python3.8/site-packages/osaca/parser/parser_AArch64.py", line 286, in parse_line
instruction_form[self.INSTRUCTION_ID] = result[self.INSTRUCTION_ID]
TypeError: 'NoneType' object is not subscriptable
setup.py currently has #!/usr/bin/python
as shebang and therefore fails on systems where /usr/bin/python is non existent or of version 2. The shebang should either be #!/usr/bin/env python
or #!/usr/bin/env python3
to be sure. Other files might also be affected.
On a number of benchmarks, OSACA fails on ICL with KeyError: 'imd'
. With other target architectures, the error does not occur.
Example: echo 'cmpb $0x0,0x42(%rsi)' | osaca --arch ICL -
Output:
Traceback (most recent call last):
File "/usr/local/bin/osaca", line 11, in <module>
load_entry_point('osaca==0.3.14', 'console_scripts', 'osaca')()
File "/usr/local/lib/python3.6/dist-packages/osaca/osaca.py", line 388, in main
run(args, output_file=args.out)
File "/usr/local/lib/python3.6/dist-packages/osaca/osaca.py", line 340, in run
inspect(args, output_file=output_file)
File "/usr/local/lib/python3.6/dist-packages/osaca/osaca.py", line 294, in inspect
semantics.add_semantics(kernel)
File "/usr/local/lib/python3.6/dist-packages/osaca/semantics/arch_semantics.py", line 31, in add_semantics
self.assign_tp_lt(instruction_form)
File "/usr/local/lib/python3.6/dist-packages/osaca/semantics/arch_semantics.py", line 176, in assign_tp_lt
instruction_form['instruction'][:-1], instruction_form['operands']
File "/usr/local/lib/python3.6/dist-packages/osaca/semantics/hw_model.py", line 113, in get_instruction
for instruction_form in name_matched_iforms if self._match_operands(
File "/usr/local/lib/python3.6/dist-packages/osaca/semantics/hw_model.py", line 115, in <genexpr>
operands))
File "/usr/local/lib/python3.6/dist-packages/osaca/semantics/hw_model.py", line 466, in _match_operands
operands_ok = operands_ok and self._check_operands(i_operand, operand)
File "/usr/local/lib/python3.6/dist-packages/osaca/semantics/hw_model.py", line 487, in _check_operands
return self._check_x86_operands(i_operand, operand)
File "/usr/local/lib/python3.6/dist-packages/osaca/semantics/hw_model.py", line 541, in _check_x86_operands
return i_operand['class'] == 'immediate' and i_operand['imd'] == 'int'
File "/home/andreas/.local/lib/python3.6/site-packages/ruamel/yaml/comments.py", line 753, in __getitem__
return ordereddict.__getitem__(self, key)
KeyError: 'imd'
I admit after reading the documentation a couple of times, it is not clear to me what types of files OSACA should take. IACA takes only binaries (AFAIK), so I had assumed it is binaries, but it seems like you can do source level annotations of C files like //STARTLOOP
so then I kind of assumed you feed in a source file, but then how it is going to get the binary (I assume you don't compile under the covers).
I think the answer is that OSACA also only takes binary files, but then it does some magic using debug info to map back to source files and loop for markers, but I think the documentation could maybe be a bit more explicit (and mention the requirement to use -g
since by default a lot of people won't).
As seen in https://travis-ci.com/RRZE-HPC/OSACA/builds/56366538, the first tox test case checks out, but does not print any useful data. The test case should therefore really check that the expected output is generated, not just that the command returned a valid exit code.
For example, it crashes when trying to analyze cqto
with following messages:
Traceback (most recent call last):
File "/Users/tom/workspace/OSACA/env/bin/osaca", line 11, in <module>
load_entry_point('osaca==0.2.1', 'console_scripts', 'osaca')()
File "/Users/tom/workspace/OSACA/env/lib/python3.7/site-packages/osaca-0.2.1-py3.7.egg/osaca/osaca.py", line 836, in main
osaca = OSACA(args.arch, assembly)
File "/Users/tom/workspace/OSACA/env/lib/python3.7/site-packages/osaca-0.2.1-py3.7.egg/osaca/osaca.py", line 408, in __init__
self.inspect()
File "/Users/tom/workspace/OSACA/env/lib/python3.7/site-packages/osaca-0.2.1-py3.7.egg/osaca/osaca.py", line 427, in inspect
self.check_instr(line)
File "/Users/tom/workspace/OSACA/env/lib/python3.7/site-packages/osaca-0.2.1-py3.7.egg/osaca/osaca.py", line 483, in check_instr
tc = Testcase(mnemonic, list(reversed(param_list_types)), '32')
File "/Users/tom/workspace/OSACA/env/lib/python3.7/site-packages/osaca-0.2.1-py3.7.egg/osaca/testcase.py", line 72, in __init__
self.__define_operands()
File "/Users/tom/workspace/OSACA/env/lib/python3.7/site-packages/osaca-0.2.1-py3.7.egg/osaca/testcase.py", line 134, in __define_operands
if isinstance(operands[0], Register):
IndexError: list index out of range
To predict a lower throughput bound, an optimal schedule could be used (if requested by user).
E.g. movl %fs:40, %eax
E.g. lea (,%rax,8), %rbx
.
IACA's output contains a line that shows the "Block Throughput" of a loop kernel.
Figure 4 in your paper (https://arxiv.org/pdf/1809.00912.pdf) shows a similar line with the "Total number of estimated throughput".
However, at least the current version of OSACA does not output this information. Is there an option that enables this output, or do you consider the throughput to be the maximum of the numbers in the line under the table in the "Combined Analysis Report"?
Can you get OSACA to predict an execution schedule?
With AVX512 instructions are mostly defined with masking support and not without. E.g., https://hjlebbink.github.io/x86doc/html/ADDPD.html. The ZMM version is defined with masking, but also works without. This needs to be considered when searching for instruction forms in the CSV.
As a workaround I am duplicating all instructions with masking support, with {opmask}
stripped from it: https://github.com/RRZE-HPC/OSACA/blob/imported_intel_models/osaca/data/model_importer.py#L139
The first OSACA analysis for a micro-arch after installing currently includes the read-in of the yaml file and only after that OSACA benefits from the fast cached pickle reuse.
Since most of the users don't adjust the data files before the first run, it might be beneficial to add the pickle files to the Pypi release to speedup the first run.
Of course it is necessary to check the module does not get blown up too much through this.
Right now, the user needs to specify the uarch by the --arch
flag.
It would be a plus for integration with other tools, if OSACA
--arch
flag becomes optional.My suggestion for default uarchs would be the most current Intel/Arm micro-arch it supports, i.e.:
Is it currently possible to differentiate between simple (base+offset) and complex (base+offset+index*scale) addressing as memory parameters in CSV the file syntax?
Before having been processed by the isa_semantics
, an instruction line data structure will contain a list of operands. Afterwords, this list is replaced by a dict of lists.
I would prefer to preserve the original list and add a new item to the instruction line dict instead of replacing it. This would not break code which works without the semantic processing, if it has been applied none the less.
Hey,
i'm getting the following error:
Traceback (most recent call last):
File "/opt/intel/intelpython3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/opt/intel/intelpython3/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/jens/.local/lib/python3.6/site-packages/osaca/__main__.py", line 4, in <module>
main()
File "/home/jens/.local/lib/python3.6/site-packages/osaca/osaca.py", line 415, in main
run(args, output_file=args.out)
File "/home/jens/.local/lib/python3.6/site-packages/osaca/osaca.py", line 364, in run
machine_model=mmodel, isa=misa)
File "/home/jens/.local/lib/python3.6/site-packages/osaca/osaca.py", line 299, in inspect
raise e
File "/home/jens/.local/lib/python3.6/site-packages/osaca/osaca.py", line 285, in inspect
parsed_code = parser.parse_file(code)
File "/home/jens/.local/lib/python3.6/site-packages/osaca/parser/base_parser.py", line 58, in parse_file
asm_instructions.append(self.parse_line(line, i + 1 + start_line))
File "/home/jens/.local/lib/python3.6/site-packages/osaca/parser/parser_x86att.py", line 260, in parse_line
"Could not parse instruction on line {}: {!r}".format(line_number, line)
ValueError: Could not parse instruction on line 5: 'jmpq *%ds:0x2517800(,%rdx,8)'
No idea what it means, but the simplest Rust compiler output generates it; https://godbolt.org/z/T9bh3P
Error: ValueError: Could not parse instruction on line 34: ' callq *%rax'
The following benchmark leads to a KeyError on IVB. It works on other targets.
echo 'cmp 0x8(%r13),%r8d' | osaca --arch IVB -
Output:
Traceback (most recent call last):
File "/usr/local/bin/osaca", line 11, in <module>
load_entry_point('osaca==0.3.14', 'console_scripts', 'osaca')()
File "/usr/local/lib/python3.6/dist-packages/osaca/osaca.py", line 387, in main
run(args, output_file=args.out)
File "/usr/local/lib/python3.6/dist-packages/osaca/osaca.py", line 339, in run
inspect(args, output_file=output_file)
File "/usr/local/lib/python3.6/dist-packages/osaca/osaca.py", line 293, in inspect
semantics.add_semantics(kernel)
File "/usr/local/lib/python3.6/dist-packages/osaca/semantics/arch_semantics.py", line 31, in add_semantics
self.assign_tp_lt(instruction_form)
File "/usr/local/lib/python3.6/dist-packages/osaca/semantics/arch_semantics.py", line 282, in assign_tp_lt
if INSTR_FLAGS.HAS_LD in instruction_form['flags']
File "/usr/local/lib/python3.6/dist-packages/osaca/semantics/hw_model.py", line 184, in get_load_latency
return self._data['load_latency'][reg_type] if self._data['load_latency'][reg_type] else 0
File "/home/andreas/.local/lib/python3.6/site-packages/ruamel/yaml/comments.py", line 753, in __getitem__
return ordereddict.__getitem__(self, key)
KeyError: 'gpr'
Example: add -0x1(%ebx),%eax
Output:
Traceback (most recent call last):
File "/home/andreas/.local/lib/python3.6/site-packages/osaca/parser/parser_x86att.py", line 255, in parse_line
result = self.parse_instruction(line)
File "/home/andreas/.local/lib/python3.6/site-packages/osaca/parser/parser_x86att.py", line 273, in parse_instruction
result = self.instruction_parser.parseString(instruction, parseAll=True).asDict()
File "/home/andreas/.local/lib/python3.6/site-packages/pyparsing.py", line 1955, in parseString
raise exc
File "/home/andreas/.local/lib/python3.6/site-packages/pyparsing.py", line 3814, in parseImpl
raise ParseException(instring, loc, self.errmsg, self)
pyparsing.ParseException: Expected end of text, found '-' (at char 4), (line:1, col:5)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/andreas/.local/lib/python3.6/site-packages/osaca/osaca.py", line 270, in inspect
parsed_code = parser.parse_file(code)
File "/home/andreas/.local/lib/python3.6/site-packages/osaca/parser/base_parser.py", line 57, in parse_file
asm_instructions.append(self.parse_line(line, i + 1 + start_line))
File "/home/andreas/.local/lib/python3.6/site-packages/osaca/parser/parser_x86att.py", line 258, in parse_line
'Could not parse instruction on line {}: {!r}'.format(line_number, line)
ValueError: Could not parse instruction on line 7: 'add -0x1(%ebx),%eax'
It is not possible anymore to run iaca_marker from kerncraft out of OSACA due to an updated version of kerncraft:
$ osaca -m asmfile.s
Traceback (most recent call last):
File "/proj/ciptmp/ol68umur/git/osacaVirt/bin/osaca", line 11, in <module>
load_entry_point('osaca==0.1', 'console_scripts', 'osaca')()
File "/proj/ciptmp/ol68umur/git/osacaVirt/lib/python3.5/site-packages/osaca-0.1-py3.5.egg/osaca/osaca.py", line 875, in main
block_selection='manual', pointer_increment=1)
File "/proj/ciptmp/ol68umur/git/osacaVirt/lib/python3.5/site-packages/kerncraft/iaca.py", line 300, in iaca_instrumentation
assembly_orig = input_file.readlines()
AttributeError: 'str' object has no attribute 'readlines'
Consider the following loop:
add [rdx], 1
I imagine OSACA will report it has having no loop carried dependencies. However, the actual iteration time of such a loop is 4-6 cycles, since there is a dependency through the memory location [rdx]
, which isn't detected by OSACA which only knows about register dependencies (I think).
Obviously, real world examples are more complicated, but the basic idea is the same: some subsequent iteration (not necessarily exactly the succeeding one, it could be later) may read a value from memory written by a the same or a previous iteration, forming a dependency.
Doesn't seem to install properly in v3, I always get this error when running:
$ osaca --arch IVB example.c
Traceback (most recent call last):
File "/usr/local/bin/osaca", line 11, in <module>
load_entry_point('osaca==0.3.1.dev0', 'console_scripts', 'osaca')()
File "/usr/local/lib/python3.5/dist-packages/osaca-0.3.1.dev0-py3.5.egg/osaca/osaca.py", line 232, in main
check_user_dir()
File "/usr/local/lib/python3.5/dist-packages/osaca-0.3.1.dev0-py3.5.egg/osaca/osaca.py", line 129, in check_user_dir
if dir_comp.left_list != dir_comp.same_files:
File "/usr/lib/python3.5/filecmp.py", line 247, in __getattr__
self.methodmap[attr](self)
File "/usr/lib/python3.5/filecmp.py", line 135, in phase0
self.right_list = _filter(os.listdir(self.right),
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.5/dist-packages/osaca-0.3.1.dev0-py3.5.egg/osaca/data/'
It seems like the data
directory is not installed by ./setup.py
.
Ports Pressure in cycles
| 0 | 0DV | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
----------------------------------------------------------------
| | | | | | | | | | X addl $1, 8(%rax)
It recognizes the instruction when you remove the l
suffix. But the issue is addl $1, 8(%rax)
is the only way to write this instruction -- gas
refuses to assemble add $1, 8(%rax)
because the add
is ambiguous.
OSACA can't parse instructions like this lea -1(%rbx), %rax
.
When running
sudo python3 ./setup.py install
in the v3 branch of OSACA, installation fails with the following error:
Processing dependencies for osaca==0.3.1.dev0
Searching for pygraphviz
Reading https://pypi.org/simple/pygraphviz/
Downloading https://files.pythonhosted.org/packages/7e/b1/d6d849ddaf6f11036f9980d433f383d4c13d1ebcfc3cd09bc845bda7e433/pygraphviz-1.5.zip#sha256=50a829a305dc5a0fd1f9590748b19fece756093b581ac91e00c2c27c651d319d
Best match: pygraphviz 1.5
Processing pygraphviz-1.5.zip
Writing /tmp/easy_install-4094m0_g/pygraphviz-1.5/setup.cfg
Running pygraphviz-1.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-4094m0_g/pygraphviz-1.5/egg-dist-tmp-cxvkeot1
warning: no files found matching '*.png' under directory 'doc'
warning: no files found matching '*.html' under directory 'doc'
warning: no files found matching '*.txt' under directory 'doc'
warning: no files found matching '*.css' under directory 'doc'
warning: no previously-included files matching '*~' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '.svn' found anywhere in distribution
no previously-included directories found matching 'doc/build'
pygraphviz/graphviz_wrap.c:2987:29: fatal error: graphviz/cgraph.h: No such file or directory
#include "graphviz/cgraph.h"
^
compilation terminated.
error: Setup script exited with error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
To fix it, I had to do a sudo apt-get install libgraphviz-dev
(Ubuntu) - maybe that library should be added to the list of dependencies in the README?
Hi, i have an asm sequence which forces osaca into a lifelock. I tried runnig with python -m trace to find out whats wrong, but had no luck pinpointing the issue without deep knowledge about the osaca internals, so I have to hand it of to you.
The problem exists in osaca-0.3.14 and osaca-0.4.0 (for latter: after fixing the parser bug at semantics/kernel_dg.py line 316 which prevents using hex for callq and other instructions).
My call: python3 -m osaca --arch BDW --ignore-unknown --verbose /dev/shm/osaca_43305_556_575.s
โ cat /dev/shm/osaca_43305_556_575.s
# OSACA-BEGIN
push %r12
push %r13
push %r14
push %r15
push %rbp
mov %ecx,%r12d
mov %esi,%r14d
mov %r12d,%ecx
mov %r14d,%esi
mov %rdx,%r13
mov %rdi,%rbp
callq 0x4210d0 #<x264_pixel_satd_8x4> (File Offset: 0x210d0)
mov %rdx,%r8
movzbl (%rdi),%r9d
movslq %esi,%rsi
movslq %ecx,%rcx
movzbl (%r8),%r10d
vmovd %r9d,%xmm13
movzbl 0x4(%r8),%r9d
vpinsrb $0x1,(%rsi,%rdi,1),%xmm13,%xmm14
lea (%rsi,%rsi,2),%rdx
vmovd %r10d,%xmm1
vpinsrb $0x1,(%rcx,%r8,1),%xmm1,%xmm0
vmovd %r9d,%xmm7
vpinsrb $0x1,0x4(%rcx,%r8,1),%xmm7,%xmm5
vpinsrb $0x2,(%rdi,%rsi,2),%xmm14,%xmm15
vpinsrb $0x2,(%r8,%rcx,2),%xmm0,%xmm6
vpinsrb $0x2,0x4(%r8,%rcx,2),%xmm5,%xmm9
vpinsrb $0x3,(%rdx,%rdi,1),%xmm15,%xmm4
movzbl 0x4(%rdi),%r11d
lea (%rcx,%rcx,2),%rax
vpinsrb $0x3,(%rax,%r8,1),%xmm6,%xmm10
vpinsrb $0x3,0x4(%rax,%r8,1),%xmm9,%xmm11
vmovd %r11d,%xmm2
vpinsrb $0x1,0x4(%rsi,%rdi,1),%xmm2,%xmm8
vpinsrb $0x2,0x4(%rdi,%rsi,2),%xmm8,%xmm3
movzbl 0x1(%rdi),%r10d
movzbl 0x5(%rdi),%r9d
movzbl 0x1(%r8),%r11d
vmovd %r10d,%xmm1
movzbl 0x5(%r8),%r10d
vmovd %r9d,%xmm7
vpmovzxbd %xmm4,%xmm4
vmovd %r11d,%xmm2
vpmovzxbd %xmm10,%xmm10
vpinsrb $0x3,0x4(%rdx,%rdi,1),%xmm3,%xmm12
vpsubd %xmm10,%xmm4,%xmm14
vpinsrb $0x1,0x5(%rsi,%rdi,1),%xmm7,%xmm5
vmovd %r10d,%xmm4
vpinsrb $0x1,0x5(%rcx,%r8,1),%xmm4,%xmm10
vpinsrb $0x1,0x1(%rcx,%r8,1),%xmm2,%xmm8
vpinsrb $0x1,0x1(%rsi,%rdi,1),%xmm1,%xmm0
vpinsrb $0x2,0x5(%rdi,%rsi,2),%xmm5,%xmm9
vpinsrb $0x2,0x1(%r8,%rcx,2),%xmm8,%xmm3
vpinsrb $0x2,0x1(%rdi,%rsi,2),%xmm0,%xmm6
vpmovzxbd %xmm12,%xmm12
vpmovzxbd %xmm11,%xmm11
vpsubd %xmm11,%xmm12,%xmm13
vpinsrb $0x2,0x5(%r8,%rcx,2),%xmm10,%xmm11
vpslld $0x10,%xmm13,%xmm15
vpinsrb $0x3,0x1(%rdx,%rdi,1),%xmm6,%xmm13
vpaddd %xmm15,%xmm14,%xmm12
vpinsrb $0x3,0x5(%rdx,%rdi,1),%xmm9,%xmm15
vpinsrb $0x3,0x1(%rax,%r8,1),%xmm3,%xmm14
vpinsrb $0x3,0x5(%rax,%r8,1),%xmm11,%xmm1
movzbl 0x2(%rdi),%r11d
movzbl 0x2(%r8),%r9d
vpmovzxbd %xmm15,%xmm15
vmovd %r11d,%xmm8
vmovd %r9d,%xmm5
vpinsrb $0x1,0x2(%rsi,%rdi,1),%xmm8,%xmm3
vpinsrb $0x1,0x2(%rcx,%r8,1),%xmm5,%xmm9
vpinsrb $0x2,0x2(%rdi,%rsi,2),%xmm3,%xmm7
vpinsrb $0x2,0x2(%r8,%rcx,2),%xmm9,%xmm4
vpinsrb $0x3,0x2(%rdx,%rdi,1),%xmm7,%xmm3
vpinsrb $0x3,0x2(%rax,%r8,1),%xmm4,%xmm7
vpmovzxbd %xmm1,%xmm1
movzbl 0x6(%r8),%r11d
vpsubd %xmm1,%xmm15,%xmm0
vpmovzxbd %xmm13,%xmm13
vpslld $0x10,%xmm0,%xmm2
vpmovzxbd %xmm14,%xmm14
vpsubd %xmm14,%xmm13,%xmm6
vpaddd %xmm2,%xmm6,%xmm11
vmovd %r11d,%xmm6
vpinsrb $0x1,0x6(%rcx,%r8,1),%xmm6,%xmm2
movzbl 0x6(%rdi),%r10d
vpinsrb $0x2,0x6(%r8,%rcx,2),%xmm2,%xmm8
vmovd %r10d,%xmm10
vpinsrb $0x1,0x6(%rsi,%rdi,1),%xmm10,%xmm1
vpinsrb $0x3,0x6(%rax,%r8,1),%xmm8,%xmm9
vpinsrb $0x2,0x6(%rdi,%rsi,2),%xmm1,%xmm0
movzbl 0x3(%rdi),%r9d
movzbl 0x7(%rdi),%r11d
vpmovzxbd %xmm3,%xmm3
vpmovzxbd %xmm7,%xmm7
vmovd %r9d,%xmm14
vmovd %r11d,%xmm8
vpsubd %xmm7,%xmm3,%xmm10
vpinsrb $0x1,0x3(%rsi,%rdi,1),%xmm14,%xmm15
vpinsrb $0x1,0x7(%rsi,%rdi,1),%xmm8,%xmm3
vpinsrb $0x3,0x6(%rdx,%rdi,1),%xmm0,%xmm5
vpinsrb $0x2,0x3(%rdi,%rsi,2),%xmm15,%xmm1
vpinsrb $0x2,0x7(%rdi,%rsi,2),%xmm3,%xmm7
vpaddd %xmm11,%xmm12,%xmm3
vpmovzxbd %xmm5,%xmm5
vpmovzxbd %xmm9,%xmm9
vpsubd %xmm9,%xmm5,%xmm4
vpslld $0x10,%xmm4,%xmm13
vpinsrb $0x3,0x7(%rdx,%rdi,1),%xmm7,%xmm15
vpaddd %xmm13,%xmm10,%xmm10
vpinsrb $0x3,0x3(%rdx,%rdi,1),%xmm1,%xmm13
movzbl 0x7(%r8),%edx
movzbl 0x3(%r8),%r10d
vpmovzxbd %xmm15,%xmm15
vmovd %edx,%xmm5
vpinsrb $0x1,0x7(%rcx,%r8,1),%xmm5,%xmm9
vmovd %r10d,%xmm0
vpinsrb $0x1,0x3(%rcx,%r8,1),%xmm0,%xmm6
vpinsrb $0x2,0x7(%r8,%rcx,2),%xmm9,%xmm4
vpinsrb $0x2,0x3(%r8,%rcx,2),%xmm6,%xmm2
vpinsrb $0x3,0x7(%rax,%r8,1),%xmm4,%xmm1
vpinsrb $0x3,0x3(%rax,%r8,1),%xmm2,%xmm14
vpmovzxbd %xmm1,%xmm1
vpmovzxbd %xmm13,%xmm13
vpsubd %xmm1,%xmm15,%xmm0
vpmovzxbd %xmm14,%xmm14
vpslld $0x10,%xmm0,%xmm2
vpsubd %xmm14,%xmm13,%xmm6
vpsubd %xmm11,%xmm12,%xmm1
vpaddd %xmm2,%xmm6,%xmm8
vpaddd %xmm8,%xmm10,%xmm12
vpsubd %xmm8,%xmm10,%xmm0
vpaddd %xmm12,%xmm3,%xmm8
vpaddd %xmm0,%xmm1,%xmm7
vpsubd %xmm12,%xmm3,%xmm3
vpsubd %xmm0,%xmm1,%xmm5
vunpcklps %xmm7,%xmm8,%xmm6
vunpcklps %xmm5,%xmm3,%xmm2
vunpckhps %xmm7,%xmm8,%xmm9
vunpckhps %xmm5,%xmm3,%xmm4
vunpcklpd %xmm2,%xmm6,%xmm10
vunpckhpd %xmm2,%xmm6,%xmm11
vunpcklpd %xmm4,%xmm9,%xmm12
vpaddd %xmm11,%xmm10,%xmm14
vunpckhpd %xmm4,%xmm9,%xmm13
vpsubd %xmm11,%xmm10,%xmm1
vpaddd %xmm13,%xmm12,%xmm15
vpsubd %xmm13,%xmm12,%xmm0
vpaddd %xmm15,%xmm14,%xmm9
vpaddd %xmm0,%xmm1,%xmm7
vpsubd %xmm15,%xmm14,%xmm8
vpsubd %xmm0,%xmm1,%xmm6
vmovdqu 0x279d68(%rip),%xmm15 # 0x69b170 #<hpel_ref1+0x130> (File Offset: 0x29b170)
vpsrld $0xf,%xmm9,%xmm2
vpsrld $0xf,%xmm7,%xmm10
vpand %xmm15,%xmm2,%xmm3
vmovdqu 0x279d40(%rip),%xmm4 # 0x69b160 #<hpel_ref1+0x120> (File Offset: 0x29b160)
vpand %xmm15,%xmm10,%xmm11
vpsrld $0xf,%xmm8,%xmm12
vpsrld $0xf,%xmm6,%xmm14
vpmulld %xmm3,%xmm4,%xmm5
vpand %xmm15,%xmm12,%xmm13
vpmulld %xmm11,%xmm4,%xmm3
vpand %xmm15,%xmm14,%xmm1
vpmulld %xmm13,%xmm4,%xmm2
vpaddd %xmm3,%xmm7,%xmm7
vpmulld %xmm1,%xmm4,%xmm0
vpaddd %xmm5,%xmm9,%xmm4
vpxor %xmm5,%xmm4,%xmm5
vpxor %xmm3,%xmm7,%xmm9
vpaddd %xmm2,%xmm8,%xmm8
vpaddd %xmm9,%xmm5,%xmm3
vpxor %xmm2,%xmm8,%xmm2
vpaddd %xmm0,%xmm6,%xmm6
vpaddd %xmm2,%xmm3,%xmm4
vpxor %xmm0,%xmm6,%xmm0
vpaddd %xmm0,%xmm4,%xmm2
vpxor %xmm1,%xmm1,%xmm1
vpaddd %xmm2,%xmm1,%xmm1
vpsrldq $0x8,%xmm1,%xmm3
vpaddd %xmm3,%xmm1,%xmm5
vpsrlq $0x20,%xmm5,%xmm6
vpaddd %xmm6,%xmm5,%xmm7
vmovd %xmm7,%ecx
movzwl %cx,%eax
shr $0x10,%ecx
add %ecx,%eax
shr %eax
retq
# OSACA-END
Have a class representation of an InstructionForm, which can be created from an assembly line or by describing a mnemonic and its parameters.
This class should be able to generate a canonical string representation and handle (or at least reference) performance relevant information.
simply pip install osaca
on a system without the requisite dependencies already installed (pyparsing, ruamel.yaml, etc):
Build ISA and architecture cache
Traceback (most recent call last):
File "_build_cache.py", line 7, in <module>
from osaca.semantics.hw_model import MachineModel
File "../../osaca/semantics/__init__.py", line 6, in <module>
from .isa_semantics import ISASemantics, INSTR_FLAGS
File "../../osaca/semantics/isa_semantics.py", line 5, in <module>
from osaca.parser import AttrDict, ParserAArch64, ParserX86ATT
File "../../osaca/parser/__init__.py", line 8, in <module>
from .parser_x86att import ParserX86ATT
File "../../osaca/parser/parser_x86att.py", line 5, in <module>
import pyparsing as pp
ModuleNotFoundError: No module named 'pyparsing'
the install succeeds, but not using a bdist (and this ISA/architecture cache is not built)
The following benchmark leads to a KeyError on SNB. It works on other targets.
echo 'test %r14,0x38(%rsp)' | osaca --arch SNB -
Output:
Traceback (most recent call last):
File "/usr/local/bin/osaca", line 11, in <module>
load_entry_point('osaca==0.3.14', 'console_scripts', 'osaca')()
File "/usr/local/lib/python3.6/dist-packages/osaca/osaca.py", line 387, in main
run(args, output_file=args.out)
File "/usr/local/lib/python3.6/dist-packages/osaca/osaca.py", line 339, in run
inspect(args, output_file=output_file)
File "/usr/local/lib/python3.6/dist-packages/osaca/osaca.py", line 293, in inspect
semantics.add_semantics(kernel)
File "/usr/local/lib/python3.6/dist-packages/osaca/semantics/arch_semantics.py", line 31, in add_semantics
self.assign_tp_lt(instruction_form)
File "/usr/local/lib/python3.6/dist-packages/osaca/semantics/arch_semantics.py", line 282, in assign_tp_lt
if INSTR_FLAGS.HAS_LD in instruction_form['flags']
File "/usr/local/lib/python3.6/dist-packages/osaca/semantics/hw_model.py", line 184, in get_load_latency
return self._data['load_latency'][reg_type] if self._data['load_latency'][reg_type] else 0
File "/home/andreas/.local/lib/python3.6/site-packages/ruamel/yaml/comments.py", line 753, in __getitem__
return ordereddict.__getitem__(self, key)
KeyError: 'gpr'
Running osaca
fails with python 2.7 yields
$ osaca --help
Traceback (most recent call last):
File "/home/mgodbolt/apps/miniconda/envs/osaca/bin/osaca", line 11, in <module>
load_entry_point('osaca==0.1', 'console_scripts', 'osaca')()
File "/home/mgodbolt/apps/miniconda/envs/osaca/lib/python2.7/site-packages/pkg_resources/__init__.py", line 570, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/home/mgodbolt/apps/miniconda/envs/osaca/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2687, in load_entry_point
return ep.load()
File "/home/mgodbolt/apps/miniconda/envs/osaca/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2341, in load
return self.resolve()
File "/home/mgodbolt/apps/miniconda/envs/osaca/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2347, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/mnt/data/apps/miniconda/envs/osaca/lib/python2.7/site-packages/osaca-0.1-py2.7.egg/osaca/osaca.py", line 54
print('Copying files in user directory...', file=self.file_output, end='')
^
SyntaxError: invalid syntax
It was installed without issue in a conda
container created with conda create -n osaca python=2.7
, and then python ./setup.py install
Judging from the error, it looks like Python 3 is the minimum requirement. The installer should check the python version, if this is the case, or the print()
call should be a Python 2.7 print
statement if 2.7 is supported.
As described in more detail in this gist not all types of loop carried dependencies (LCDs) can be detected. E.g., the carried dependency in this loop cannot be detected:
add eax, 1 ; A (deps: E-previous)
add eax, 1 ; B (deps: A)
add eax, 1 ; C (deps: B)
; swap eax, ebx (exploded version of xchg eax, ebx)
mov ecx, eax ; D (deps: C)
mov eax, ebx ; E (deps: F-previous)
mov ebx, ecx ; F (deps: D)
... since it takes two iterations for the loop to complete.
I think this can be fixed fairly simply - you just keep unrolling copies until for each node in the current copy the set of original nodes that feed into every node (i.e., the "reachable from" or "depends on" set) is the same as the previous iteration. So you'll need 3 copies rather than 2 in the common case (where there are no skipped a generation things like the above) - the 3rd copy just verifies that iteration is complete. In this case you'd need 4 copies. In general, the number of copies you need to create should be bounded by the size of the loop (although I don't have a proof of that), so small in practice.
In the architecture CSV or another file format ports and port names should be defined. Also DV ports should not require special handling, making the code less complex.
During every execution of OSACA, first, a warning is shown:
$ osaca
/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed,
may indicate binary incompatibility. Expected 96, got 88
return f(*args, **kwds)
As far as I can observe, right now the functionality of the tool is not affected by this warning.
Hi all,
I want to ask about Icelake support in OSACA. It's one of recently released architecture and I will be happy to see this in your analyzer.
I see some mentions of ICL in the code, but it's not supported architecture anyway. Do you have any plans to support this?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.