google / exegesis Goto Github PK

License: Apache License 2.0

C++ 95.31% Makefile 0.10% Starlark 4.58%

exegesis's Introduction

State of the repository

As of January 2023, this project is no longer being maintained. Many parts of this projects have open-source alternatives that are part of a bigger effort and that are actively maintained:

llvm-exegesis allows analyzing individual instructions or snippets of assembly code.
llvm-mca is a CPU pipeline simulation tool.
uops.info provides detailed performance characteristics and a machine-readable database of x86-64 instructions.

Goal

Google's EXEgesis project aims to improve code generation in compilers, via:

Providing machine-readable lists of instructions for hardware vendors and microarchitectures.
Providing tools for debugging the performance of code based on this data.

For a high-level overview of our efforts, see the slides for a tech talk about EXEgesis (July 2017).

We are providing tools to measure instruction latencies and µOps scheduling. We have contributed that part into LLVM as as the llvm-exegesis tool.

Details

This repository provides a set of tools for extracting data about instructions and latencies from canonical sources and converting them into machine-readable form. Some require parsing PDF files; others are more straightforward.

The output data is available in the form of a Protocol Buffer message.

It includes:

A textual description. e.g. Add with carry imm8 to AL.
The raw encoding. e.g. 14 ib and equivalent LLVM mnemonic. e.g. ADC8i8

What's Next

Intel x86-64 - done

Get Involved

Issue tracker: https://github.com/google/EXEgesis/issues
Mailing list: [email protected]

We welcome patches -- see CONTRIBUTING for more information on how to submit a patch.

exegesis's People

Contributors

Stargazers

Watchers

exegesis's Issues

Missing instructions after transform (AAA, AAD, ..., REP MOVS, ...)

After the #2 I did a new run. To make it easier for myself I moved the scripts and files into separate directory so I wouldn't accidentally lose them. After I did a bazel build and verified it succeeded, I got the pbtxt with the following command:

set -e

SRC=/archive/Documents/readable-intel-manual/325462-sdm-vol-1-2abcd-3abcd.pdf
DST=/archive/Documents/readable-intel-manual/instructions

pushd /archive/Documents/CPU-instructions
    bazel run cpu_instructions/tools:parse_sdm -- \
      --cpu_instructions_input_spec=$SRC \
      --cpu_instructions_output_file_base=$DST \
      --cpu_instructions_transforms=default
popd

I did a conversion into JSON and then retrieved the mnemonic names in each file and cross-checked them. I found out that the following came visible:

TEST

The following instructions went to missing:

AAA
AAD
AAM
AAS
ARPL
BOUND
DAA
DAS
FCLEX
FINIT
FSAVE
FSTCW
FSTENV
FSTSW
INTO
JCXZ
LAHF
LDS
LES
POPA
POPAD
POPFD
PUSHA
PUSHAD
PUSHFD
REP INS
REP LODS
REP MOVS
REP OUTS
REP STOS
REPE CMPS
REPE SCAS
REPNE CMPS
REPNE SCAS
SAHF
XACQUIRE
XLAT
XRELEASE

This is interesting, because in the instructions.pbtxt "XLAT" and "DAA" appears to be present, but they are missing from the _transformed.pbtxt.

The manual used here was: Combined Volumes: 1, 2A, 2B, 2C, 2D, 3A, 3B, 3C, 3D and 4, Order Number: 325462-062US, March 2017

Followed Readme into Abort trap

...
INFO: Elapsed time: 89.711s, Critical Path: 33.29s
INFO: 449 processes: 449 darwin-sandbox.
INFO: Build completed successfully, 452 total actions
INFO: Running command line: bazel-bin/exegesis/tools/parse_intel_sdm '--exegesis_input_spec=${HOME}/Downloads/325462-sdm-vol-1-2abcd-3abcd.pdf' '--exegesis_output_file_ba
INFO: Build completed successfully, 452 total actions
I0917 22:17:41.201684 157932992 pdf_document_utils.cc:376] Reading configuration file exegesis/x86/pdf/sdm_patches//.
I0917 22:17:41.202687 157932992 pdf_document_utils.cc:376] Reading configuration file exegesis/x86/pdf/sdm_patches//..
I0917 22:17:41.202735 157932992 pdf_document_utils.cc:376] Reading configuration file exegesis/x86/pdf/sdm_patches//sdm_2019_10_vol1234.pbtxt
I0917 22:17:41.203758 157932992 pdf_document_utils.cc:376] Reading configuration file exegesis/x86/pdf/sdm_patches//sdm_2017_03_vol123.pbtxt
I0917 22:17:41.204741 157932992 pdf_document_utils.cc:376] Reading configuration file exegesis/x86/pdf/sdm_patches//sdm_2018_03_vol1234.pbtxt
I0917 22:17:41.205634 157932992 pdf_document_utils.cc:376] Reading configuration file exegesis/x86/pdf/sdm_patches//sdm_2017_10_vol1234.pbtxt
I0917 22:17:41.206516 157932992 pdf_document_utils.cc:376] Reading configuration file exegesis/x86/pdf/sdm_patches//sdm_2017_07_vol1234.pbtxt
I0917 22:17:41.207487 157932992 pdf_document_utils.cc:376] Reading configuration file exegesis/x86/pdf/sdm_patches//sdm_2016_12_vol123.pbtxt
I0917 22:17:41.208472 157932992 pdf_document_utils.cc:376] Reading configuration file exegesis/x86/pdf/sdm_patches//sdm_2016_09_vol2.pbtxt
I0917 22:17:41.209432 157932992 pdf_document_utils.cc:376] Reading configuration file exegesis/x86/pdf/sdm_patches//sdm_2018_05_vol1234.pbtxt
I0917 22:17:41.210330 157932992 pdf_document_utils.cc:376] Reading configuration file exegesis/x86/pdf/sdm_patches//sdm_2017_12_vol1234.pbtxt
I0917 22:17:41.211222 157932992 pdf_document_utils.cc:376] Reading configuration file exegesis/x86/pdf/sdm_patches//sdm_2018_11_vol1234.pbtxt
I0917 22:17:41.212044 157932992 pdf_document_utils.cc:376] Reading configuration file exegesis/x86/pdf/sdm_patches//sdm_2016_04_vol2a.pbtxt
I0917 22:17:41.212370 157932992 pdf_document_utils.cc:376] Reading configuration file exegesis/x86/pdf/sdm_patches//sdm_2019_05_vol1234.pbtxt
I0917 22:17:41.213207 157932992 pdf_document_utils.cc:376] Reading configuration file exegesis/x86/pdf/sdm_patches//sdm_2019_01_vol1234.pbtxt
I0917 22:17:41.214068 157932992 pdf_document_utils.cc:376] Reading configuration file exegesis/x86/pdf/sdm_patches//sdm_2016_06_vol2.pbtxt
I0917 22:17:41.215736 157932992 pdf_document_utils.cc:376] Reading configuration file exegesis/x86/pdf/sdm_patches//sdm_2016_04_vol2b.pbtxt
F0917 22:17:42.612874 157932992 xpdf_util.cc:323] Check failed: all_patches.documents().empty() || patches != nullptr Unable to find document_id 'title: "Intel\302\256 64 and IA-32 Architectures Software Developers Manual, Combined Volumes: 1, 2A, 2B, 2C, 2D, 3A, 3B, 3C, 3D and 4"
creation_date: "D:20200526123021Z"
modification_date: "D:20200526132553-07\'00\'"
' in '${HOME}/Downloads/325462-sdm-vol-1-2abcd-3abcd.pdf'
*** Check failure stack trace: ***
    @        0x1010ce2ef  google::LogMessageFatal::~LogMessageFatal()
    @        0x1010ca849  google::LogMessageFatal::~LogMessageFatal()
    @        0x100ead28c  exegesis::pdf::ParseOrDie()
    @        0x100e9046a  exegesis::x86::pdf::ParseSdmOrDie()
    @        0x100e81ba7  main
Abort trap: 6

Integrating with OSS-Fuzz

Greetings EXEgesis developers and contributors,

We’re reaching out because your project is an important part of the open source ecosystem, and we’d like to invite you to integrate with our fuzzing service, OSS-Fuzz. OSS-Fuzz is a free fuzzing infrastructure you can use to identify security vulnerabilities and stability bugs in your project. OSS-Fuzz will:

Continuously run at scale all the fuzzers you write.
Alert you when it finds issues.
Automatically close issues after they’ve been fixed by a commit.

Many widely used open source projects like OpenSSL, FFmpeg, LibreOffice, and ImageMagick are fuzzing via OSS-Fuzz, which helps them find and remediate critical issues.

Even though typical integrations can be done in < 100 LoC, we have a reward program in place which aims to recognize folks who are not just contributing to open source, but are also working hard to make it more secure.

We want to stress that anyone who meets the eligibility criteria and integrates a project with OSS-Fuzz is eligible for a reward.

To help you getting started, we can attach our fuzzer for your project that you are welcome to use directly, or to use it as a starting point.

If you're not interested in integrating with OSS-Fuzz, it would be helpful for us to understand why—lack of interest, lack of time, or something else—so we can better support projects like yours in the future.

If we’ve missed your question in our FAQ, feel free to reply or reach out to us at [email protected].

Thanks!

Tommy
OSS-Fuzz Team

ARM fully supported?

The top-level README says ARM Cortex is "underway", but the tools README lists 4 supported versions of the XML data. Is there more work left to do on ARM or is the top-level README out of date?

failed to build

My build failed for "missing dependency declarations".
I did the following:

installed bazel
clone repo
change directory to repo
then lastly

$ bazel build //exegesis/tools:parse_sdm -c opt
INFO: Found 1 target...
ERROR: /private/var/tmp/_bazel_user/f8a4e05608f194d30caf1f6a29617bdb/external/utf_archive/BUILD.bazel:1:1: undeclared inclusion(s) in rule '@utf_archive//:utf':
this rule is missing dependency declarations for the following files included by 'external/utf_archive/libutf/runestrncmp.c':
  '/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/8.0.0/include/stdarg.h'.
warning: unknown warning option '-Wno-misleading-indentation'; did you mean '-Wno-missing-declarations'? [-Wunknown-warning-option]
1 warning generated.
Target //exegesis/tools:parse_sdm failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 2.370s, Critical Path: 1.66s


$ bazel help startup_options
[bazel release 0.4.5-homebrew]

What may I have done wrong?

Generate C/C++ header from .bptxt file

How to convert .pbtxt to valid C/C++ headers?

I've used protoc.exe but it needs .proto files

link error when libunwind-dev is installed

When libunwind is available, glog's configure will use it, but we don't link with it in the bazel build rule.

TEST - Logical Compare missing

I've been using the output from this project lately and this came up. TEST seems to be missing in the pbtxt output entirely.

It has not done this for anything else in this project, only this one is missing in between of SYSRET and TZCNT.

That's the bug. But also I got something to say about the rest of the project. At the best the output I've gotten from it is like this:

instructions {
  description: "Move r32 to r/m32."
  vendor_syntax {
    mnemonic: "MOV"
    operands {
      encoding: MODRM_RM_ENCODING
      name: "r/m32"
      usage: USAGE_WRITE
    }
    operands {
      encoding: MODRM_REG_ENCODING
      name: "r32"
      usage: USAGE_READ
    }
  }
  available_in_64_bit: true
  legacy_instruction: true
  encoding_scheme: "MR"
  raw_encoding_specification: "89 /r"
  group_id: "MOV-Move"
}

Overall it's enough for assembling and disassembling with this thing. But not enough for simulating the architecture or calculating costs for optimization.

It also doesn't provide way to differentiate between near and far return.

I've been happy to the results despite that. It's way much nicer to browse than the large PDF files.

Manual I used as input was: "Intel® 64 and IA-32 Architectures
Software Developer’s Manual", Order Number: 325383-060US, September 2016

Provide Example of Generating min_latency, max_latency, etc

Hello,

I was very excited to discover the Exegesis repo and am eager to try it out. I have followed the instructions given in the root README.md file, but I noticed that the resulting output file /tmp/intel_isa.pbtxt does not include any of the latency related fields given in this .proto file (such as min_latency, max_latency, etc):

https://github.com/google/EXEgesis/blob/master/exegesis/proto/instructions.proto

If anyone could kindly provide an example command that populates these fields, I would sincerely appreciate it.

I apologize for posting this as an issue when it might be more appropriate for the @googlegroups list, but my email to that list bounced.