I have a data table (512 uint64s) that I'd like to be able to index from PeachPy. It is possible to access a global Go symbol via the SB pseudo-register?

My work-around at the moment is to pass the table as an extra slice to my function.

Function.isa_extensions should return isa.Extensions object with the ISA extensions used in its code

Will multiple return values be supported for Go?

Currently the code that detects relocation position in instructions with Constant operands assumes that disp32 is the last part of the encoding. This assumption does not hold if instruction has both a Constant operand and an immediate operand, which can happen e.g. in [V]CMPxx instructions. Need a more generic code to detect relocation position.

Is it possible to generate executable file instead of object? If so, how to define entry function?
P. S. I'd like to try using peachpy as a backend for my compiler.

Go/SysO ABIs return result on stack. PeachPy currently generates sequence to return result in rax/xmm0.

Hi again!

I tried generating code for IMUL(rax, rbx, 10) which outputs IMULQ BX, $10, AX which is not recognized by Go. Removing BX from the instruction, IMULQ $10, AX, makes it valid.

PeachPy allocates rbp as a general-purpose register in NaCl SFI ABI when it runs out of other registers. This normally leads to SFI violation, because only special pseudo-instructions can modify rbp.

The MOVZX instruction is missing go_name entries meaning the instruction is emitted as a sequence of BYTE directives.

The Go assembler supports this instruction as MOVBLZX, MOVBWZX, MOVBQZX

Example:

x86_64\abi.py Line 134:

    elif osname == "Windows" and machine == "AMD64" and pointer_size == 8:
        return microsoft_x64_abi

But when I get here on my Windows 8.1 64 bit, running Python35 (32 bit Python35 BTW.. probably should install 64 bit) it skips this abi, because my pointer_size = 4, not 8.

..not sure if running a 32 bit Python on a 64 bit machine is even something you want to deal with.

I didn't mean to install 32 bit Python, and will correct in a bit - let me know if you want me to flog PeachPy in this environment a bit more before I correct and I will.

RETURN tests needed for non-Golang ABIs:

1. Return 0 should generate XOR eax, eax for all integer types.
2. Return negative values should generate MOV eax, imm32 or MOV rax, imm32.
3. Return positive uint32 should always generate MOV eax, imm32.
4. Return GP register with and without extension.
5. Return XMM register from float and double functions should generate MOVSS/MOVSD accordingly.
6. Return XMM register from float and double functions with __m256* argument should generate VMOVSS/VMOVSD accordingly and no VZEROUPPER.
7. Return MM register from __m64 function should generate __m64 and no EMMS.
8. Return XMM register from __m128, __m128d, __m128i functions should generate MOVAPS/MOVAPD/MOVDQA accordingly.
9. Return XMM register from __m128, __m128d, __m128i functions with __m256* argument should generate VMOVAPS/VMOVAPD/VMOVDQA accordingly and no VZEROUPPER.
10. Return YMM register from __m256, __m256d, __m256i functions should generate VMOVAPS/VMOVAPD/VMOVDQA accordingly and no VZEROUPPER.

For example, CRC32 has the following supported forms:

        * CRC32(r32, r8/m8)
        * CRC32(r32, r16/m16)
        * CRC32(r32, r32/m32)
        * CRC32(r64, r8/m8)
        * CRC32(r64, r64/m64)

Given an r64 as the first operand, how can I tell the code generator that the second operand is an m64 and not an m8? Trying CRC32(rax, [rbx]) generates CRC32B 0(BX), AX which assumes that [rbx] is m8. How could I make the generated code to be CRC32Q 0(BX), AX instead?

This will solve a lot of problems, in particular using Constant in EVEX-encoded instructions.

I can used it to help me understand assembly better and even do some insane things.
Great jobs guys 🆒

Using SWAP.REGISTER causes erroneous code generation.
The immediate reason is that check_operand does not create a copy of the register. As a result, change in the register's virtual_id or physical_id affects all previously generated instructions.
However, if this problem is fixed, another error emerges in register allocation.

The Function class supports target argument to specify target microarchitecture or ISA. Function.add_function must check that all instruction set extensions required for the instruction are supported by the target.

RETURN pseudo-instruction currently accepts only constant values.

• RETURN should accept general-purpose registers for functions with integer or pointer return type. Lower to STORE.RESULT for Go and to MOV[ZX/SX](rax/eax, reg) for other ABIs
• RETURN should accept xmm registers for functions with floating-point return type. Lower to STORE.RESULT for Go and to MOVSS/MOVSD(xmm0, xmmN) for other ABIs.
• RETURN should accept mm/xmm/ymm registers for functions with vector return type. Lower to vector mov instructions for non-Go ABIs, prohibit in Go ABI.

Hi!

The generated code for PSHUFD has the wrong order of operands. For example, the generated code for PSHUFD(xmm1, xmm0, 0) is PSHUFD X0, $0, X1. This should be PSHUFD $0, X0, X1.

By the way, nice project!

I have a question:
Say I am developing an application in C/C++ and part of the code I develop in assembly, If I use peachpy to write this assembly code, How am I going to debug this assembly code?

In visual studio while using yasm assebler this was possible. Is similar kind of thing can be done in PeachPy?

How PeachPy is different from using intrinsics?
ThankYou

I noticed that you link to my newhope code under "Users" (yay, PeachPy has saved me a lot of time and frustration, thanks).

I split off the ChaCha20 implementation from my newhope package, and am transitioning to a self hosted git solution, so the documentation probably should point to: https://git.schwanenlied.me/yawning/chacha20

Apologies for "not very idiomatic" python/PeacyPy usage (In particular, someone with better Python skills than myself probably could reduce the code duplication). That said, it works, supports both SSE2 and AVX, and is fast.

PBLENDVB is one of a handful of instructions that has an implicit register argument at xmm0. Consider this trivial example:

from peachpy import *
from peachpy.x86_64 import *

with Function("test", (), None, target=uarch.haswell):
    x = XMMRegister()
    y = XMMRegister()
    PBLENDVB(x, y, xmm0)
    RETURN()

This fails with the exception Instruction PBLENDVB requires operand 3 to be allocated to xmm0 register, but xmm0 is a live register. Looking at the check in question, we have

if instruction.name in {"BLENDVPD", "BLENDVPS", "PBLENDVB", "SHA256RNDS2"}:
    assert len(instruction.operands) == 3, \
        "expected 3 operands, got %d (%s)" % \
        (len(instruction.operands), ", ".join(map(str, instruction.operands)))
    xmm0_operand = instruction.operands[2]
    assert isinstance(xmm0_operand, XMMRegister), \
        "expected xmm registers in the 3rd operand, got %s" % str(xmm0_operand)
    # Check that xmm0 is not live at this instruction
    if instruction._live_registers.get(xmm0._internal_id, 0) & XMMRegister._mask != 0:
        raise RegisterAllocationError(
            "Instruction %s requires operand 3 to be allocated to xmm0 register, " +
            "but xmm0 is a live register" % instruction.name)
    if xmm0_operand.is_virtual:
        xmm0_binded_registers.add(xmm0_operand._internal_id)

I have found no way to actually use these instructions, as xmm0 will apparently always be live at the instruction in question.

Curiously, the ROL and related instructions work as expected when using cl as an argument. The difference suggests the possible issue: ROL et al check first that the rotation count operand is a virtual register, whereas PBLENDVB et al do not and always make the liveness check.

Now, if we get past this issue by letting every register be virtual, e.g.,

from peachpy import *
from peachpy.x86_64 import *

with Function("test", (), None, target=uarch.haswell):
    x = XMMRegister()
    y = XMMRegister()
    z = XMMRegister()
    PBLENDVB(x, y, z)
    RETURN()

we run into a different issue: the comparison here

other_xmm0_registers = filter(operator.methodcaller("__ne__", xmm0_register), xmm0_binded_registers)

does not work, because earlier we are pushing integers into xmm0_binded_registers:

xmm0_binded_registers.add(xmm0_operand._internal_id)

Furthermore, in the next line self._conflicting_registers[3][xmm0_register] simply does not exist. Some investigation leads me to believe that self._register_allocators[XMMRegister._kind].conflicting_registers[xmm0_register] is what is intended here, but I'm not entirely sure. Similarly, a few lines below, self._register_allocations[XMMRegister._kind][xmm0_register] = xmm0.physical_id should probably be self._register_allocators[XMMRegister._kind].register_allocations[xmm0_register] = xmm0.physical_id.

Currently PeachPy does not object the syntax xmmN(k0). This is illegal according to Intel spec (aaa = 000 used to encode no mask)

Hi,

I tried to print some information in Register._ init _ to get some information of register allocation. Before it ran into the real codes, there're lots of register allocated. Where do they allocated ? What for ?

Thanks,

peachpy.Loader needs implementation for Native Client, on top of dynacl

The generator outputs an error for functions with signed/unsigned_int argument. Here's the last few lines of the error:

File "/Library/Python/2.7/site-packages/peachpy/x86_64/function.py", line 987, in __init__
    self.go_signature = function.go_signature
  File "/Library/Python/2.7/site-packages/peachpy/x86_64/function.py", line 158, in go_signature
    go_argument_types = list(map(c_to_go_type, map(operator.attrgetter("ctype"), self.arguments)))
  File "/Library/Python/2.7/site-packages/peachpy/x86_64/function.py", line 149, in c_to_go_type
    }[ctype.size]
KeyError: None

It also outputs an error when the return type is signed/unsigned_int. Right now I'm substituting int64_t/uint64_t for signed/unsigned_int and just change the function signature to use int/uint since int and uint type in go is actually represented as 64-bit in amd64 as stated here. It would also be nice if Go's uintptr can also be supported.

Perhaps check for missing -I or somehow tell parser to initialize it to empty?

peachpy/x86_64/main.py
options.include is unset in many of your examples, so line 260:
include_directories = [os.path.abspath(include_dir) for include_dir in options.include]
fails with
TypeError: 'NoneType' object is not iterable

Quickfix: add
include_directories = []
if options.include is not None:

PeachPy appears to support the ALIGN directive to align, e.g., loops, but it seems to have no effect on the output.

Since the code generator needs to know the size of instructions, it should have enough information to work out current (relative) offset to the beginning of the function. On the other hand, Functions do not seem to have any kind of alignment, so they can start anywhere.

Hi,

Looks like an awesome project, but I'm not able to get started.

My env is Win64 with Python3.5

If I take the example on the main page, under "Using PeachPy as a command-line tool" and then I execute the specified line

"python -m peachpy.x86_64 -mabi=ms -mimage-format=ms-coff -o example.obj example.py"

I get errors.:

File "C:\Users\xxx\AppData\Local\Programs\Python\Python35-32\lib\site-packages\peachpy\encoder.py", line 151, in fixed_string
assert isinstance(size, (int, long)) and size > 0, "size %u is not a positive integer" % size
NameError: name 'long' is not defined

I'm finding similar issues for the other examples. I've taken a look, but before I invest any time in seeing if my env is missing something, I thought I should check that I'm even expect to be able to output anything meaningful on my platform.

I'd also love any 'in progress but not ready' documentation you may have on this latest version. There's very little out there, and online examples I see are for the old version.

I've been fiddling with FASMg for a bit now, and this looks like a similar but better idea, and I get to stay in Python, so I'm quite eager to try it out.

1. Collect the list of non-volatile registers written by the function
2. Insert sequences to save the registers in prologue (right after entry point) and restore in epilogue (just before RETURN).
3. If function uses any AVX-mode instructions and does not return __m256, __m256d or __m256i, insert VZEROUPPER before RETURN.
4. If function uses any MMX-mode instructions and does not return __m64, insert EMMS before RETURN.
5. Prologue/epilogue must be generated before RETURN is lowered.

peachpy.Loader needs implementation for Windows

If virtual register is allocated before physical registers, PeachPy can bind it to the same physical register.

I try to use PeachPy to list the assembly of those NNPACK kernels in gas syntax form.
I simply add the flag "-S -assembly-format=gas" to PeachPy command in ninja.build, then run a normal build.
And the building process can't pass one line of assert: https://github.com/Maratyszcza/PeachPy/blob/master/peachpy/x86_64/pseudo.py#L63
Obviously, gas won't match the word gnu.

In the abi_map, I can't find the gnu format, so I guess the gnu tag might hold the same meaning as gas.
https://github.com/Maratyszcza/PeachPy/blob/master/peachpy/x86_64/__main__.py#L38

PeachPy-generated code does not obey NaCl SFI restrictions: it crosses bundle boundaries. Needs to be fixed.

https://golang.org/pkg/builtin/#uintptr

It seems to be this can map directly to uintptr_t ?

diff --git a/peachpy/x86_64/function.py b/peachpy/x86_64/function.py
index f4614d6..00c81b9 100644
--- a/peachpy/x86_64/function.py
+++ b/peachpy/x86_64/function.py
@@ -130,6 +130,8 @@ class Function:
                 return "boolean"
             elif c_type.is_size_integer:
                 return "int" if c_type.is_signed_integer else "uint"
+            elif c_type.is_pointer_integer:
+                return "uintptr"
             elif c_type.is_signed_integer:
                 return {
                     1: "int8",

My rationale for wanting this is to make it easier to have slices and structs as arguments.

A slice header ( https://golang.org/pkg/reflect/#SliceHeader ) can be simulated by having three arguments,

s = Argument(ptr(uintptr_t))
s_len = Argument(int64_t)
s_cap = Argument(int64_t)

(I know the documentation says to use ptrdiff_t to get Go's int, but that really makes the code confusing... In my code I'm using int64 explicitly because I know I'm on 64-bit platform.)

However, due to the lack of mapping for uintptr, this fails when trying to generate the commented function header -- I have to put an explicit integer type for s.

Similarly if I want to pass a pointer to a struct, using uintptr_t seems the only sane option.

i follow the readme to use asm in python on win7 64bit ultimate like below:

from peachpy.x64 import *

abi=peachpy.c.ABI("x64-ms")
assembler = Assembler(abi)
x = peachpy.c.Parameter("x",peachpy.c.Type("int32_t"))
y = peachpy.c.Parameter("y",peachpy.c.Type("int32_t"))
with Function(assembler,"DotProduct", (x, y), "Unknown") as asm_function:
    reg_x = GeneralPurposeRegister64()
    reg_y = GeneralPurposeRegister64()

    LOAD.PARAMETER(reg_x, x)
    LOAD.PARAMETER(reg_y, y)

    ADD(reg_x ,reg_y)

    RETURN()

python_function = asm_function.finalize(abi.detect()).encode().load()

print(python_function(2, 2)) 

it occur attributeerror: 'Function' object has no attribute 'finalize', something wrong with my code? help !

I have a 4th-generation Intel Haswell CPU on Windows. I want to use PeachPy to speed up the inner loops of my code written in Python.

After reading several papers on PeachPy, I haven't been able to understand whether execution of PeachPy-generated assembly code in Python is:
a) full-speed using native CPU instructions
b) emulated

a) implies that PeachPy could offer higher execution speed than Cython, without a separate assembler.
b) implies that PeachPy could be used to cross-develop and functional test for all supported architectures on one machine, but a separate assembler step is needed for each platform and OS.

Both of these are good goals. Which one describes PeachPy?

Problems with the current code:

• Functionality is very complex and scattered all over the place
• There are likely bugs and/or inefficiencies
• Unclear how it works with Golang asm
• Need something like -fomit-frame-pointer
• Does not use read zone
• Inefficient addressing of storage for clobbered xmm registers when stack is realigned (store/load instructions use rsp-based addressing instead of rbp)
• Argument addressing does not switch to rbp when stack is realigned

Analysis requires two passes: a forward pass and a backward pass.
Forward pass:

1. Initially no state is active
2. If an AVX-mode instruction is observed (except VZEROUPPER or VZEROALL), assume avx state.
3. If an SSE-mode instruction is observed, assume sse state. If SSE-mode instruction is observed in avx state, report an error.
4. If a VZEROUPPER or VZEROALL instruction is observed, clear the state.
5. If an instruction has no AVX/SSE mode, assign the current state
   Backward pass:
6. If an instruction with sse or avx state is observed, change the current state to that value.
7. If an instruction without a state is observed, assign its mode to the current value.

Later, this information can be used in instruction lowering in the following way:

1. If an instruction has an avx state, use AVX form for lowering
2. If an instruction has an sse state or no state, use SSE form for lowering

Get this error ->
python -m peachpy.x86_64 -mabi=sysv -mimage-format=elf -o matmul.o matmul-opt.py
/usr/bin/python: No module named avx; 'peachpy.x86_64' is a package and cannot be directly executed

Also running make gives this ->
user@ne:~/Downloads/PeachPy-master/examples/make$ make
python -m peachpy.x86_64 -MMD -MF matmul-opt.d -o matmul-opt.o matmul-opt.py
/usr/bin/python: No module named avx; 'peachpy.x86_64' is a package and cannot be directly executed
Makefile:12: recipe for target 'matmul-opt.o' failed
make: *** [matmul-opt.o] Error 1

This is on ubuntu wily with built in Python 2.7.10.

I'm trying to move a constant value to an xmm register through these instructions:

    mask = Constant.uint64x2(0xFF, 0xFF, 'mask')
    MOVDQU(xmm0, mask)

which produces this instruction:

    MOVOU const0xFF00000000000000FF00000000000000(SB), X0

This will output const0xFF00000000000000FF00000000000000: not defined in Go. How can I generate code like this instead?

    DATA     mask<>+0(SB)/8, $0xFF
    DATA     mask<>+8(SB)/8, $0xFF
    GLOBL    mask<>(SB), 8, $16
    ...
    ...
    ...
    MOVOU mask<>(SB), X0