Optimize kernel compile time in cross-compile environment about raspberry-pi-pcie-devices HOT 11 CLOSED

Wonder if it'll be any faster in native AArch64 compile on the new Apple Silicon Macs, not that I'd recommend buying them considering a 10 minute compile time on your current one is already pretty good.

from raspberry-pi-pcie-devices.

As a follow-on, rather than having to mess with ejecting SD cards, mounting them, copying, unmounting, and putting the SD card back in.

• do the modules_install and copy of image files and DTBS to local directories (keep them separate)
• (if you want to SSH in with a password, enable SSH for root on the Pi (sudo nano /etc/ssh/sshd_config and change PermitRootLogin yes. Use sudo passwd root to set the root password)
• reboot or restart sshd
• copy your SSH public key to /root/.ssh/authorized_keys so it doesn't prompt for a password
• use scp -r or rsync to copy all the files across your network

It's also far easier to do it that way on a CM with onboard EMMC than messing with rpiboot to mount the EMMC as a USB storage device.

(edited to make enabling root password optional. It's a significant security risk to do so).

from raspberry-pi-pcie-devices.

First attempt: I dropped the job count (using -j), and increased the VirtualBox vb.cpus to 6, and let it rip.

Things got... shaky.

A lot of things would compile, and the VM was using ~570% CPU the whole time, but I noticed a lot of compile warnings and errors, like:

  CC [M]  drivers/media/common/siano/smscoreapi.o
  CC      drivers/mfd/arizona-core.o
compilation terminated.
compilation terminated.
compilation terminated.
  CC [M]  net/ipv6/netfilter/nf_conntrack_reasm.o
  AR      drivers/iio/resolver/built-in.a
aarch64-linux-gnu-gcc: fatal error: Killed signal terminated program cc1
compilation terminated.
aarch64-linux-gnu-gcc: fatal error: Killed signal terminated program cc1
aarch64-linux-gnu-gcc: fatal error: Killed signal terminated program cc1
aarch64-linux-gnu-gcc: fatal error: Killed signal terminated program cc1
aarch64-linux-gnu-gcc: fatal error: Killed signal terminated program cc1
compilation terminated.
compilation terminated.
compilation terminated.

from raspberry-pi-pcie-devices.

From this comment by norse-dougbert:

Don't know how many cpus on your build machine, but after years of kernel building my make -j $PARALLEL parameter became 1.6 times actual cpus, hence 8 cores became -j 12 or 13. Of course drives are much faster now then back then

So going to try -j10 on another build after resetting the environment.

from raspberry-pi-pcie-devices.

Results:

$ time make -j10 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image modules dtbs
...
real	10m12.872s
user	55m31.871s
sys	5m29.739s

CPU load was around 580% from VBoxHeadless, laptop got... very nice and toasty. It was on my lap, probably should not do that next time!

But 10 minutes! My Mac is running faster than the Focus was now—though I didn't optimize the runs on the Focus either. I'm going to crank it up, giving the VM 8 cores and going to 14 jobs!

from raspberry-pi-pcie-devices.

Whee! That got the laptop cranking... 94°C and ~750% avg CPU usage. I'm also going to add an optimization to the vagrantfile to automatically set half the logical cores on the machine.

But not a huge gain, all said and done:

real	9m11.867s
user	65m27.295s
sys	6m48.170s

I might go a little insane and try consuming more. Bad idea? Maybe. For science? It must be done.

from raspberry-pi-pcie-devices.

Now trying with 14 logical cores (using updated algorithm of [total logical cores] - 2 and:

make -j22 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image modules dtbs

Heh... CPU up to ~1080%, temp went straight up to 94°C again within 30s. Let's see how fast this thing will run, though! Here's the result:

real	9m47.122s
user	118m15.642s
sys	15m35.538s

(As an aside: this kills the battery.)

from raspberry-pi-pcie-devices.

Looks like diminishing returns (or even slower process) once we pass up 20 concurrent jobs. Trying again with 14 logical cores and 16 jobs:

real	11m23.431s
user	114m36.113s
sys	14m20.233s

So maybe VirtualBox's warning about using half the logical cores (same as number of physical cores) has something to it!

from raspberry-pi-pcie-devices.

Back to 8 CPU cores (logical CPU count / 2), and -j12:

$ time make -j12 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image modules dtbs
...
real	9m16.682s
user	65m53.267s
sys	6m51.502s

I think I'll push a commit with these changes and move on from there!

from raspberry-pi-pcie-devices.

@6by9 - That's almost exactly what I'm doing now: https://github.com/geerlingguy/raspberry-pi-pcie-devices/tree/master/extras/cross-compile#copying-built-kernel-via-remote-sshfs-filesystem

It was especially important as I rearranged my desk and now it's almost impossible to get to my card reader (I had to order a new one with a long cable so I can put it back on top of my desk)... much easier to not have to swap cards!

from raspberry-pi-pcie-devices.

Did you think about putting the whole builddir/container/vm in a tmpfs? 🚀😄
Will probably not make it much faster but as the CPU seems to have reached its limit, this could squeeze out some more seconds 😄😄

from raspberry-pi-pcie-devices.