coatless-mac / r-macos-rtools Goto Github PK

As of R 3.4.* , the state of the toolchain changes with every release. The main component being changed is the compiler used to build the R CRAN binaries.

In particular, we have:

• R 3.7.* uses: Clang 8.0.0 for OS X 10.11
  • Found at: /usr/local/clang8
• R 3.6.* uses: Clang 7.0.0 for OS X 10.11
  • Found at: /usr/local/clang7
• R 3.5.* uses: Clang 6.0.0 for OS X 10.11
  • Found at: /usr/local/clang6
• R 3.4.* uses: Clang 4.0.0 for OS X 10.11
  • Found at: /usr/local/clang4

The gfortran binaries are largely stable:

• R 3.4.* - Present: GNU Fortran 6.1 for OS X 10.11
• Before that, there was a highly custom gfortran that is still problematic to handle...

Going forward, we'll likely need to:

1. Detect the R version;
2. Scope the gfortran installer used; and
3. Clean up remnants.

I get the same set of messages when I attempt to update the move package or install terra from GitHub (https://github.com/rspatial/terra) so it seems like there is a path problem somewhere but I can't tell where or how to fix.

Here's the start of the error messages.

• installing source package ‘move’ ...
  ** package ‘move’ successfully unpacked and MD5 sums checked
  ** using staged installation
  ** libs
  clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I'/Library/Frameworks/R.framework/Versions/4.0/Resources/library/Rcpp/include' -I/usr/local/include -fPIC -Wall -g -O2 -c bgbVar.c -o bgbVar.o
  clang++ -mmacosx-version-min=10.13 -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I'/Library/Frameworks/R.framework/Versions/4.0/Resources/library/Rcpp/include' -I/usr/local/include -fPIC -Wall -g -O2 -c bgbVarRcpp.cpp -o bgbVarRcpp.o
  clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I'/Library/Frameworks/R.framework/Versions/4.0/Resources/library/Rcpp/include' -I/usr/local/include -fPIC -Wall -g -O2 -c bgb_bbmm.c -o bgb_bbmm.o
  clang++ -mmacosx-version-min=10.13 -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I'/Library/Frameworks/R.framework/Versions/4.0/Resources/library/Rcpp/include' -I/usr/local/include -fPIC -Wall -g -O2 -c emd.cpp -o emd.o
  In file included from emd.cpp:1:
  In file included from /Library/Frameworks/R.framework/Resources/include/R.h:50:
  In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cmath:304:
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/math.h:760:41: error: no member named 'fabsf' in the global namespace; did you mean 'labs'?
  abs(float __lcpp_x) _NOEXCEPT {return ::fabsf(__lcpp_x);}
  ~~^
  /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/stdlib.h:148:7: note: 'labs' declared here
  long labs(long) __pure2;
  ^
  In file included from emd.cpp:1:
  In file included from /Library/Frameworks/R.framework/Resources/include/R.h:50:
  In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cmath:304:
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/math.h:764:42: error: no member named 'fabs' in the global namespace
  abs(double __lcpp_x) _NOEXCEPT {return ::fabs(__lcpp_x);}

I guess the installer has to be updated for Mac OS Mojave.
It gets sucked in Running package scripts for 5+ min and then fails the installation.
It just suggest to contact the developer. Could you help me?

Hi,
I tried downloading the R toolchain for mac OS high sierra and it failed. I have the updated version of RStudio. Not sure why it won't work.

Thanks,
Hannah

everytime I try to install it through the installer for mac catalina I have "installation fail: installer encountered an error ... contact the software manufacturer ..)

Hello! I was unsuccessfully attempting to install rstanarm on Mac OSX 10.15.3, and I was instructed to use this toolchain to fix the issue I was encountering.

Unfortunately, after installing the C++ toolchain and attempting to install rstanarm in a fresh session, compilation failed reading the math.h header:

In file included from /usr/local/clang7/include/c++/v1/cmath:305:
/usr/local/clang7/include/c++/v1/math.h:301:15: fatal error: 'math.h' file not found
#include_next <math.h>
^~~~~~~~

1 error generated.
make: *** [stan_files/bernoulli.o] Error 1
rm stan_files/bernoulli.cc
ERROR: compilation failed for package ‘rstanarm’

• removing ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library/rstanarm’
• restoring previous
  ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library/rstanarm’

As the error suggests, there is a file at /usr/local/clang7/include/c++/v1/math.h that attempts to include another file called math.h on line 301. Does anyone have any idea how or why math.h might fail to read another file called math.h?

The goal here is to provide a second level of security related to #8 by checking the downloaded packages are signed.

Consider:

pkgutil --check-signature $PKG_INSTALLER_NAME

Package "macos-rtools.pkg":
   Status: signed by a certificate trusted by Mac OS X
   Certificate Chain:
    1. Developer ID Installer: James Balamuta (LNVWJ4S7DC)
       SHA1 fingerprint: 4B CF 66 88 56 AC 73 E8 70 14 5C 2C 7E BA E2 67 90 3C 85 BF
       -----------------------------------------------------------------------------
    2. Developer ID Certification Authority
       SHA1 fingerprint: 3B 16 6C 3B 7D C4 B7 51 C9 FE 2A FA B9 13 56 41 E3 88 E1 86
       -----------------------------------------------------------------------------
    3. Apple Root CA
       SHA1 fingerprint: 61 1E 5B 66 2C 59 3A 08 FF 58 D1 4A E2 24 52 D1 98 DF 6C 60

If status is not "signed by a certificate trusted by Mac OS X", then an exit is triggered.

Thus, we likely will want to verify the Developer ID (full name + ID):

1. Developer ID Installer: James Balamuta (LNVWJ4S7DC)
1 %     2   % 3%  4       %  5  %    6   %     7

The following extract out the name and the ID

PKG_AUTHOR=`pkgutil --check-signature $PKG_INSTALLER_NAME | awk /'Developer ID Installer/{print $5" "$6}'`
PKG_DEV_ID=`pkgutil --check-signature $PKG_INSTALLER_NAME | awk /'Developer ID Installer/{print $7}'  | tr -d '()'`

In Simon's case, we have:

pkgutil --check-signature ~/Downloads/clang-6.0.0.pkg 

Package "clang-6.0.0.pkg":
   Status: signed by a certificate trusted by Mac OS X
   Certificate Chain:
    1. Developer ID Installer: Simon Urbanek (VZLD955F6P)
       SHA1 fingerprint: 7B 6B 81 12 E6 26 8C 16 F8 D4 0F 94 E4 3E 62 69 2E 92 22 81
       -----------------------------------------------------------------------------
    2. Developer ID Certification Authority
       SHA1 fingerprint: 3B 16 6C 3B 7D C4 B7 51 C9 FE 2A FA B9 13 56 41 E3 88 E1 86
       -----------------------------------------------------------------------------
    3. Apple Root CA
       SHA1 fingerprint: 61 1E 5B 66 2C 59 3A 08 FF 58 D1 4A E2 24 52 D1 98 DF 6C 60

Alternatively, we probably should verify just the SHA1 fingerprint

if pkgutil --check-signature $PKG_INSTALLER_NAME | grep 'SHA1 fingerprint: 4B CF 66 88 56 AC 73 E8 70 14 5C 2C 7E BA E2 67 90 3C 85 BF'; then
    echo 0
else
    exit 1
fi

• Remove box url
• Use go.illinois.edu/ to generate a short link
• Update in README.md

The installation failed.

The Installer encountered an error that caused the installation to fail. Contact the software manufacturer for assistance.

From stan-dev/rstan#716 (comment),

We need to verify write ability of ~/.R/Makevars and ~/.Renviron.

We'll likely need to change it from root back to $USER.

Address robustness with hardening
Consider adding a "bundled" offline installer by merging existing .pkg builds

Details on improving the installer build process can be found inside of the 2019 MacAdmins conference talk by Rich Trouton.

Installer Package Scripting: Making your deployments easier, one ! at a time – Rich Trouton: https://www.youtube.com/watch?v=fIkh7RzrjqY

Slides for the talk are at:
https://bpb-us-e1.wpmucdn.com/sites.psu.edu/dist/4/24696/files/2019/07/psumac2019-345-Installer-Package-Scripting-Making-your-deployments-easier-one-at-a-time.pdf

And a cached copy can be found here:

psumac2019-345-Installer-Package-Scripting-Making-your-deployments-easier-one-at-a-time.pdf

Sent a tweet out to him as well:

https://twitter.com/axiomsofxyz/status/1613307509271306242

Thanks a lot for creating this tool!

I found it necessary to add /usr/local/gfortran/bin to the PATH in .Renviron in order to compile packages containing fortran code. Before adding that to the path I got

make[1]: gfortran: No such file or directory

This was on macOS Catalina version 10.15.2 with a fresh R-devel 4.0.0 while attempting to install package ucminf from CRAN.
It is entirely possible that I'm just missing something as I'm in over my head here, but thought I'd mention it.

Cheers
Rune

Thanks for the great installer. I just wondered for which macOS versions it works.

We recently upgraded a machine in our lab to macOS Monterey, and since then compiling R packages does no longer work. Is this installer supposed to work under Monterey? Or is it independent of the macOS version?

Thanks again,

Tim

clang6 installer doesn't automatically set the path for users... 😡

• Determine where to set path:
  • Verify path does not already have the appropriate bin statement
    TBA
  • Check for the kind

case $SHELL in
*/zsh) 
   shell_type="zsh"
   ;;
*/bash)
   shell_type="bash"
   ;;
*)
   exit 1 # error'd 
esac

• If zsh exists, then write to ~/.zshrc otherwise, write to ~/.bash_profile or /etc/path.d/clang6`
• Note: Writing to /etc/path.d/ is preserved during a system upgrade compared to /etc/path.

In the ~/.bash_profile or ~/.zshrc, append to the end:

if [ -d "/usr/local/clang6/bin" ] ; then
    export PATH="/usr/local/clang6/bin:$PATH"
fi

Consider including Java:

# Headless download
# curl ... 

# This URL seems unstable.

# Then do a hdi mount/install/eject

# Run javareconf
R CMD javareconf

I can't install on my mac: Apple can't check for malicious software.

Thanks for writing this easy to use installer!

I upgraded R from 3.5.2 to 3.6.1 and used it to update a couple packages requiring clang and gfortran (data.table 1.12.4 and KernSmooth 2.23.16, respectively). However, by default the path to the gfortran installation was not added to the .Renviron file, and thus I was still getting the error "gfortran not found".

Adding it to the .Renviron file fixed this. It may be helpful to include writing this path to the .Renviron file as a default along with clang.

Hi,
I got it opened!
But the installation failed. I am on macOS Big Sur 11.0.1.
I took a screen shot:

Originally posted by @paprika920 in #39 (comment)

My MAC tells me it cannot open the package as it is unable to scan it for malicious software. It says the package needs to be updated.

From #23, I think we might have to notarize the installer app.

Details:

• https://developer.apple.com/developer-id/
• https://appleinsider.com/articles/19/09/03/apple-eases-app-notarization-conditions-ahead-of-macos-catalina-release

I may not have been paying attention while the installer was running and hence missed a notification or comment about this, but it tripped me up for a bit.

Specifically, I somehow missed that my old .Renviron was moved to .Renviron.bck and a new file created. It broke some other stuff that I wasn't expecting and it took me a while to piece together that this is what had happened.

With such an easy to use (and appreciated!) tool like this with everything packaged up into a macOS installer, it might make sense to be more explicit about what's going to happen to the contents of their .Renviron file. Again, it's possible I missed it because I was just clicking through the installer, but that might be the sort of user behavior you're going to see with a tool like this.

Thanks again for building this!

No documentation given for failure.

Per Simon's e-mail on R-SIG-MAC, we're going to implement a check that verifies the md5 hash of the file.

Sample function:

#1 Path to File
#2 md5 Hash to Check Against
check_md5_hash() {
	
    # Obtain file hash
    FILE_HASH=$(md5 $1 | awk '{print $4}')
    
    # MD5 (path/to/file.pkg) = c29700c4e7b2914073ef7e741eb105bc
    #  1 %         2        %3%      4 
     
    # Check against hash
    if [ "$FILE_HASH" = "$2" ]; then
       echo 0
    else 
       exit 1
    fi
}

Hard coded hashes:

• clang-4.0.0-darwin15.6-Release.tar.gz: 2bec4fbae8d9caf6499c941ef9b87eae
  • Downloaded from: http://r.research.att.com/libs/clang-4.0.0-darwin15.6-Release.tar.gz
• gfortran (OS dependent)
  • gfortran-6.1-ElCapitan.dmg: 457b59a6453069cf72dee9a4f9bf1b0a
    • Downloaded from: http://coudert.name/software/gfortran-6.1-ElCapitan.dmg
  • gfortran-6.3-Sierra.dmg: 1849cea667bb714c5c04a8565a9fe231
    • Downloaded from: http://coudert.name/software/gfortran-6.3-Sierra.dmg

Not much info given on failed page

The installer encountered an error that caused the installation to fail. Contact the software manufacturer for assistance.

I have already Xcode, Clang and Gfortran installed, could that be the problem?

Thank you

Need to show the installer isn't timing out just requiring more time...

Download and install the clang6.pkg via:

curl -O https://cran.r-project.org/bin/macosx/tools/clang-6.0.0.pkg
sudo installer -pkg clang-6.0.0.pkg -target /
rm -rf clang-6.0.0.pkg

The macOS-rtools-4.0.0.pkg failed to install on my MacBook Air M2. See image.
I have installed R 4.2.2 GUI 1.79 High Sierra build (intel).
what to do?

double clicking the installer pkg results in this message getting displayed:

“macos-rtools-3.2.2.pkg” can’t be opened because Apple cannot check it for malicious software.

I'm running macOS Catalina, 10.15.3

Probably should investigate testing the installation routine via the new GitHub Actions.

Here's the program from the coatless prof

#include <RcppArmadillo.h>   

// [[Rcpp::depends(RcppArmadillo)]]

// [[Rcpp::export]]
void hello_world() {
  Rcpp::Rcout << "Hello World!" << std::endl;  
}

// After compile, this function will be immediately called using
// the below snippet and results will be sent to the R console.

/*** R
hello_world() 
*/

Here's the beginning of the error messages

clang++ -mmacosx-version-min=10.13 -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include -I"/Library/Frameworks/R.framework/Versions/4.0/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/4.0/Resources/library/RcppArmadillo/include" -I"/Users/gcn/Documents/workspace/ISIMIPData/temp" -I/usr/local/include -fPIC -Wall -g -O2 -c helloworld.cpp -o helloworld.o
In file included from helloworld.cpp:1:
In file included from /Library/Frameworks/R.framework/Versions/4.0/Resources/library/RcppArmadillo/include/RcppArmadillo.h:31:
In file included from /Library/Frameworks/R.framework/Versions/4.0/Resources/library/RcppArmadillo/include/RcppArmadilloForward.h:26:
In file included from /Library/Frameworks/R.framework/Versions/4.0/Resources/library/Rcpp/include/RcppCommon.h:29:
In file included from /Library/Frameworks/R.framework/Versions/4.0/Resources/library/Rcpp/include/Rcpp/r/headers.h:67:
In file included from /Library/Frameworks/R.framework/Versions/4.0/Resources/library/Rcpp/include/Rcpp/platform/compiler.h:100:
In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/cmath:304:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/math.h:760:41: error: no member named 'fabsf' in the global namespace
abs(float __lcpp_x) _NOEXCEPT {return ::fabsf(__lcpp_x);}
~~^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/math.h:764:42: error: no member named 'fabs' in the global namespace; did you mean 'abs'?
abs(double __lcpp_x) _NOEXCEPT {return ::fabs(__lcpp_x);}
~~^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/math.h:760:1: note: 'abs' declared here
abs(float __lcpp_x) _NOEXCEPT {return ::fabsf(__lcpp_x);}
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/math.h:768:47: error: no member named 'fabsl' in the global namespace
abs(long double __lcpp_x) _NOEXCEPT {return ::fabsl(__lcpp_x);}

And the final error message is

make: *** [helloworld.o] Error 1
Error in Rcpp::sourceCpp("temp/helloworld.cpp") :
Error 1 occurred building shared library.

From @jimhester on rOpensci slack:

Why do you modify ~/.R/Makevars rather than the site level R_HOME/etc/Makevars.site?

• Consider generalizing the bash script to allow for site configuration...
• Test to see if a new install of R overrides the Makevars.site value.
• Add a line to the README.md about this?

Incorporate in the headless Xcode CLI install for Mojave.

sudo installer -pkg \
/Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg \
-target /

Avoid messing with Makevars via:

sudo touch ~/.R/Makevars && sudo echo 'CPPFLAGS="-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include"' >> ~/.R/Makevars

c.f. https://twitter.com/axiomsofxyz/status/1045463113863180290, https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#macOS

From Dealing with Catalina and Dealing with Catalina II, we need to modify how the compiler is setup to ease the C++ exception issue.

@bgoodri seems to have found a good option here:

CXX14 = /usr/local/clang7/bin/clang++ -stdlib=libc++ -nostdinc++ -I/usr/local/clang7/include/c++/v1
SHLIB_CXX14LD = /usr/local/clang7/bin/clang++ -L/usr/local/clang7/lib/

Note, the -nostdinc++ is disabling standard directorys, c.f. https://gcc.gnu.org/onlinedocs/gcc/Directory-Options.html

-nostdinc
Do not search the standard system directories for header files. Only the directories explicitly specified with -I, -iquote, -isystem, and/or -idirafter options (and the directory of the current file, if appropriate) are searched.

-nostdinc++
Do not search for header files in the C++-specific standard directories, but do still search the other standard directories. (This option is used when building the C++ library.)

Poking softwareupdate on macOS 10.14 gives:

* Command Line Tools (macOS Mojave version 10.14) for Xcode-10.3
        #        Command Line Tools (macOS Mojave version 10.14) for Xcode (10.3), 199140K [recommended]

With macOS Catalina 10.15, the output changed to:

* Label: Command Line Tools for Xcode-11.3
	Title: Command Line Tools for Xcode, Version: 11.3, Size: 224878K, Recommended: YES, 

We initially attempted to fix this by splitting on OS:

https://github.com/rmacoslib/r-macos-rtools/blob/8816c5c0a37f9db3177db1406d07b38e5f8083da/scripts/postinstall#L68-L83

I think the issue here is we're using head instead of tail.

Elsewhere around the macOS community, homebrew approaches the headless install with:

  clt_label_command = "/usr/sbin/softwareupdate -l | " \
                      "grep -B 1 -E 'Command Line Tools' | " \
                      "awk -F'*' '/^ *\\*/ {print $2}' | " \
                      "sed -e 's/^ *Label: //' -e 's/^ *//' | " \
                      "sort -V | " \
                      "tail -n1"
  clt_label = `#{clt_label_command}`.chomp

This is limited to macOS 10.13 and above.

Related to #16

From Stan user jroon on their discourse setup:

Catalina 10.15.2
R 3.5.1

Does not adhere to:

# clang: start
SHLIB_CXXLDFLAGS+=-Wl,-rpath,${R_HOME}/lib ${R_HOME}/lib/libc++abi.1.dylib
SHLIB_CXX14LDFLAGS+=-Wl,-rpath,${R_HOME}/lib ${R_HOME}/lib/libc++abi.1.dylib
# clang: end

https://discourse.mc-stan.org/t/dealing-with-catalina-ii/11802/93