Implement cross-compilation to Android

[brson]

We have Android code in tree but no way to build it. It should not be all that difficult to integrate the NDK since we have some minimal cross-compile support already.

[amuxtux]

the libraries to be cross compiled are

1. libuv (for librustrt.so as dependency ), Its already here https://github.com/ILyoan/libuv
2. libcore.so
3. librustrt .so
4. libstd.so

Other tasks:

1. setup standalone ndk toolchain
   http://www.kandroid.org/ndk/docs/STANDALONE-TOOLCHAIN.html
2. specifying NDK tool chain possibly in configure phase

[brson]

As mentioned previously, this is my branch where I have been hacking on Rust's cross-compile: https://github.com/brson/rust/commits/cross2

All it does so far is add CFG_HOST_TRIPLES and CFG_BUILD_TRIPLE. Under this scheme, the target triples would be systems that we build core and std, and for the host triples we would build the entire compiler. The build triple is the architecture we bootstrap from.

[yichoi]

How can one specify multiple host-triples or target-triples? Separating with a space doesn't seem to work.

[yichoi]

It looks to me like that "for loop" of target-triples make each target directory. Assuming arm-unknown-android as target triples and x86_64-apple-darwin as host-triples, One possible way to make build directory is to adding host-triples value to target-triples, the other possible way is to modify configure script to handle host-triples and target-triples separately.

[brson]

@yichoi on the command line they are specified like ../configure --target-triples=x86_64-unknown-linux-gnu,i686-unknown-linux-gnu

[brson]

@yichoi yeah, the host and target triples could be disjoint, though I've been assuming that target triples are a superset of host triples.

[brson]

@yichoi I pushed a cleanup commit to my cross2 branch that deletes a bunch of code to make further changes easier: brson@abb5c65

I'm going to do a little more cleanup to my cross2 branch to make sure it works like I think it does (I'm testing e.g. configure --host-triples=x86_64-unknown-linux-gnu --target-triples=x86_64-unknown-linux-gnu,i686-unknown-linuxgnu with the intent that we build cross-compiled libraries, but not a cross-compiled compiler, for the non-host-triple target-triples. I'd like to know what you think about this approach. It might be best if you take over this work since you are more familiar with what exactly we need to do for Android.

I have some examples below of how I imagine the configuration should work and the directory structure it should produce.

In the makefiles on cross2 ''host' platforms are those for which we build ompilers; 'target' platforms are the platforms are the ones the 'host' compilers target.

As an example, in a scenario where CFG_HOST_TRIPLES=x86_64-unknown-linux-gnu and CFG_TARGET_TRIPLES=x86_64-unknown-linux-gnu arm-unknown-android I would expect to see final directory output like:

x86_64-unknown-linux-gnu/
  bin/
    - x86_64 compiler, etc.
  lib/
    - x86_64 libs
    rustc/
      x86_64-unknown-linux-gnu/
      arm-unknown-android/
        bin/
          - no contents here since all our binaries depend on librustc and we won't be building a rustc to run on arm
        lib/
          librt
          libcore
          libstd      

In an even more complex scenario where we are building the entire compiler for multiple architectures (as indicated by multiple host triples) we would expect the following output (and the current build does this sort of thing already).

x86_64-unknown-linux-gnu/
  - as above
i686-unknown-linux-gnu/
  bin/
  lib/
    rustc/
      x86_64-unknown-linux-gnu/
      arm-unknown-android/
        bin/
          - no contents here since all our binaries depend on librustc and we won't be building a rustc to run on arm
        lib/
          librt
          libcore
          libstd

Some next steps (I'll leave these to you if you like):

• Each target triple needs to have it's own variables for controlling the C++ toolchain. A lot of this is defined in platform.mk
• The configure script needs a flag like --enable-android-ndk=${path} to configure the above
• rustc's linking logic in rustc::back::link will need to be modified. This code tends to be fairly platform-specific, but since the NDK uses gcc we may be able to get away with a simple change to specify which instance of gcc rustc should call to do the linking, maybe with a flag like --cc-name. If that isn't sufficient we may need to teach rustc about the android NDK itself, adding an --android-ndk-path= or similar. Perhaps this has been added already and just needs to be wired up to the makefiles.

[brson]

@graydon @nikomatsakis you may be interested in this subject

[brson]

I did some tweaks and testing to my branch and it gets through the full build under several cross-compile scenarios (just using x86 and i686), but tests.mk isn't quite right. I'm going to leave it as is until I get some feedback from @yichoi.

[brson]

@nikomatsakis asked on IRC why we don't just build the compiler for ARM. That would allow us to leave the build as it is. I had a strong belief that one wouldn't usually want a compiler to run on ARM, but that may not be true. It could be simpler to build compilers everywhere.

As an alternative to making this distinction between 'host' and 'target', which effectively just changes which binaries are built, we could teach the configure script to disable components or groups of components per target, and disable the ARM compiler by default.

[yichoi]

In case of x86_64-unknown-linux-gnu as host-triples, arm-unknown-android as target-triples. I've copied host-triples value to target-triples then made almost works except libstd of Rust building at stage2. I am investigating about that.

x86_64-unknown-linux-gnu/stage2/bin/rustc --cfg stage2 -O --target=arm-unknown-android -o x86_64-unknown-linux-gnu/stage2/lib/rustc/arm-unknown-android/lib/libstd.so /home/yichoi/rust_latest/src/libstd/std.rc && touch x86_64-unknown-linux-gnu/stage2/lib/rustc/arm-unknown-android/lib/libstd.so
Call result #4 has unhandled type i32
UNREACHABLE executed at /home/yichoi/rust_latest/src/llvm/lib/CodeGen/CallingConvLower.cpp:165!
Stack dump:
0. Running pass 'Function Pass Manager' on module 'std.rc'.

1.  Running pass 'ARM Instruction Selection' on function '@rust_uv_ip4_addr__c_stack_shim'
   

In case of x86_64-apple-darwin as host-triples, arm-unknown-android as target-triples, I am thinking a way to specify CFG_LIB_NAME=lib$(1).so or CFG_LIB_NAME=lib$(1).dylib for each target-triples. It might need to expand configuration method in platform.mk

[yichoi]

you may check https://github.com/yichoi/rust/commits/cross3. However libstd.so cannot built yet.

[ILyoan]

I believe this is because of foreign function abi.
It used to use x86 cabi to register foreign function when target was given as arm-android, but now it is default (cabi::llvm_abi_info).
I don't know the difference between two exactly, but after reverted to old setting it compiles well.
And probably we might need to implement cabi_arm.rs ?

[yichoi]

https://github.com/yichoi/rust/commits/cross3 works with cabi_arm.rs regression patch.
@ILyoan will send pull request of cabi_arm.rs regression patch.

[brson]

@yichoi Configuring with --target-triples=x86_64-unknown-linux-gnu,i686-unknon-linux-gnu I see a bunch of lines like:

/home/brian/dev/rust3/mk/target.mk:84: warning: overriding commands for target `x86_64-unknown-linux-gnu/stage0/lib/rustc/x86_64-unknown-linux-gnu/lib/libmorestack.a'
/home/brian/dev/rust3/mk/target.mk:84: warning: ignoring old commands for target `x86_64-unknown-linux-gnu/stage0/lib/rustc/x86_64-unknown-linux-gnu/lib/libmorestack.a'
/home/brian/dev/rust3/mk/target.mk:84: warning: overriding commands for target `x86_64-unknown-linux-gnu/stage0/lib/rustc/x86_64-unknown-linux-gnu/lib/librustrt.so'
/home/brian/dev/rust3/mk/target.mk:84: warning: ignoring old commands for target `x86_64-unknown-linux-gnu/stage0/lib/rustc/x86_64-unknown-linux-gnu/lib/librustrt.so'
/home/brian/dev/rust3/mk/target.mk:84: warning: overriding commands for target `x86_64-unknown-linux-gnu/stage0/lib/rustc/x86_64-unknown-linux-gnu/lib/libcore.so'
/home/brian/dev/rust3/mk/target.mk:84: warning: ignoring old commands for target `x86_64-unknown-linux-gnu/stage0/lib/rustc/x86_64-unknown-linux-gnu/lib/libcore.so'
/home/brian/dev/rust3/mk/target.mk:84: warning: overriding commands for target `x86_64-unknown-linux-gnu/stage0/lib/rustc/x86_64-unknown-linux-gnu/lib/libstd.so'
/home/brian/dev/rust3/mk/target.mk:84: warning: ignoring old commands for target `x86_64-unknown-linux-gnu/stage0/lib/rustc/x86_64-unknown-linux-gnu/lib/libstd.so'

Probably has to do with this line at the beginning of the output indicating I have duplicate target triples: cfg: target triples x86_64-unknown-linux-gnu x86_64-unknown-linux-gnu i686-unknown-linux-gnu

Then later the build fails with:

x86_64-unknown-linux-gnu/stage0/bin/rustc --cfg stage0  -O  --target=x86_64-unknown-linux-gnu -o x86_64-unknown-linux-gnu/stage0/lib/rustc/x86_64-unknown-linux-gnu/lib/librustc.so /home/brian/dev/rust3/src/librustc/rustc.rc && touch x86_64-unknown-linux-gnu/stage0/lib/rustc/x86_64-unknown-linux-gnu/lib/librustc.so
error: linking with `cc` failed with code 1
note: cc arguments: -L/home/brian/dev/rust3/build/x86_64-unknown-linux-gnu/stage0/lib/rustc/x86_64-unknown-linux-gnu/lib -m64 -o x86_64-unknown-linux-gnu/stage0/lib/rustc/x86_64-unknown-linux-gnu/lib/librustc-c84825241471686d-0.6.so x86_64-unknown-linux-gnu/stage0/lib/rustc/x86_64-unknown-linux-gnu/lib/librustc.o -L/home/brian/dev/rust3/build/x86_64-unknown-linux-gnu/stage0/lib/rustc/x86_64-unknown-linux-gnu/lib -lcore-c3ca5d77d81b46c1-0.6 -L/home/brian/dev/rust3/build/x86_64-unknown-linux-gnu/stage0/lib/rustc/x86_64-unknown-linux-gnu/lib -lstd-4782a756585a81-0.6 -L/home/brian/dev/rust3/build/x86_64-unknown-linux-gnu/stage0/lib/rustc/x86_64-unknown-linux-gnu/lib -lsyntax-84efebcb12c867a2-0.6 -Lrustllvm -lrustllvm -shared -lrustrt -lrt -ldl -lm -lmorestack -Wl,-rpath,$ORIGIN/. -Wl,-rpath,/home/brian/dev/rust3/build/x86_64-unknown-linux-gnu/stage0/lib/rustc/x86_64-unknown-linux-gnu/lib -Wl,-rpath,/usr/local/lib/rustc/x86_64-unknown-linux-gnu/lib
note: /usr/bin/ld: cannot find -lrustllvm
collect2: error: ld returned 1 exit status

Which is fixed by this patch: brson@7625022

The modifications to platform.mk look like they don't go far enough to me, requiring special cases in rt.mk, etc. to use the _arm compilers. Instead I think it makes sense to parametrize more of the compiler variables over the target triples (like much of the other code in the makefiles). Essentially, the MAKE_CC macro should be more like your MAKE_CC2 macro, allowing each target to have its own compiler. I realize this is a big change.

[brson]

@yichoi Nice job, and thank you for working on this.

[brson]

hm. perhaps you were expecting me to configure like ../configure --host-triples=x86_64-unknown-linux-gnu --target-triples=i686-unknown-linux-gnu, as the configure script copies triples from CFG_HOST_TRIPLES to CFG_TARGET_TRIPLES. That's a fair way to do it.

[yichoi]

Under x86_64-unknown-linux-gnu, ../configure --target-triples=arm-unknown-android works.
Under x86_64-apple-darwin build, ../configure --target-triples=arm-unknown-android works too.
currently I've modified for target-specific library extension (*.so, *.dylib).

[yichoi]

current arm latest working branch along with latest libuv
https://github.com/yichoi/rust/commits/arm_latest

[brson]

@yichoi thanks. giving it a look now

[brson]

@yichoi I want to try to make some drastic changes to the makefiles so there are fewer ARM special cases. I'm thinking of paramaterizing all the toolchain-related variables over the target triple - basically removing the few variables that use HOST_$(target) and making most everything in platform.mk target-specific. The end result should be that the ARM cross should be more similar to the x86_64->i686, etc. cross. I'm going to do some hacking on it this weekend, maybe try to do the ARM build myself, using your branch, so I understand exactly how it is supposed to work.

My current branch: https://github.com/brson/rust/tree/cross4

[yichoi]

@brson if you need help, please let me help you. Moreover I want to make Servo makefile for ARM with similar convention of Rust makefile if decided.

[yichoi]

What I have done :

• native compile and cross compile handling
• library extension handling (so, dylib)
• android.def for rt
• morestack.a handling (it will be removed when we complete the implementation of morestack for ARM)

[brson]

@yichoi I made a bunch of changes to the makefiles and then reincorporated your Android changes to my cross5 branch. Sadly, I did not use any of your patches directly, since they no longer applied after all the refactoring, but I did copy and paste major pieces.

I did two things slightly differently than you: 1) I am building and linking morestack.S, but it is empty; 2) I added an --android-ndk-path flag to rustc.

With this branch I can build core and std for android when configuring with ../configure --target-triples=arm-unknown-android --android-ndk-path=~/dev/android-standalone-ndk-14. Can you give this a try and see if it works for you?

I'm sorry about all the back-and-forth on this issue, and hope it's not discouraging you.

[yichoi]

@brson oh, you did it ahead of me. I have been working to rewrite makefiles for various build host and arm target https://github.com/yichoi/rust/commits/cross5, which works successful and almost similar with what you did.
Mainly, I focused to rewrite to push all target specific flags to platform.mk.

[sanxiyn]

We have been using arm-unknown-android as a triple, but Android NDK uses arm-linux-androideabi. I think it is best to follow that.

[yichoi]

@brson Build for ARM with your patch worked on linuxpc, failed on mac however it's morestack problem. if you disable morestack.a at link.rs, it will work on mac, too. as you know if we complete morestack for ARM, it will be unnecessary.

[brson]

@yichoi I am truly sorry I keep stepping on your toes here. Our patches are indeed quite similar.

I like the way you handled platform.mk, just writing out the variable names like CFG_INSTALL_NAME_x86_64-unknown-linux-gnu. I'm so used to putting $(1)s everywhere that I didn't even consider that. The one thing I don't like so much is the way it names the compiler by prefixing the CFG_GCCISH_CROSS variable, vs. defining CC_$(triple), etc. in mine - it seems to be specific to android by expecting compilers to be named $prefix-gcc.

It also doesn't create a per-triple X variable for naming executable extensions, but honestly that's probably not a major problem at the moment since nobody is going to be cross compiling from windows to android. Some people do have a desire to cross-compile from linux to windows (wine) though. I think that is beside the point for the android cross though so not a major problem if we don't do that, and it may actually be preferable not to do that now to avoid the extra complexity.

I'm going to run your branch through the try bots today and see what they think. Perhaps you can let me know how you think we should proceed from here.

@sanxiyn Changing the triple sounds fine to me. Does LLVM accept either?

[sanxiyn]

Yes, LLVM accepts both.

[brson]

@yichoi fyi here are the results of pushing your branch to the try bots:

http://buildbot.rust-lang.org/builders/try-win/builds/354/steps/clean/logs/stdio
http://buildbot.rust-lang.org/builders/try-linux/builds/368/steps/test/logs/stdio
http://buildbot.rust-lang.org/builders/try-mac/builds/407/steps/test/logs/stdio

I'm not sure offhand why some would be segfaulting because of these changes.

[yichoi]

@brson Above logs resulted from libuv. when I changed to libuv @da33bba, test of linux and mac was successful. The result of windows came from the reason because I didn't put flags for mingw32.

Basically, I have added flags for mingw32 my cross5, however I cannot fully test since rustc.exe in snapshot failed on my mingw32 environment at early stage0.

[brson]

@yichoi Your libuv branch includes some upstream commits that are more recent than our current branch, and it changes some structure definitions. If we can avoid it I'd prefer not to take those changes at the same time, so we don't have to worry about uv regressions. Can your patches work on top of uv commit da33bba7c04e0873b457a9a4290bed2adf620154?

[yichoi]

@brson yes, as long as native target triples (x86 linux, x86 mac). I will test more for android, mingw32, then make some patch based on libuv @da33bba, if needed.

[brson]

@yichoi I've put another branch up at https://github.com/brson/rust/tree/cross6 that tries to apply your uv patches to the previously mentioned uv branch. It also includes some fixes to the makefiles that you'll want to grab. I'm going to try to get this passing the try bots tonight. Hopefully we can merge soon. Two things I haven't tested carefully yet though are make clean and make install.

[yichoi]

@brson thanks. I will check your branch and try

[brson]

I've updated my branch to fix some problems I've seen on the bots, first that toolchain variables aren't defined for a few triples, including FreeBSD, and second to fix check-fast.

[yichoi]

I've tested my makefile patches of cross5 on Rust/master branch

1. test results with no cross compilation are
   linux : passed make check
   macos : passed make check
   mingw32 : passed make check-fast, failed make check-lite,make check. (I am not sure whether this result comes from makefile or test environment)

It looks to me that test supposes to perform build-triple only. It may be meaningless to test target-triples which is not build-triple.

2. Installation and uninstallation works at linux/macos/mingw32 with multiple target triples.

[brson]

I believe that plain make check is broken on windows for other reasons (problems with the license check script), so that's fine.

[brson]

As mentioned on IRC, I've taken your most recent branch and added a few more commits that fix the build in various scenarios, and pushed to my cross7 branch. I'm going to focus on merging this branch now.

[yichoi]

I've pushed the patch to pass android-cross-path to rustc.

[brson]

I opened a pull request to merge this.

It doesn't include your latest patch to add an android ndk flag to rustc, so that can be done in a followup.

[brson]

I see that our branches have diverged a lot again, so I'm going to take yours and run it through the bots and try to get it merged.

[brson]

Ok, opened an updated pull request. I think this is the one!

[brson]

Looks like this has been merged. Thanks everyone. Please file any followups as new issues. FWIW, I think in the next few weeks we should be aiming to make it Android cross-compilation usable, at least for experimintation, in 0.6. From my perspective that means getting morestack in, writing some docs, adding whatever polish is missing, possibly working on getting the test suite to pass.