16.1 Delivering for Android

To use LispWorks in an Android project, the Android project needs the following:

1. The LispWorks Android archive file lispwors.aar. This defines the support classes in the Java package com.lispworks. This file is part of the LispWorks distribution, and can be found in the etc directory in the LispWorks distribution:

   (lispworks-file "etc/lispworks.aar")
   

   You need to add this file to your project. In Android Studio, you should follow the instructions in the Android Studio guide, section "Add your library as a dependency" in Create an Android library. In the first step, use the "Add the compiled AAR" branch, that is use New Module.

2. The two files generated by deliver-to-android-project (the Lisp heap and the dynamic library).

   The Lisp heap needs to be in the assets directory of the APK, so in Android Studio with typical settings it needs to be in the assets directory of one of the source sets.

   The dynamic library needs to be in the appropriate architecture sub-directory (armeabi-v7a for ARM 32-bit, arm64-v8a for ARM 64-bit, x86 for x86 32-bit, x86_64 for x86 64-bit) under the libs directory in the APK, so in typical Android Studio settings it needs to be in the correspondingly named sub-directory under the jniLibs directory of one of the source sets.

deliver-to-android-project is intended to simplify the process of delivering into an Android Studio project, and if you pass the project directory or a module directory, it puts the Lisp heap and dynamic library in the correct place for Android Studio to find them. If you pass the project directory, it creates these files:

Note: You can develop your application using only one architecture of LispWorks (32-bit or 64-bit, ARM or x86), but before uploading to Google Play, you probably want to support both ARM architectures. Simply including the files for both ARM architectures in single APK (by running deliver-to-android-project on each architecture) will work, but you may want to reduce the size of the APK. See 16.1.1 Configuration for Separate APKs for different architectures for ways to deal with that.

The lispworks.aar file is required to tell Android Studio (or another Java IDE) about classes in the com.lispworks Java package, so you need it while working on the Java code that interfaces with Lisp.

The heap and dynamic library are needed only when you actually build the project. At run time, they are accessed only by com.lispworks.Manager.init, which loads the library, retrieves the heap from the assets and then calls into the library to initialize LispWorks.

Once these three files are in place, the Android project can be built and installed like any Android project. To use LispWorks, the method com.lispworks.Manager.init must be called to initialize LispWorks. If library-name was passed to deliver-to-android-project, then com.lispworks.Manager.init must be called with a matching name, otherwise the default "LispWorks" is used. com.lispworks.Manager.init can be called at any point during the lifetime of the Android app.

com.lispworks.Manager.init is asynchronous, in other words by the time it returns Lisp is not ready yet. com.lispworks.Manager.init optionally takes a Runnable argument, which is called when LispWorks is ready. Alternatively the method com.lispworks.Manager.status can be used to determine when LispWorks is ready. See the entry for com.lispworks.Manager.init for more details.

com.lispworks.Manager.init loads LispWorks and initializes it. Apart from standard initialization and starting multiprocessing, the startup function also initializes the Java interface using init-java-interface, passing it the appropriate arguments. That includes passing the keyword :report-error-to-java-host, which makes the function report-error-to-java-host invoke the user Java error reporters, and the keyword :send-message-to-java-host which makes the function send-message-to-java-host call the Java method addMessage. See 41 Android Java classes and methods for the details.

The startup functions also set up a global "last chance" internal debugger hook, which is invoked once the debugger actually gets called (after any hooks you set up like error handlers, debugger wrappers and cl:*debugger-hook*). The hook reports the error to the Java host (that is, invokes the user error reporters) and calls cl:abort. If you did not define a cl:abort restart, that will cause the current process to die, unless it is inside a call from Java, where it will cause this call to return. The return value is a zero of the correct type (see in 15.3.1 Direct calls and 15.3.2 Using proxies).

Once initialization finished, if a function was passed to deliver-to-android-project as its function argument, it is invoked asynchronously, and then the Runnable which you passed to com.lispworks.Manager.init (if any) is invoked. From this point onwards, Lisp is ready to receive calls from Java, and can make calls into Java.

On Android when doing GUI operations it is essential to do them from the GUI thread, which is the main thread on Android. The functions android-funcall-in-main-thread and android-funcall-in-main-thread-list can be used to invoke a Lisp function on the main thread. To facilitate testing, these functions are also available on non-Android ports.

There is no proper debugger on Android itself, so it is important to ensure your code is working before delivering.

16.1.1 Configuration for Separate APKs for different architectures

The dynamic library and Lisp heap files that deliver-to-android-project generates are architecture specific, that is they are either 32-bit or 64-bit and either ARM or x86, depending on the image in which deliver-to-android-project was invoked. The architecture can be 32-bit or 64-bit ARM, which correspond to the armeabi-v7a or arm64-v8a Android ABIs respectively, or 32-bit or 64-bit x86, which correspond to x86 and x86_64 respectively.

In most of cases, you will want your application to be compatible with both ARM ABIs, because Google Play requires compatibility with arm64-v8a (from September 2019), but at the moment many devices are still armeabi-v7a (see Android Developers Blog(19 December 2017): Improving app security and performance on Google Play for years to come). Therefore you will need to deliver on both architectures.

Incorporating all architectures into the same APK works (creating a "universal APK"), and this is the simplest solution. For this, you just need to deliver all architectures to the project directory, and both will be incorporated into the APK and work as expected.

The problem with incorporating both ARM architectures is that the delivered Lisp heap files are large (depending on what your application does and how it is delivered, but typically 5 - 10 MB and can be larger), so the APK that the end user will download is large too. It is possible to reduce the size of the APK that the end user downloads by creating several APKs, one for each ABI and containing only the corresponding Lisp heap, so each APK will be much smaller than the universal APK. In this case, Google Play will check the device before downloading, and download only the appropriate APK that matches the ABI of the device.

Android Studio has a mechanism to create such separate APKs, which is the splits.abi block (see Build multiple APKs). However, we did not find a simple way to specify which Lisp heap file to include from the assets directory for the different ABIs. Thus another mechanism is needed, and you can choose one of the following:

1. Probably the best approach is the flavors mechanism that is used in the OthelloDemo example, and is discussed in 16.1.2 ABI splitting using flavors in the OthelloDemo. This has the advantage that it involves a simple change to the build.gradle file using well documented features of Android Studio, and it is easy to see which files go into which APK. Unless you have a reason not to use this mechanism, we recommend that you use it. This will also allow x86 builds to be incorporated as well.
2. You can build the APKs in separate projects for each architecture that have the same applicationId, with a sourceSets block in build.gradle to share all the sub-directories of a common source set. The projects will also have their own assets and jniLibs in their main source sets and you can then deliver LispWorks to the separate projects' main source sets. In this case you will not need the splits.abi block, but the project-path argument of deliver-to-android-project will need to be different between the different architectures. This option is useful if there are substantial differences between the application versions for each architecture.
3. If you are proficient with Gradle, you can write Gradle code that deals with the ABIs. Your code will need to check which ABI is built (armeabi-v7a or arm64-v8a), and ensure that only one of the Lisp heaps is packaged in the APK: the one ending with .armeabiv7a.lwheap for the armeabi-v7a ABI, and the one ending with .arm64v8a.lwheap for arm64-v8a. Once you have this Gradle code, you just need to add the splits.abi block with the two ABIs to create the two separate APKs. This code will also need to set the versionCode appropriately, because the APKs must have a different value for this to be considered as different by Google Play.
4. If you build APKs using scripts, you can add some commands at the beginning of the scripts to ensure that only the appropriate Lisp heap is in the assets directory, maybe copying it from some other directory. This will also allow you to just use the splits.abi block. Again, you will also need to do something about making a different versionCode for each APK.
5. You can manage the Lisp heap files and versionCode by hand. This the simplest but most laborious and most error-prone approach. You still need the splits.abi block.
6. As long as the vast majority of Android devices still support armeabi-v7a (32-bit), you can try to "cheat". Build only the armeabi-v7a version of your application with a versionCode greater than 1, and also create a dummy APK with the same applicationId as your application with versionCode 1 but without any libraries (this dummy needs to be created only once). Then upload your application's APK and the dummy APK. Since the dummy APK does not have libraries, it will be regarded as supporting all architectures, so will satisfy the Google Play requirement of supporting arm64-v8a. However, Google Play will use the armeabi-v7a APK for all devices that support that ABI because its versionCode is greater than 1, and the dummy APK will be used only in devices that do not support armeabi-v7a. We have not tested this.

16.1.2 ABI splitting using flavors in the OthelloDemo

See 16.4 The Othello demo for Android for details about running the OthelloDemo example.

The example uses flavors to create a separate APKs for each ABI (armeabi-v7a, arm64-v8a, x86 and x86_64), most importantly to avoid packaging unneeded heaps (for the other ABIs). This is implemented by the following lines in the build.gradle file of the app module (android/OthelloDemo/app/build.gradle):

flavorDimensions "abi"
productFlavors {
    arm64v8a {
        dimension "abi"
        versionCode 10004 
    }
    armeabiv7a  {
        dimension "abi"
        versionCode 4
    }
 
    // These are for the emulator (or x86/x86_64 device
    // if you can find one)
    x86_64         {
        dimension "abi"
        versionCode 30004
    }
 
    x86         {
        dimension "abi"
        versionCode 20004
    }

The lines above define a "dimension" called abi, and adds four flavors for it: arm64v8a, armeabiv7a, x86 and x86_64. When Android Studio builds with one of the flavors, it will also look for files in an additional flavor-specific "source set" directory app/src/arm64v8a, app/src/armeabiv7a, app/src/x86 or app/src/x86_64, which are initially empty. The flavors also define the different versionCode for each flavor, which is sufficient as long as there are no other dimensions that need multiple APKs for the same application.

Note: These flavor names arm64v8a, armeabiv7a, x86 and x86_64 are not prescribed by Android Studio, but they are what deliver-to-android-project looks for when deciding where to write the Lisp heap and dynamic library files. Using them allows the call to deliver-to-android-project in LispWorks to be simpler and to be the same on all architectures. You could use different flavor names, but then the arguments to deliver-to-android-project would need to be different between the architectures to ensure that it writes the files in the correct directory. The names of the "source set" directories would also need to match the names used for the flavors.

The demo calls deliver-to-android-project with its project-path argument being the root project path. By design, when it runs on ARM 64-bit it looks for app/src/arm64v8a/ in the project directory and puts the files in it if it exists. Similarly, it looks for app/src/armeabiv7a/ on ARM 32-bit, app/src/x86 on 32-bit x86 and app/src/x86_64 on 64-bit x86. Therefore, the APK for the arm64v8a flavor will contain only the 64-bit heap and library, and similarly for the APKs for the other flavors will contain only the corresponding heap and library.

The APKs are recognized by Android and Google Play as ABI-specific because of the location of the dynamic libraries. deliver-to-android-project puts the ARM 64-bit library in the jniLibs/arm64-v8a/ sub-directory under the arm64v8a "source set" directory, so Android Studio packages it in libs/arm64-v8a/ inside the arm64v8a APK, which marks this APK for Google Play as an APK for the arm64-v8a ABI that can be used only on 64-bit ARM devices. Similarly, deliver-to-android-project puts the ARM 32-bit library in jniLibs/armeabi-v7a/, it is packaged in libs/armeabi-v7a which marks the APK for the armeabi-v7a ABI that can be used on any ARM device that supports 32-bit. The thing same happens for the x86 and x86_64 ABIs.

If you have other features that need to be different between the ABIs, they can be added in these flavors too, either as files in the "source set" directories or as properties inside the flavor block in build.gradle.

With this flavors mechanism, you do not need the splits.abi block, which would just increase the number of build variants, some of which will be non-functional (for example when armeabiv7a is paired with arm64-v8a). If you have your own foreign libraries, you have two options:

• Put your libraries together with the Lisp generated library under app/src/arm64v8a/jniLibs/arm64-v8a/, app/src/armeabiv7a/jniLibs/armeabi-v7a/, app/src/x86/jniLibs/x86/ or app/src/x86_64/jniLibs/x86_64/ as appropriate.
• Put the libraries in the usual place under app/src/main/jniLibs/, add the splits.abi block to the build.gradle file, and just ignore the non-functional build variants. You can also filter out the non-functional build variants using the variantFilter block.

In addition, the flavors make it easy to to place the files generated by deliver-to-android-project in directories outside the directory tree of the project, without interfering with other features of the project. You can do this by using the sourceSets block to point Android Studio to some other directories. The example build.gradle file contains a commented out sourceSets block, which you can include if you want to use this mechanism. You will have to edit the actual setRoot paths to match the setup of the your machine.

Note: if you remove the flavors from the example or rename them, you have to also ensure that the directories named arm64v8a, armeabiv7a, x86 and x86_64 do not exist, because deliver-to-android-project uses the existence of these directories as a flag that it should use it. It does not check the build.gradle file.