ARTist for Developers
ARTist provides a lot of opportunities for developers that want to analyze and modify applications, be it for work, curiosity or just pure fun.
But first things first, it is important to distinguish whether you want to use ARTist by creating modules or if you want to extend ARTist and its tools. In case you want to write own modules and use them to analyze, customize or otherwise modify applications, you should start here. Otherwise, if you want to work with ARTist directly for, e.g., extending the core functionality or fixing bugs, you should start here.
If you want to dive a bit deeper and understand the underlying ideas and design rationale behind ARTist modules, read the modules page first.
In case you want to create an own ARTist module, the module SDK is the official way to go. The workflow will roughly work like this:
- Install the module SDK for the preferred platform and Android version
Currently, there are module SDK packages per Android version per hardware architecture, but this will improve in the future =)
- Fork or otherwise copy
- Now it’s time to implement your own logic. The existing
codelib is essentially just the boilerplate code required for ARTist and
a bunch of APIs used by the
template-module, but you can implement anything here. Be creative!
- It is build time! First, build the codelib
./gradlew buildor use Android Studio) and copy the resulting
APKfile into the root directory if your module. Make sure the name is
codelib.apk. Second, you are ready to build the final module now. If you successfully installed the module SDK in step 1, then the
Makefilethat already exists in your project will work out of the box. A simple
in the module’s root dir is already sufficient to trigger the build. The result will be placed in the
- Now that you have build your module, it is time for a test drive.
Copy it to your device with
adb push /path/to/module/out/module.zip /path/on/device
or use your favorite file manager. In
ArtistGui, navigate to the
Modulesscreen and hit the small plus in the top right corner. Instructions on how to setup and install ArtistGui are below. This is a file chooser that you can use to import the module you just pushed to the device. A single touch will open up the zip, so make sure to hold until the actual zip file is marked and then use the
- You are done! Your module is ready and you can now navigate to the
Instrumentationscreen. For each application you pick, there will now be a checkbox for your new module that allows you to apply it.
Experience has shown that at some point you find an app that works quite well with your module and is therefore great for debugging. Use the search feature on top to find it quickly.
Congrats, you are now a module developer. It will require some time
and patience to find your way around the ARTist classes and the
compiler’s intermediate representation, so it might be beneficial to
have a look at the instrumentation passes that are by default in the
template-module. They are explicitly written to document common usage
of convenience functions and boilerplate code to give you a head start.
This section explains the necessary steps to deploy ARTist within an application, so that you can install and use it on arbitrary rooted Android devices. The original compiler will stay unchanged and we will not interfere with Android’s existing installation toolchain. In general, this deployment option is the least invasive one available. If you want to deploy ARTist as the default compiler instead, jump to the next section.
The Underlying Rationale
So why would you have ARTist wrapped in an app instead of replacing the system compiler if we have root anyway? There are multiple reasons for this but eventually all boils down to the following:
Giving the user full control in the least invasive way possible
Essentially, there are two core arguments:
- Take action exactly how and when the user requests it
- Otherwise do not touch the system
The first one for example explains why we do not simply replace the system’s compiler because in this case ARTist would recompile each and every application that is newly installed or updated. Instead, we take a more selective approach and let the user decide which apps should be recompiled and how. The deployment as an application is giving us a lot more flexibility and the chance to increase usability by providing an easy-to-use graphical interface. Of course, we could have done this with ARTist as the system compiler as well by providing an app that simply configurates the deployed compiler, but this is way more invasive.
The second reason is due to the simple fact that we expect the app to run on end user devices and therefore avoid any unnecessary interruptions. While we give our best to keep ARTist as stable as possible, it is not only the framework but also the modules that determine how stable a target runs after the process of recompilation. We envision users to be able to load and execute arbitrary modules that might be created from anybody. Therefore, for the sake of stability and security, they should not be executed unnecessarily.
But enough about the reasons now, let’s get started with the setup.
Setting Up ArtistGui
The typical use case is that a developer wants to implement own functionality within the ARTist framework and ship this to her users using the ArtistGui app. The documentation on how to create own modules can be found here. However, we recomend you first complete this setup and test it with one of the default modules before creating your own one.
Here is a high-level overview of how we will proceed:
- Download the ArtistGui app
- Create a configuration for the build scripts
- Build & deploy ArtistGui
- Recompile your first application
We start with closing ArtistGui from the GitHub repository
# If you have a GitHub account setup, you can also use the ssh version # git clone --recursive firstname.lastname@example.org:Project-ARTist/ArtistGui.git git clone --recursive https://github.com/Project-ARTist/ArtistGui.git
You may have noted the
---recursive switch. It ensures you not
only download the ArtistGui app itself but also the attached submodule.
In this case, we also download the
which we will discuss later.
Now there are two possibilities how to proceed:
- If you do not want to make any changes to ARTist itself, then you can just download a binary release. The concrete steps are outlined here. If you are a module developer or just want to quickly try out ARTist, this is the way to go.
- If you want to test and use your own version of ARTist, then you have to set it up and compile it first, so please proceed to the ARTist instructions, then come back and proceed here.
If you have a look at the
page of ARTist, you will find bundles you can download. The name of the
files encode what setups they support. First, pick the hardware
architecture that fits your setup, for example
arm. Second, decide which
Android versions you want to support. Take
for example, it encodes its support for Android 7 (
a7) and Android 7.1
arm devices. What you need to do is to first download the
zip file and then extract its contents to the following
There are folders for each supported Android version in the top-level
directory of the zip file. If the folders already exist in
they are probably empty.
Future versions of ArtistGui will be able to download correct ARTist versions at runtime.
Alright, since you have the compiler available now, it is time to install ArtistGui and get started using ARTist on your device.
Here you will learn how to setup the development scripts in ArtistGui to build ARTist (maybe remotely) and copy the result back to the correct location in ArtistGui.
Move into the cloned ArtistGui folder and have a look at the scripts directory.
As you can see, there are build scripts for each of the supported Android
versions. In a nutshell, they connect to the build server, build the
art module and copy the resulting artifacts back into the folder where
ArtistGui expects them. The scripts are created for a setup where
the AOSP is build on a remote server that is accessible via
with an active
sshfs mount on your local machine. If this is NOT
your intended workflow, we invite you to create alternative build
scripts and share them via pull requests =)
For this tutorial, we will assume you use the default setup with
sshfs. There is no need to adapt the scripts in this case,
can put your own paths in a configuration file.
There is a template called
ssh-2x.config.example. You need to
create an own version for each Android API level you want to support.
In our example, we will use Nougat 7.1 (API 25)
cp ssh-2x.config.example ssh-25.config
Now edit your new configuration and fill in your paths and desired configurations. The comments contain explanations for each single entry. Your result might look similar to this
#!/usr/bin/env bash ndk_path="/home/<username>/android-environment/sdk/ndk-bundle/" # ssh alias for your build server # => You could also use `localhost` if you don't use a build server. # `server_aosp` and `mounted_aosp` would be equal in this case server_alias="buildserver02" # path to aosp root on the build server server_aosp="/home/<remote_username>/android/aosp/android-7.1.1" # local mount point for aosp mounted_aosp="/home/<user>/mount/buildserver02/android/aosp/android-7.1.1/" # the api level for which aosp is checked out and will be compiled # artist currently supports api levels 23 (marshmallow), 24 (nougat) and 25 (nougat 7.1) api_level=25 # how many threads are available for compilation on the build server threads=128 # Select the target architectures: 32bit (false) / 64bit (true) # => needs to get selected from Android 8.0 Oreo onwards arch_64=false # Strip debug symbols / shrink binaries debug_binaries=false
Now move up to the ArtistGui root folder again
and start your script
Now you are ready to install ArtistGui on a device and use the ARTist code you just compiled.
Now that everything is set up and we have an ARTist version ready, we
can finally build and install
Have a look
here for information on how to get started with using ArtistGui
to interact with ARTist on the device. That’s it. You can now use
ArtistGui to deploy and test your modules.
This part of the documentation focuses on directly working with and modifying ARTist. If you just want to build a module and/or work on other parts of the ecosystem, chances are high you do not need it. If, however, working with AOSP does not sound like a horrible idea to you, go ahead and learn how to modify ARTist.
The ARTist build toolchain is a bit involved, so it takes some setup to get started. However, once it is up and running, developing wit ARTist becomes much more fluent.
This setup is requires for all cases where you want to work on anything related to ARTist’s native compiler code. If you, however, want to work on some other tools from the ARTist ecosystem, such as dexterous or monkey-troop, you can probably skip this part.
Setting up AOSP
ARTist builds on the dex2oat compiler of the Android Runtime, therefore it is built as a part of the art project, which again is a module of the Android Open Source Project (AOSP).
A few words about working with AOSP. The project is pretty huge and building it requires time and resources. We therefore recommend to build it on a compute or build server and work on art via a mounted share (
sshfs). If you plan to use our ArtistGui app for deployment, we provide convenience scripts to work with a remote AOSP version.
Setting up the Build Machine
Google provides a documentation on how to
setup your linux machine, including the installation of a proper
Java version, usage of the
repo tool to checkout the source code
and a build using
make. Our first step is therefore setting up the
machine for working with AOSP according to Google’s documentation.
Follow the steps in the linked Google documentation until you reach the
repo is used for the initialization.
Downloading the Source
repo to initialize the source tree and sync the changes,
you need to pick a version tag corresponding to the Android release
version you want to download.
At the time of this writing, ARTist is stable for three Android versions and in experimental stage for a fourth one. In the following, we list the tags from which we branched off and the corresponding art branch:
- Marshmallow: android-6.0.1_r62 (Build: MTC20F) => artist_marshmallow_master
- Nougat: android-7.0.0_r12 (Build: NBD90W) => artist_nougat_master
- Nougat 7.1: android-7.1.1_r6 (Build: NMF26Q) => artist_nougat_7.1_master
- Oreo (experimental): android-8.0.0_r9 (Build: OPR3.170623.007) => artist_oreo_master
At the time of this writing, only 7 and 7.1 are supported, but we are working on getting 6 and 8 on board again.
Choose the tag that corresponds to the Android version you want to build and invoke the corresponding command below in the AOSP root directory:
# repo init -u https://android.googlesource.com/platform/manifest -b android-6.0.1_r62 # repo init -u https://android.googlesource.com/platform/manifest -b android-7.0.0_r12 # repo init -u https://android.googlesource.com/platform/manifest -b android-7.1.1_r6 # repo init -u https://android.googlesource.com/platform/manifest -b android-8.0.0_r9 repo init -u https://android.googlesource.com/platform/manifest -b <RELEASE_TAG>
In the next step, we will use a local manifest version we prepared that includes the ARTist GitHub repository as a remote origin for the art project. Depending on the version you picked earlier and whether you have a GitHub account, choose one of the following commands:
# curl -o .repo/local_manifests/artist.xml https://artist.cispa.saarland/res/marshmallow/local_manifests/artist.https.xml # curl -o .repo/local_manifests/artist.xml https://artist.cispa.saarland/res/nougat_7.0/local_manifests/artist.https.xml # curl -o .repo/local_manifests/artist.xml https://artist.cispa.saarland/res/nougat_7.1/local_manifests/artist.https.xml # curl -o .repo/local_manifests/artist.xml https://artist.cispa.saarland/res/oreo/local_manifests/artist.https.xml # if you have a GitHub account and a corresponding ssh key available, you can also use artist.ssh.xml instead of the https version curl -o .repo/local_manifests/artist.xml https://artist.cispa.saarland/res/<VERSION>/local_manifests/artist.https.xml
downloads the whole source. This will take a while, even with a strong internet connection.
After the sync succeeded, it is now time to build AOSP. Note that our custom manifest points art to the GitHub repo but the checked out state still corresponds to the unchanged AOSP version, so that a build now does not yet include any ARTist code. The AOSP modules have complicated dependencies and therefore we want to have a full build available before starting with ARTist. Consecutive builds will only be incremential.
First, source a script that sets up the environment so that other convenience commands are available.
Then invoke the
lunch utility to choose the hardware platform you
want to build for.
Finally, AOSP is ready to be built! Do not forget to provide a reasonable number of threads for speeding up the build. If you have no ideas how to pick the number of threads, picking the number of available cores can be used as a simple heuristics.
# e.g. make -j8 make -j<THREADS>
Again, this will take a while. A full clean build on a build server with 128 virtual cores and 3/4 terabyte RAM takes ~ 20 minutes, so if you are building on your workstation, better run it overnight.
Switching to ARTist
Now that all dependencies are satisfied, it is time to checkout ARTist. The core structure is two-fold: The first part is implemented in the art project and the second part resides in its own ARTist repository. We try to keep the changes in art as low as possible and keep ARTist version independent, but as of now, we still need both projects.
Navigate to the art AOSP module
and checkout the ARTist branch corresponding to the Android version you synced
# git checkout artist_marshmallow_master # git checkout artist_nougat_master # git checkout artist_nougat_7.1_master # git checkout artist_oreo_master git checkout <BRANCH>
The ARTist code, which resides in a subdirectory of art
art/compiler/optimizing/artist), does not require to distinguish
between Android versions.
We finally reached the point where we can build ARTist. After switching back to the AOSP root
it is time to rebuild the art module
mmma art -j<THREADS> # First time build needs to build the dependencies, too
mmma will build art including its dependencies. The build system
might decide that it has to rebuild some dependencies because of the
changes in the art repository, but this is only required once unless
you initialize another Android version. For subsequent builds, you can
substantially improve build time by using
mmm instead of
mmm art -j<THREADS> # Subsequent build without dependencies
This is the point where the one-time setup is complete. From now on,
you should only be required to rebuild art using
mmm after you
made changes, so the hard part is over. And as already mentioned earlier,
we even have scripts to automate this via ssh.
The build artifacts that are interesting to us are the
binary that is the compiler binary and in particular the
libart-compiler.so library, because they incorporate the ARTist code.
This concludes the common setup for ARTist. Depending on whether you want to deploy ARTist with an application on a rooted device or create your own AOSP build with ARTist as the system compiler, you can jump to the corresponding section below.
This section focuses on the use case where ARTist’s version of the dex2oat compiler replaces the original one in the system directory. Keep in mind that this deployment option requires root if you just want to replace the compiler in an existing build, or a custom Android ROM that you compiled yourself if you want to have a more involved setup.
For this section, we assume you successfully completed the setup for building ARTist outlined above.
Choosing to replace the default compiler with our ARTist version allows to not only recompile installed applications, but also components of the Android operating system itself. To be more precise, we are talking about Android’s Java middleware (e.g., the system server that runs all system services), system applications and even the boot.oat file (cf. preloaded classes in pre-ART Android). While this might be achievable by replacing the system compiler in a rooted stock ROM, we will detail the use case where we are creating a custom ROM, i.e., we are compiling our own version of Android.
With the completion of the common setup above, you are already set for developing ARTist modules. However, in contrast to the app-layer deployment, we do not wrap it in an application but use it as the system compiler and hence, we need to start the full compiled AOSP for ARTist to take action.
On the machine where you want to execute the emulator, switch to the
AOSP root directory. If you use an
sshfs setup, move to the corresponding
mount point. Make sure to source the environment setup script and use the
source build/envsetup.sh lunch
because they make the emulator binaries immediately available to your shell. Depending on the concrete architecture, you can now issue a command to start an emulator equipped with your AOSP build
# emulator64-mips # emulator-mips # emulator64-arm # emulator-arm # emulator64-x86 emulator-x86
As the emulator for the AOSP build is configured for a fresh start by default, you can now witness the initial boot sequence.
The log exposes what is compiled by dex2oat and hence exposed to ARTist: In the very beginning, the system server itself is compiled, which looks like this on the x86 emulator:
/system/bin/dex2oat --zip-fd=6 --zip-location=services.jar --oat-fd=7 --oat-location=/data/dalvik-cache/arm/system@email@example.com@classes.dex --instruction-set=arm --instruction-set-variant=generic --instruction-set-features=default --runtime-arg -Xms64m --runtime-arg -Xmx512m --compiler-filter=speed --swap-fd=9
You want to increase the timeout interval of
Watchdog? Sure. Add
logging to the
ActivityManagerService whenever an activity of
your choice is started? Go ahead.
For one of our projects, we actually injected a call to one of our
own libraries into each basic block of each single method (!) in the
system server, which covers something like 23k methods, and the
resulting system was still running stable, just a bit slower. We also
learned that ARTist is a great tool to quickly hook into some APIs during
research or when debugging. You will learn why hooks are particularly
easy to put in place using ARTist
here, but for the moment, you have to trust us.
Beside the system services, also a lot of system applications are compiled at first boot and hence are within ARTist’s reach. The interesting fact here is that this system-centric deployment does not only imply that all apps that are newly installed or updated are forced to go through ARTist, it also means you have complete access to other system parts that are not available to you in the app-layer deployment with ArtistGui.
Another noteworthy part that is compiled by dex2oat is the
file mentioned earlier. It is the compiled version of what was referred to as
the preloaded classes before the introduction of ARTist. The Zygote
process would load them at system boot and since all app processes are
forked from Zygote, they all share those libraries with a copy-on-write
memory mapping. This architecture saved memory and computation time.
In ART, those classes are compiled into the
And guess what? This is done by dex2oat as well. However, in contrast to
the system services, it is not compiled during first boot but at AOSP
compile time. OS upgrades are an exception, of course, because then
the system potentially recompiles everything including
system services and all apps. The implication is that we can also
boot.oat with ARTist, but this is done during AOSP
compilation and the default toolchain settings suppress the output.
You can, however, force a rebuild by changing the
file and then recompile it with
# e.g., mmm -j8 framework mmm -j<THREADS> framework
showcommands proves that, indeed, dex2oat is used.
mmm showcommands framework ... out/host/linux-x86/bin/dex2oatd --runtime-arg -Xms64m --runtime-arg -Xmx64m --image-classes=frameworks/base/preloaded-classes --dex-file=out/target/common/obj/JAVA_LIBRARIES/core-oj_intermediates/javalib.jar --dex-file=out/target/common/obj/JAVA_LIBRARIES/core-libart_intermediates/javalib.jar --dex-file=out/target/common/obj/JAVA_LIBRARIES/conscrypt_intermediates/javalib.jar --dex-file=out/target/common/obj/JAVA_LIBRARIES/okhttp_intermediates/javalib.jar --dex-file=out/target/common/obj/JAVA_LIBRARIES/core-junit_intermediates/javalib.jar --dex-file=out/target/common/obj/JAVA_LIBRARIES/bouncycastle_intermediates/javalib.jar --dex-file=out/target/common/obj/JAVA_LIBRARIES/ext_intermediates/javalib.jar --dex-file=out/target/common/obj/JAVA_LIBRARIES/framework_intermediates/javalib.jar --dex-file=out/target/common/obj/JAVA_LIBRARIES/telephony-common_intermediates/javalib.jar --dex-file=out/target/common/obj/JAVA_LIBRARIES/voip-common_intermediates/javalib.jar --dex-file=out/target/common/obj/JAVA_LIBRARIES/ims-common_intermediates/javalib.jar --dex-file=out/target/common/obj/JAVA_LIBRARIES/apache-xml_intermediates/javalib.jar --dex-file=out/target/common/obj/JAVA_LIBRARIES/org.apache.http.legacy.boot_intermediates/javalib.jar --dex-location=/system/framework/core-oj.jar --dex-location=/system/framework/core-libart.jar --dex-location=/system/framework/conscrypt.jar --dex-location=/system/framework/okhttp.jar --dex-location=/system/framework/core-junit.jar --dex-location=/system/framework/bouncycastle.jar --dex-location=/system/framework/ext.jar --dex-location=/system/framework/framework.jar --dex-location=/system/framework/telephony-common.jar --dex-location=/system/framework/voip-common.jar --dex-location=/system/framework/ims-common.jar --dex-location=/system/framework/apache-xml.jar --dex-location=/system/framework/org.apache.http.legacy.boot.jar --oat-symbols=out/target/product/generic/symbols/system/framework/arm/boot.oat --oat-file=out/target/product/generic/dex_bootjars/system/framework/arm/boot.oat --oat-location=/system/framework/arm/boot.oat --image=out/target/product/generic/dex_bootjars/system/framework/arm/boot.art --base=0x70000000 --instruction-set=arm --instruction-set-variant=generic --instruction-set-features=default --android-root=out/target/product/generic/system --include-patch-information --runtime-arg -Xnorelocate --no-generate-debug-info --multi-image --no-inline-from=core-oj.jar --generate-mini-debug-info --compile-pic --compiled-classes=frameworks/base/compiled-classes-phone ...
You can try it out by using one of the existing modules, such as
Logtimization, which just logs some information to prove
ARTist was actually executed.
Now it is on you: Make your custom ROM truly yours. You are ready to
proceed to the modules
page and learn how to add your own Module to the mix.