Android APK Proposal

[pawandubey]

Hi, I am Pawan, a GSoC aspirant in the short term and an aspiring contributor in the long term

Initially I was interested in the task to export to RAR, ISO and BZIP etc formats through ShrinkWrap, as listed on the ideas page. However, with my ongoing communications with alrubinger and kpiwko , this has now evolved into primarily adding an Android APK export functionality in ShrinkWrap.

As of what intel I have managed to gather from scavenging online resources, is this:

1. APK is a zip-like archiving format, especially developed for Android based operating systems.

2. And APK package contains of a fixed directory structure containing the following:

• Manifest file(for meta info about the archive itself)
• res folder(for all non-logic resources(e.g strings, images etc))
• src (for the java source files)
• assets
• obj (to store compiled java source files as .class files)
• bin (to store .dex files created)

3. The workflow of creating an APK package goes something like this:

• The proper directory structure as described above is created
• Java source files are added to the src folder and the resources to the resources and assets folder.
• Manifest file is created with proper references to all the resources etc.
• The java resource files are compiled to R.java file by aapt tool provided in the android SDK.
• The java source files are compiled into a .class file and stored in the obj directory using the javac tool.
• The .dex files are the Android executables. We need to convert the .class files created in the previous step and the third party dependencies by compiling them to classes.dex file by the dx tool. This is stored in the bin directory.
• The aapt(or apkbuilder?) tool is used to package the contents of the directory into an .apk file(unsigned).
• The unsigned package then needs to be signed using some suitable tool. Like jarsigner?
• Then the signed package needs to be zipaligned as a final step.

As far as the project is concerned, we will need to build an api for each step of the workflow as I didnt find any reusable code as yet. Any further guidance as to how to proceed further would be mightily appreciated. Especially, as kpiwko mentioned, can smikloso provide some insight here please?

[kpiwko]

Hello Pawan,

I think is the right decomposition of the problem, nicely done! I just wanted to note there are 2 APIs though - one is user facing API, that is used to define the APK archive itself, second is an iternal API - we actually does not need to expose those that much - making it an implementation details, e.g. converting classes to dex should happen automagically and user is not expected to call this step on his own. We would still need to have an intermediate format - hence the need for second API.

For archive signing, jarsigner and keytool is indeed the tool. Here API should expose a way how to let user specify signing key, which sane default - debug key, more details here: http://developer.android.com/tools/publishing/app-signing.html

Karel

[tkriz]

Hey,

good work on your current research. I'd like to show you some more goodies you might wanna dig into. First of all, there are two libraries I've previously made, which are more or less only a proof of concept with many flaws. One is for preparing a shrinkwrap archive and the other is for building it into actual APK. If I was about to write them now, I'd definitely take another approach and that's using 3rd party libraries to make the most of the work for me. For example, there is an ApkBuilder class from Google, which can take care of the building. Also, be sure to take a look at the internals of maven-android-plugin, as they use the Google's build libraries a lot (you can find those libraries in Maven Central: http://search.maven.org/#search%7Cga%7C1%7Ccom.android.tools).

Still, there is one problem, which might need to be solved. When you want to create shrinkwrap jar archive, you can put in .class objects and there will be put into the archive. If you'd want to do the same with android classes, you'd probably hit the wall as I did and that wall is resource IDs. Each compiled resource (images, layouts, each string, etc.) has its own ID and these IDs are a sequence (read the aapt source code if you'd like to learn the details) and the sequence is dependent on the compiled resources.arsc file. This means when your new package has less of different resources, they'll have different IDs. And because the ID is a final static int, the reference to it in your code will get replaced by actual value in compile time, so there's basically no way to easily find and replace those when you're adding the class into the new apk package.

I have a draft for a workaround, although it'd need some bytecode manipulation. Let there be a method, in the shrinkwrap android archive for adding an image resource:

AndroidArchive addAsImageResource(int currentResId, Asset resource);

Then the process of adding and requesting resources would work like so. The user calls the addResource method, with the image we want to add into the archive and it's current resource ID.

Asset testImage = ...;
AndroidArchive a = ShrinkWrap.create(AndroidArchive.class).addAsImageResource(R.drawable.test_image, testImage);

1) the AndroidArchive has to save the resource ID and keep the required information about its binding to the added resource

2) once new resources.arsc is compiled by aapt, we should find the resource's new id and make a binding to the old id (like Map<Integer, Integer> where the key is old resource ID and the value is new resource ID).

3) before we make the dx compile the .dex file, we need to step in and modify the bytecode of all classes that extends android.content.Context so that they will make use of this binding, so when something wants to access any resource, we will first check our bindings, if we can't find the key, it means the user didn't add this resource into the archive and it should fail and when we do find the key, then we need to load a resource by the new id we've stored as a value.

If you'd need any more information on this, feel free to contact me.

[pawandubey]

Hi Tadeas,

Thanks for all this amazing stuff!

I was myself looking to use the apkbuilder from Google, so thanks for conforming my hunch in that regard.

As for other libraries required, as Karel said, we can broadly divide this process into two layers of abstraction. One would be the things the end user would be exposed to and would need to do(like specifying the source files and resources as well as the final package name and providing his key for signing the package) and the other would be the build process which would happen in the background(we can use apkbuilder and jarsigner + zipalign here). I think the major part of "original" work has to be done on first part, that is providing users with an API for the process. As for the second part, I already had some resources, and thanks to you pointing me to maven, they have greatly increased

As for the problem you pointed out, thank you for letting me know in advance. I didnt realise that there would be an issue with the resource IDs after the build process. As of now, your approach seems simple and easy to implement(I hope ). But I would definitely look to see if an alternate solution already exists somewhere around(the best part about open source ) 

[tkriz]

I myself have been doing a lot of research for the resources issue and didn't really find anything. Maybe you'll be more successful. Anyway, I tried a lot approaches to the resource id issue, but none of them worked. Unfortunately I didn't have time to try the last one I've written down in my previous post. The only part that might be problematic would imho be the bytecode manipulation.

[pawandubey]

Just an update.

I was looking into the apkbuilder tool and after installing the SDK, I found it was missing!

So a bit of googling brought around the fact that google removed the APK builder from the standard SDk, and it has to be downloaded again.

P.S: I think the apkbuilder can do the most of the work for us. I am still looking for an alternate solution to the problem pointed out by Tadeas, although it is a problem that should not hinder the initial development.

[pawandubey]

Hi again. Time to delve into the details a bit.

So lets see what I have to do,

1) Create proper directory structure

     This can be done with Java's native mkdirs(); method. Will need a one time writing of a function to create the required directories in a required location. User will need to input the location which will serve to be the location of the package.

2) Add/Transfer files into the required directories.

     This can be done in two ways I suppose, or three, if we can include 3rd party libraries. One will be using the native java NIO2 methods, which has been available since 1.7. But since the build has to be done on 1.5, I suppose thats not feasible(correct me if I am wrong here kpiwko alrubinger). Second way is to write our own functions for file input and output streams. But that will be like reinventing the wheel. As of now, I suppose Apache commons FileUtils library lists a lot of useful methods for this. Or maybe we have something already written for this in the codebase?

Next stages are related to compilation and I will ask about them later once these two steps have been clarified.

[kpiwko]

Hey,

1) I believe there is no need to call mkdirs directly. Archive can hold all the structure internally and once dumped to FS (e.g. something like ExplodedExporter should be able to convert this into required FS structure.

shrinkwrap/impl-base/src/main/java/org/jboss/shrinkwrap/impl/base/exporter/ExplodedExporterImpl.java at master · shrinkw…

Temporary archive - e.g. needed for next stages, such as compilation via dx, can be dumped into tmpdir by default but having a convenience method to change this location in API is always handy, so +1.

2) Yes, please keep support for JDK 1.5 (or say 1.6). Android does not support 1.7 for all the Android API level, hence we need to be able to run with pre JDK 1.7. I believe there is already plenty of IOUtils/FileUtils etc in SW, for instance here:

shrinkwrap/impl-base/src/main/java/org/jboss/shrinkwrap/impl/base/io/IOUtil.java at master · shrinkwrap/shrinkwrap · Git…

If you don't manage to find feasible function there, just use 3rd party library. That's the best way to go, at least shorterm ;-) In SW world, all the entries in archive are represented as assets so I believe there will not be an urgent need to work with FS thought.

[pawandubey]

Thanks for clearing up the jdk part Karel. However, I did not get this part completely.

"I believe there is no need to call mkdirs directly. Archive can hold all the structure internally and once dumped to FS (e.g. something like ExplodedExporter should be able to convert this into required FS structure.

shrinkwrap/impl-base/src/main/java/org/jboss/shrinkwrap/impl/base/exporter/ExplodedExporterImpl.java at master · shrinkw…"

We first need the files to be added to the APK in the required structure right? When you say that "Archive can hold all the structure internally", which archive are you referring to, as we wont be having any archive at the initial stage?

[pawandubey]

Okay, scratch that last post, I got that clear now.
From the next step onwards, most of the things can be done automatically using the apkbuilder library from Google. However, how exactly we will produce the Manifest file is still not clear. Would you mind guiding me on the Manifest file creation part kpiwko / alrubinger / aslak.aslak.conduct.no ?

[kpiwko]

Pawan, would it be possible just to load androidManifest.xml from a static file? How much of its content do you expect to be dynamically constructed based on archive content?

[pawandubey]

kpiwko From the Android Dev site,

Every application must have an AndroidManifest.xml file (with precisely that name) in its root directory. The manifest file presents essential information about your app to the Android system, information the system must have before it can run any of the app's code. Among other things, the manifest does the following:

• It names the Java package for the application. The package name serves as a unique identifier for the application.
• It describes the components of the application — the activities, services, broadcast receivers, and content providers that the application is composed of. It names the classes that implement each of the components and publishes their capabilities (for example, which Intent messages they can handle). These declarations let the Android system know what the components are and under what conditions they can be launched.
• It determines which processes will host application components.
• It declares which permissions the application must have in order to access protected parts of the API and interact with other applications.
• It also declares the permissions that others are required to have in order to interact with the application's components.
• It lists the Instrumentation classes that provide profiling and other information as the application is running. These declarations are present in the manifest only while the application is being developed and tested; they're removed before the application is published.
• It declares the minimum level of the Android API that the application requires.
• It lists the libraries that the application must be linked against.

The points in bold are to be generated dynamically. The classes that the package contains will vary from package to package and also the libraries(if any) that the classes will depend upon. This is what we have to provide an api for.