Discovering a New World

The first official Android release of HTTrack has just been published on the Android store.

This was really fun to port HTTrack on this platform, actually. I never played too much with smartphones before (I usually don’t play at all with them, by the way, because everything I need to work or to connect to online resources is a real computer, not a smart-toy), but even if the httrack engine is 100% pure-C, the portage was not too difficult.

• the SDK provided by Google is clean enough do to complicated things without too much hassle (such as cross-compiling ARM bytecode on x86 machines) and is available on most platforms (at least Windows and Linux)

• the Eclipse-derivated developer environment (ADT) is rather well integrated, with all the nice features you would expect (including helping you tracking errors in obscure XML files, or starting gently device emulators)

• you do not even need any smartphone to start ; you can emulate on almost any type of devices (smartphones, tablets…) using the provided emulator and OS images

• the Android development API (in Java) is pretty straightforward, even when you start to mix native interface code (JNI)

To be clear, my first “hello, world” sample was created within minutes, and I then decided to directly start the httrack port.

The Genesis

I’m developing HTTrack on Linux, but I also use Windows as my main development platform, for practical reasons, and for the Windows release, which is currently the main “target” of HTTrack (and also, steam has more games on Windows, even if the Linux port is also quite excellent, but this is another story).

So I started by building the httrack engine code for android, of course. The first difficult task was to be able to build the original source code with the default build system (ie. Autotools/libtool). I could have created an Android mk file for that, but on the other hand I preferred to let libtool handle the dirty things (such as checking if v*printf was working, etc.) and have an unified build.

Another issue was the Android system itself: many common libraries, such as iconv or openSSL, are not available on this platform ; so I had to disable several features for the first releases.

After some tweaking, I finally got a working Makefile using something similar to:

./configure -host=arm-none-linux \
	--prefix=${DEST_DIR} \
	--enable-shared --disable-static \
	--with-sysroot=${ANDROID_SDK}/platforms/android-5/arch-arm \
	--with-zlib=${ANDROID_SDK}/platforms/android-5/arch-arm/usr \
	--includedir=${ANDROID_SDK}/platforms/android-5/arch-arm/usr/include \
	CC="${ANDROID_SDK}/toolchains/arm-linux-androideabi-4.6/prebuilt/linux-x86_64/bin/arm-linux-androideabi-gcc --sysroot=${ANDROID_SDK}/platforms/android-5/arch-arm -I${ANDROID_SDK}/platforms/android-5/arch-arm/usr/include -L${ANDROID_SDK}/platforms/android-5/arch-arm/usr/lib" \
	CFLAGS="-DANDROID -D_ANDROID -DLIBICONV_PLUG -I${DEST_DIR}/include -L${DEST_DIR}/lib" \
 && sed -i -e 's/CPPFLAGS = .*/CPPFLAGS =/' Makefile \
 && sed -i -e 's/CPPFLAGS = .*/CPPFLAGS =/' src/Makefile \
 && find . -name Makefile -exec sed -i -e "s/\(.*LDFLAGS = \)-version-info .*/\1 -avoid-version/" {} \; \
 && make -j8 && make install DESTDIR=${DEST_DIR} \
 && make install

Yuk, yes, I know, this is a little verbose, but, believe me, this is not as horrible as it seems.

The good thing is that I got rather quickly a libhttrack.so library containing everything I wanted (yay), without having two build systems. I was not too anxious, though: httrack is successfully built in a variety of platforms on Debian, including ARM architectures.

The GUI

The second step was to create some kind of interface (GUI) with the Android libraries, because you can not have (easily) a command line interface to play with. The first version only had a very minimalistic screen to play with (mostly a “Start” button).

Later, I added more cosmetic features, such as Next/Previous-style navigation, option panel with tabs, and some nice idiot-proof features (detecting common obvious mistakes, such as empty project name, empty URL, etc.)

One of the recurring complaint about the Windows version is the relative ugliness of its GUI. I must admit that my UI-design-skills are far from being perfect, and the original Windows GUI version took an insanely large amount of time to be developed (especially for an ugly result - but hey, I’m not developing a photo gallery, I’m developing a system utility).

In comparison, the Android GUI was much easier to develop (some people will probably find it likewise ugly - okay - but at least I did not spend two months on it), and much nicer in term of extensibility (adding tabs, scrollviews, etc. is pretty straightforward).

The Dependencies

As I said earlier, neither iconv nor OpenSSL are officially provided on Android platforms.

• iconv is not provided at all (even in libc)

• OpenSSL is provided as binary-only (no include files in ther SDK)

iconv is the easier one to build ; Danilo Giulianelli wrote a very clear entry on how to build iconv on Android.

OpenSSL was more difficult, and I basically used the Guardian Project patched source at the beginning, only to discover that /system/lib/libcrypto.so and /system/lib/libssl.so are actually provided - you just need to get the include files, and hope the ABI is more or less stable. My experience with OpenSSL suggest that this should be true (I used to dynamically load OpenSSL in httrack on POSIX platforms - probing any available release - in a foolish attempt not to be tied to crazy exports regulations), and the rather small subset of OpenSSL functions I am using makes me confident that this… daring solution should not cause too much trouble.

The Native Interface

In parallel, I started to write JNI interface code to be able to start the engine from the Java code.

This part was a bit more tricky, for a variety of reasons:

• JNI code is not difficult, but very strict over global references handling, or the number of local objects references you may have on the stack etc.

Therefore, if you do:

jclass myClass = (*env)->FindClass(env, name);

then you have to remember than myClass is only a local object reference, and if you keep it in some global variable, you’re gonne have a very bad time (ie. random crashes).

Therefore, you basically need to create a global reference:

static jclass findClass(JNIEnv *env, const char *name) {
  jclass localClass = (*env)->FindClass(env, name);
  /* "Note however that the jclass is a class reference and must be protected
   * with a call to NewGlobalRef " -- DARN! */
  if (localClass != NULL) {
    return (*env)->NewGlobalRef(env, localClass);
  }
  return NULL;
}

quite obvious, yes, but forget this obvious point and you are doomed!

Another detail, by the way: when you throw an exception, remember that the string passed has to be valid during the lifetime of the underlying String object. The small details are often the most annoying things…

You have also to make sure you don’t have too many local reference objects on the current stack, because the default one is pretty small (512 references at most). Each time you create an object (including strings), each time you get a member from a structure, you create an additional local reference which needs to be deleted as soon as possible when running in inner JNI code loops.

For example, converting a String[] into a char*[] should look like:

  /* Create array */
  size_t i;
  for (i = 0; i < argc; i++) {
    /* Note: a local reference is created here */
    jstring str = (jstring)(*env)->GetObjectArrayElement(env, stringArray, i);
    const char * const utf_string = (*env)->GetStringUTFChars(env, str, 0);
    argv[i] = strdup(utf_string != NULL ? utf_string : "");
    (*env)->ReleaseStringUTFChars(env, str, utf_string);
    (*env)->DeleteLocalRef(env, str);
  }
  argv[i] = NULL;

Of course, you also have the solution to create a temporary local frame (something Java do when calling functions, for example) which will allow to wipe all created local references using PopLocalFrame:

  /* create a new local frame for local objects */
  if ((*t->env)->PushLocalFrame(t->env, capacity) == 0) {
    ...
    (void) (*t->env)->PopLocalFrame(t->env, NULL);
  } else {
    ... throw something
  }

• Crashes are difficult to track when playing with native code ; the ADT debugger provided is not always able to track where the crash did occur - you’d better have a clean code. I personally use an abundant amount of assert(), which are providing some bits of information before chocking:

/* Our own assert version. */
static void assert_failure(const char* exp, const char* file, int line) {
  /* FIXME TODO: pass the getExternalStorageDirectory() in init. */
  FILE *const dumpFile = fopen("/mnt/sdcard/Download/HTTrack/error.txt", "wb");
  if (dumpFile != NULL) {
    fprintf(dumpFile, "assertion '%s' failed at %s:%d\n", exp, file, line);
    fclose(dumpFile);
  }
  abort();
}
#undef assert
#define assert(EXP) (void)( (EXP) || (assert_failure(#EXP, __FILE__, __LINE__), 0) )

I could also display some fancy message to the user directly, by calling the appropriate android java code…

• You do not have a LD_LIBRARY_PATH ready with all your beloved libraries ; it means basically that you have to load all libraries and dependencies in reverse topological order, using System.loadLibrary() calls in static initializer. I also realized that some libraries, such as OpenSSL, are actually present on the system - wthout any headers being provided, though. Therefore, you have to dig a little to sort things out…

The typical HTTrack library initializer is then looking like:

  /** OpenSSL (for HTTPS); taken from the /system libraries **/
  System.loadLibrary("crypto");
  System.loadLibrary("ssl");

  /** Iconv (Unicode). **/
  System.loadLibrary("iconv");

  /** HTTrack core engine. **/
  System.loadLibrary("httrack");

  /**
   * HTTrack Java plugin (note: dlopen()'ed by HTTrack, has symbol
   * dependencies to libhttrack.so).
   **/
  System.loadLibrary("htsjava");

  /** HTTrack Android JNI layer. **/
  System.loadLibrary("htslibjni");

Gluing Every Pieces

Connecting all pieces together took a bit more time, too. On Android, you have to handle a variety of cases:

• Permissions: this is probably one of the beginner’s first headache source. If you need to write to the external SDCard, for example, you will need to declare some special permission in your Android.xml application file (in this case, android.permission.WRITE_EXTERNAL_STORAGE) so that the system can let you do what you want. You won’t be able to connect to the outside world without android.permission.INTERNET, either.

• Orientation change: when an user is changing the screen orientation (by rotating the device), your Activity is by default killed, and recreated. This was a bit surprising for me, but fortunately you can put all working jobs on an isolated “fragment” container, and let the system know that you may gently save and restore the state of your GUI (Activity) if needed:

  /* First attempt to reattach to an existing background job */
  final FragmentManager fm = getSupportFragmentManager();
  runner = (RunnerFragment) fm.findFragmentByTag(id);

  /* Create a new "fragment" (our background job runner) */
  if (runner == null) {
    final FragmentManager fm = getSupportFragmentManager();
    runner = new RunnerFragment();
    runner.setParent(this);
    fm.beginTransaction().add(runner, id).commit();
  }

You also need to overrivde onSaveInstanceState and onRestoreInstanceState in your Activity to be able to retrieve your current context.

• Your application can be killed at any time, for example if the user hits the “back” button to exit your main GUI. You have to take care of saving necessary settings, and stop running background jobs (especially native ones!) gently, and if necessary suggest to restart an interrupted job later.

• Limited amount of resources: especially when playing with JNI, you have to take extreme care of how many objects, how many local references etc. you are using (see above remarks). You also have to take care on where you are storing data - writing directly on the SDcard root is considered a bad practice ; you also may not have any SDcard at all on the device, etc.

At Last

It took me basically two weeks to have a final release. Oh, I’m not saying that the first release is perfect, and you may experience bugs and problems, but the first tests were quite successful.

And it was a rather pleasant time - considering my total lack of experience on this field, the portage was rather straightforward, even with a project involving not trivial tasks (JNI, background jobs, etc.)

If you want to check out the code, feel free to browse it.

TL;DR: developing applications on Android, even including native code, is rather straightforward!