At my company, we have been using string views for quite some time in our codebase - long before std::string_view
was cool (that is, before C++17) - and we recently realized that some of our initial choices might not have been completely wise at that time.
Here’s what we used to write nearly ten years ago (lots of boilerplate removed):
class MyFancyView
{
public:
MyFancyView(const char* ptr, size_t ptrlen) ...
... lots of fancy functions out there
public:
const char* data; // public data pointer
size_t len; // public length
};
First obvious issue with this primitive view are the public members data
and len
, something which is now frowned upon (for good reasons, as this tend to allow hacky changes in address and pointer). The other issue is that, if we want to be closer to the standard library API, accessors needs to be data()
and size()
.
The solution is rather straightforward: we need to rewrite thousands of lines of code using those direct members, using accessors, and sometimes change the code logic to use internal view logic - typically trimming views, taking a sub-view etc. Simple but overwhelming: the operation needs to be carefully done, line by line.
Besides,
clang-rename
did not appear to help either, as they seem more convenient to replace one file at a time, with an offset provided by an editor typically.data
into data()
and len
into size()
in the codebase would lead to havoc: we have a lots of stuff out there with those names and this will never work.And there’s a last catch: programmers are lazy. There is no world where we’ll do that manually!
Oh, and there’s a perfectly valid excuse not to do it anyway: the risk of introducing subtle regressions is great, and the risk of tuning any code reviewer crazy is even greater.
So let’s start from the beginning: we’d like to rename those two members.
The strategy needs to be a bit subtle:
The first item has a pretty straightforward hack: let’s rename data
into dAtA
, and len
into lEn
in our definition file!
public:
- const char* data{ nullptr };
- size_t len{ 0 };
+ const char* dAtA{ nullptr };
+ size_t lEn{ 0 };
};
Why the weird l33t case ? I’m glad you ask! Here’s the reason:
git grep -E "(dAtA|lEn)"
The command above did not yield any matches in our codebase. We have our perfect namings!
One more thing: you might have not realize it, but those two new namings do also have the same length as previously, so that replacing them won’t change a line size. You’ll see later why this is important.
What’s happening if we build the code without any changes ? Well, we’ll obviously have a couple of those errors before the compiler gives up on our own foolishness:
../types/MixedFancy.h:84:16: error: no member named 'len' in 'MyFancyView'
return lhs.len == rhs.size() && (lhs.len == 0 || __builtin_memcmp(lhs.data, rhs.data(), lhs.len) == 0);
This message is actually cool. It provides four major information:
And remember those information are extracted from the compiler itself: we can rely on them.
What more could you even ask for ? A margarita ?
If we grep for error: no member named
, then isolate the leftmost stuff (file:line:column), we almost are set to fix the error.
With this, if we extract the file, the line, and the column, we can now do our replacement:
LANG=C sed \
-E "84s/^(.{15})len/\1lEn/" \
-i "../types/MixedFancy.h"
The leading 84
before sed s
option is the line number, and we need to capture 15 (16 - 1) characters from the beginning of the line before column 16. Then, replace the captured part, the new name, and we’re all set. Using LANG=C is required as the compiler we use (clang
) will provide offsets at byte level.
A last detail: we’re lazy (we’re merely humans after all), but the compiler shouldn’t be.
We need to tell the compiler not to worry if we have thousands of errors, by adding -ferror-limit=0
in the cmake’s cxx_flags
.
We also need to tell ninja not to be too lazy (by using -k0
)
With all this, we can have the final renaming script main logic:
ninja -k0 -C . 2>&1 |
# capture compiler errors
grep 'error: no member named' |
# extract file, line, and column
cut -f1-3 -d':' |
# sort all
sort -u |
# only keep first file/line when several patterns exist on a single line
sort -u -k1,2 -t':' |
# now bulk replace
while read line; do
# extract filename, line, column
f="$(echo "$line"|cut -f1 -d':')"
l="$(echo "$line"|cut -f2 -d':')"
c="$(echo "$line"|cut -f3 -d':')"
# replace at line l and column c either of the pattern
LANG=C sed -E -i "$f" \
-e "${l}s/^(.{$((c-1))})data/\1dAtA/" \
-e "${l}s/^(.{$((c-1))})len/\1lEn/"
done
It is highly suggested you split each step above until you’re confident enough, to be able to hunt for any mistakes.
A note on the fact we have the same name length for replaced versions:
std::find(_elements.data, _elements.data + _elements.size, fun)
This code above will trigger two warnings, on the same line, but on two different columns. If we were to replace with a name of a different length, this would completely mess up our script.
That’s nice, but we also have a lot of stuff inside macros. I know, macros are bad, but legacy reasons…
Errors inside those macros will typically generate stacked error messages; such as in this case where we have two macros calling each other involved (in this case, we were renaming MyFancyView
itself):
test/MyFancyTests.cpp:72:9: error: unknown type name 'MyFancyView'
TEST_ENTITY("〹", 0x3039, false);
^
test/MyFancyTests.cpp:64:48: note: expanded from macro 'TEST_ENTITY'
#define TEST_ENTITY(name, value, onlyAlphaNum) _TEST_ENTITY(entities, name, value, onlyAlphaNum)
^
test/MyFancyTests.cpp:48:9: note: expanded from macro '_TEST_ENTITY'
...
The error location is actually on MyFancyTests.cpp:48:9
, but its message is triggered at MyFancyTests.cpp:72:9
.
To capture those cases, a multiline grep capture (as demonstrated on StackOverflow) with grep -Pzo
will do the magic:
errMatch="(unknown type name|use of undeclared identifier|no type named|no member named)"
fromMacro="note: expanded from macro"
grep -aPzo \
"(?s)[^\n]*: error: ${errMatch} '[A-Za-z0-9\._]*'[^\n]*\n([^\n]*\n[^\n]*\n)?[^\n]*: ${fromMacro}[^\n]*(\n([^\n]*\n[^\n]*\n)?[^\n]*: ${fromMacro}[^\n]*)*"
Then, with this capture, we need to cleanup null characters (emitted by the -z flag), keep only what we need (ie. filter only the desired symbols), and capture the four information we need.
The final script I used is the following. It was aimed for clang
, but could be adapted for other compilers. Feel free to use it as inspiration, and happy renaming!
-ferror-limit=0
#!/bin/bash
set -e
# Source and build location
readonly sources=$HOME/git/codebase
readonly build=${sources}/build
readonly headerFiles=(MyFancyView.hpp)
# List of replacements
# TODO: put your own replacements here
declare -A replacements=(
["MyFancyView"]="StandardView"
["MyFancuString"]="U16StandardView"
["MyFancuUnicodeString"]="U32StandardView"
)
# Function to return aLtErNaTe case version of $1
altCase() {
local result=
local i
local up=
local c
for i in $(printf "%s" "$1" | sed -e 's/\(.\)/\1\n/g'); do
if [[ -n "$up" ]]; then
c=$(echo "$i" | tr '[:lower:]' '[:upper:]')
up=
else
c=$(echo "$i" | tr '[:upper:]' '[:lower:]')
up=yes
fi
result="${result}${c}"
done
echo "$result"
}
# Print a message and fail
die() {
echo "** FATAL: $*" >&2
exit 1
}
# Emit info stuff
info() {
echo "[ $* ] ..." >&2
}
# Rebuild logic; places errors in /tmp/errors
# TODO: put your own (re)build logic here. It should build as much as possible, ignoring errors!
rebuild() {
# clean repository
git clean -dxf
# rebuild (-k0 to continue despite of errors)
rm -rf "build" \
&& mkdir -p "build" \
&& (cd "build" && cmake -GNinja .) \
&& ninja -C "build" -k0 \
| tee /tmp/errors
}
# Code format logic, because we are worth it
format() {
# TODO: insert your code format logic here
info "Cleanup clang-format"
git diff -U0 --no-prefix --no-color | clang-format-diff-14 -style file -i
}
# TODO: put your own logic here to add -ferror-limit=0 on the build
info "Set error limit to infinity"
sed -e 's/CFLAGS=/CFLAGS=-ferror-limit=0/' -i CMakeLists.txt
git commit CMakeLists.txt -m "chore: Add -ferror-limit=0"
# Compute intermediate class names with mIxEd cAsE for line column stability
declare -A intermediate
srcAllowed=
for from in "${!replacements[@]}"; do
to="${replacements[$from]}"
intermediate[$from]=$(altCase "$from")
if [[ -n "$srcAllowed" ]]; then
srcAllowed="${srcAllowed}|"
fi
srcAllowed="${srcAllowed}${from}"
done
info "Check we don't have hits on intermediate patterns"
for from in "${!intermediate[@]}"; do
to="${intermediate[$from]}"
! git grep -q "$to" || die "Oops: $to did hit; please change the logic"
done
info "Do all intermediate replacements in types"
for from in "${!intermediate[@]}"; do
to="${intermediate[$from]}"
echo "$from → $to"
# Let's replace directly concerned files first except includes
# TODO: put your own logic here to specify the file(s) holding the types to be renamed to patch
for i in ${headerFiles[@]}; do
sed -e "s/\b${from}\b/${to}/g" -i "$i"
sed -e "s%${to}\.h\"%${from}\.h\"%g" -i "$i"
done
done
info "Do all intermediate forward declaration replacements"
for from in "${!intermediate[@]}"; do
to="${intermediate[$from]}"
echo "$from → $to"
for i in $(git grep -E "^class ${from};" | cut -f1 -d':' | sort -u); do
sed -e "s/^class ${from};$/class ${to};/g" -i "$i"
done
done
# Let's loop until we converge without errors
step=1
while true; do
info "Rebuild #$step"
rebuild
# Let the magic begin: fetch error information and replace exact pattern at the exact places
errMatch="(unknown type name|use of undeclared identifier|no type named|no member named)"
fromMacro="note: expanded from macro"
# First handle inlined macros. Trickier, as this is a multiline capture. (LANG=C because way faster)
#
# /code/library/utils/MyFancyTools.cpp:60:45: error: use of undeclared identifier 'UnicodeChar'
# constexpr UnicodeChar replacementChar = UNICODE_REPLACEMENT_CHAR;
# ^
# /code/library/unicode/unicode.h:24:36: note: expanded from macro 'UNICODE_REPLACEMENT_CHAR'
#
# -or- with nested macros:
#
# /code/test/MyFancyTests.cpp:72:9: error: unknown type name 'AString'
# TEST_ENTITY("〹", 0x3039, false);
# ^
# /code/test/MyFancyTests.cpp:64:48: note: expanded from macro 'TEST_ENTITY'
# #define TEST_ENTITY(name, value, onlyAlphaNum) _TEST_ENTITY(entities, name, value, onlyAlphaNum)
# ^
# /code/test/MyFancyTests.cpp:48:9: note: expanded from macro '_TEST_ENTITY'
LANG=C \
grep -aPzo \
"(?s)[^\n]*: error: ${errMatch} '[A-Za-z0-9\._]*'[^\n]*\n([^\n]*\n[^\n]*\n)?[^\n]*: ${fromMacro}[^\n]*(\n([^\n]*\n[^\n]*\n)?[^\n]*: ${fromMacro}[^\n]*)*" \
/tmp/errors |
tr '\0' '\n' |
grep -aE "(error: ${errMatch}|${fromMacro})" |
sed \
-e "s/^.*error: [^']* '\([^']*\)'[^\n]*$/%\1/" \
-e "s/^\([^:]*\):\([^:]*\):\([^:]*\): ${fromMacro}.*/\1\t\2\t\3/" |
tr '\n%' '\t\n' |
sed -E 's/^([^\t]*)\t([^\t]*\t[^\t]*\t[^\t]*\t)*([^\t]*)\t([^\t]*)\t([^\t]*)\t$/\3;\4;\5;\1/g' |
grep -aE ";${srcAllowed};" |
tr ';' '\t' |
grep -avE '^$' |
sort -u |
while read -r elt; do
file=$(echo "$elt" | cut -f1)
line=$(echo "$elt" | cut -f2)
col=$(echo "$elt" | cut -f3)
from=$(echo "$elt" | cut -f4)
to="${intermediate[$from]}" || die "Can't find $from in intermediate"
printf "\r\33[K\r%s %s %s %s %s" "$file" "$line" "$col" "$from" "$to"
# Note: We need LANG=C as columns are apparently expressed in byte offset
LANG=C sed -E "${line}s/^(.{$((col-1))})${from}/\1${to}/" -i "$file"
done
# Then fetch direct code error information and replace exact pattern at the exact places
#
# /code/library/FooWriter.h:35:25: error: unknown type name 'MyFancyView'
# /code/library/FooWriter.h:431:21: error: use of undeclared identifier 'MyFancyView'
info "Execute replacements"
grep -a " error: " /tmp/errors |
grep -aE "${errMatch}" |
sed -e "s/^\([^:]*\):\([^:]*\):\([^:]*\):[^']*'\([^']*\)'.*/\1\t\2\t\3\t\4/" |
grep -aE "\b(${srcAllowed})\b" |
sort -u |
while read -r elt; do
file=$(echo "$elt" | cut -f1)
line=$(echo "$elt" | cut -f2)
col=$(echo "$elt" | cut -f3)
from=$(echo "$elt" | cut -f4)
to="${intermediate[$from]}" || die "Can't find $from in intermediate"
printf "\r\33[K\r%s %s %s %s %s" "$file" "$line" "$col" "$from" "$to"
# Note: We need LANG=C as columns are apparently expressed in byte offset
LANG=C sed -E "${line}s/^(.{$((col-1))})${from}/\1${to}/" -i "$file"
done
# If no changes, bail out
if git diff-index --quiet HEAD --; then
break
fi
# Format code
format
# Commit
if [[ $step == 1 ]]; then
info "Commit result"
git commit -a -m "chore: Automated replacements ($0)"
else
info "Merging commit result"
git commit -a --amend --no-edit
fi
step=$((step+1))
done
info "Final name replacements"
for key in "${!replacements[@]}"; do
from="${intermediate[$key]}"
to="${replacements[$key]}"
echo "$from → $to"
for i in $(git grep "$from" | cut -f1 -d':' | sort -u); do
sed -e "s/\b${from}\b/$to/g" -i "$i"
done
done
info "Remove idempotent types (using X=X)"
for from in "${!replacements[@]}"; do
to="${replacements[$from]}"
echo "$from → $to"
for i in $(git grep -E "^using ${to} = ${to};" | cut -f1 -d':'); do
sed -E "/^using ${to} = ${to};/d" -i "$i"
done
done
# Format code
format
info "Merging commit result"
git commit -a --amend --no-edit
info "Final build"
rebuild
The nasty bugs
You know what it is like: you never saw them, but some people did, and these people are reliable, so there must be something wrong. But nobody on the engineering team managed to reproduce the issues. And the issues are random, and only happen rarely. Very rarely, such as once in a month for a given customer, or even once in a year for another one. And you know what: these bugs are horrible, because it seems you’ll never catch them. They hide, and silently trigger when you are asleep.
First bug report: Can not run an external crash reporter on Linux
This was the nicest one, actually, and it took only few days to investigate it. Basically, in case of crash, a crash reporter was supposed to be ran, and send useful information (stack traces et al.). Unfortunately the crash reporter appeared not to work correctly. It did not start. For an unknown reason (no error messages)
Fortunately, this bug was reproducible internally.
Second bug report: I see log files in my database on Windows
Yes, log files in my database. Or more precisely log files in my database binary files. I was a bit surprised, too. It seems that, randomly, some log files that were supposed to be sent to the logs were redirected to another random file, including database ones.
Obviously, the database was not happy after that.
That was the report after months of investigation: the first ones were just the usual there was a random crash and I have no clue why - I scratched everything and now it works.
These two issues were actually related, and the root cause was the way file descriptors are handled.
Oh, what are file descriptors, but he way ?
All C programmers know the three standard library input/output files: stdin
, stdout
, stderr
. These are the default standard input, output, and error streams. The first one typically receives the console input (keyboard), the later ones allows to display output.
If you look at the stdin manual page, you will see that
A file with associated buffering is called a stream and is declared to be a pointer to a defined type FILE. (…) At program start-up, three streams shall be predefined and need not be opened explicitly: standard input (for reading conventional input), standard output (for writing conventional output), and standard error (for writing diagnostic output).
Basically, these FILE
streams are encapsulating a file descriptor, that is, a lower-level input or output resource (which is generally identified by an integer, ie. an int
), and provide additional features, such as buffering, and offer a set of advanced functions, such as fprintf
to print formatted output.
By default, the input/output streams are connected to the console (Linux, or DOS), but you may redirect them easily - this feature is especially known in the Linux/Un*x world:
ls -l /tmp/foo >/tmp/list 2>/tmp/errorlog
In this very basic shell script example, the ls -l /tmp/foo
command standard output is redirected to the file /tmp/list
, and the standard error output is redirected to /tmp/errorlog
. Note that the >/tmp/list
can also be written as 1>/tmp/list
: the ‘1’ and ‘2’ refer to the file descriptors 1 and 2, which are the standard output and error file descriptors. The standard input is the file descriptor 0:
read A 0</tmp/bar
Why are the standard input/output streams identified by 0, 1 and 2 ? Well, this is just a convention - and, by default, file descriptors are allocated increasingly, and the first three files opened (this is done by the system) are these streams.
We can redirect the standard streams easily using shell commands - but can we do that programmatically, especially for the output streams ?
We have two output file descriptors (1 and 2), and we want to capture them in a file - say, a log file. This is quite convenient, especially for server applications: log files can then be rotated, inspected…
The standard way to reassign a file descriptor is through the dup2
standard library function. This function takes an opened file descriptor, and clone it to the second file descriptor provided. You can also extract the file descriptor beneath a FILE
stream through the fileno
standard library function: here’s a simple code to redirect the standard output to a file:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <assert.h>
/** Redirect stdout and stderr. Returns 0 upon success. **/
static int redirect_std_out_err(const char *log_name) {
int failure = 1;
/* Open log FILE stream. */
FILE *const fp = fopen(log_name, "wb");
if (fp != NULL) {
/* Extract log FILE fd, and standard stdout/stderr ones. */
const int log_fd = fileno(fp);
const int stdout_fd = fileno(stdout);
const int stderr_fd = fileno(stderr);
if (log_fd != -1 && stdout_fd != -1 && stderr_fd != -1) {
/* Clone log fd to standard stdout/stderr ones. */
if (dup2(log_fd, stdout_fd) != -1 && dup2(log_fd, stderr_fd) != -1) {
failure = 0;
}
}
/* Close initial file. We do not need it, as we cloned it. */
fclose(fp);
}
return failure;
}
int main(int argc, char **argv) {
/* Redirect stdout and stderr to a single log file */
if (argc != 2 || redirect_std_out_err(argv[1])) {
return EXIT_FAILURE;
}
/* Print output to both streams */
printf("Hey, stdout is redirected!\n");
fprintf(stderr, "Hey, stderr is also redirected!\n");
return EXIT_SUCCESS;
}
And that’s it! Simple, isn’t it ?
Redirection is what we typically do in my company - I even modified some of the involved code. But sending output to a log file has its drawbacks. Especially regarding log file size: the file is growing, and we need to clear it time to time, or the disk will eventually fill. This is generally done through a “rotate” operation, that is:
File descriptors, more generally, are special resources in most operating systems, as they are inherited when spawning a process. By default, all file descriptors are transmitted to spawned processes (on POSIX systems, fork
clone them, and exec
functions keep them opened by default), and this is a pain: pipes and sockets are also inherited. This may lead to the following case:
The thing is that spawned processed do not give a damn about already opened file descriptors (except for stdio). But the standard is broken for historical reasons, very unfortunately.
The solution is painful: you need to flag all possible opened file descriptors with the FD_CLOEXEC
flag set. This flag explicitly tells the system that the file descriptor shall not be inherited when spawning a process:
open(..., ...)
=> open(..., ... | O_CLOEXEC)
fopen(..., ...)
=> open(..., ... | O_CLOEXEC)
+ fdopen(...)
OR fopen(..., ... + "e")
dup(...)
=> fcntl(..., F_DUPFD_CLOEXEC, (long) 0)
dup2(..., ...)
=> dup3(..., ..., O_CLOEXEC)
opendir(...)
=> open(..., O_RDONLY | O_DIRECTORY | O_CLOEXEC)
+ fdopendir(...)
pipe(...)
=> pipe2(..., O_CLOEXEC)
acept(..., ..., ...)
=> accept4(..., ..., ..., O_CLOEXEC)
socket(..., ..., ...)
=> socket(..., ... | SOCK_CLOEXEC, ...)
Some of these functions are non-standard (such as accept4
) unfortunately. Some other will not be available (Windows). Some old kernels (Linux) will sometimes not support these features - and some old glibc will not work either (especially for the “e” flag). Yes, this is a real pain.
Oh, one last thing: do not dare to use fcntl(fd, F_SETFD, ...)
to clear the FD_CLOEXEC
flag: this is insanely hopeless in a multithreaded environment. Can you guess why ?
Okay, back yo our nasty bugs!
Here are the investigation steps, ordered temporally:
fork()
/execv()
the external crash reporter in a signal handler, with a status code returned by waitpid()
reporting a signal 6 caught during launch (ie. the launched program called abort()
)system()
eithersystem()
either; the command returns the status “return code 1”!system()
!At this stage, it is clear that the issue is related to the output file descriptors.
Further investigation showed that:
dup3()
calldup3()
, standard streams had their FD_CLOEXEC flag setThe solution was simply to use the default dup2()
function for standard streams. These are the only streams we want to be inherited, after all!
The biggest issue is that the bug could not be reproduced at first internally. Until our Q&A team managed to reproduce it (these guys are really good at finding bugs, I must admit) three days in a row.
The logs reports were strange, and totally unrelated. But one log was kind of interesting:
[2014/11/28-10:59:08.514] [info] ...
[2014/11/28-10:59:08.659] [info] ...
[Fatal Error] :2:999961: XML document structures must start and end within the same entity.
The error was a corrupted produced XML document, with another unrelated log inside it, in the middle of the structure. The final error did not have any timestamp, but the previous one was just before midnight (GMT). Humm, a midnight bug ? What are the chances for this being a coincidence ?
I then realized that midnight was the trigger time for a number of maintenance operations, including… log rotation.
But after thoughtfully inspecting the code, everything looked fine. And the bug never popped up on Linux, it was Windows-specific (or at least, much frequent on this operating system)
But there was another clue: all weird logs were actually logs coming from the Java virtual machine (either through Log4j, or through System.out.println()
calls)
When using System.out
(or System.err
) in Java, we are actually using a PrintStream
, which is an object wrapped in various FilterOutputStream
objects for buffering, the final instance being a FileOutputStream
, which is, according to the documentation an:
output stream for writing data to a File or to a FileDescriptor
And, in our case, the FileDescriptor
involved was FileDescriptor.out
and FileDescriptor.err
, that is, Java handles to the standard output and error streams.
Fortunately, the sources of the Open JDK are public, and it is possible to dig into the real code beneath, that is, the native code:
Hummm… there is a Windows-specific implementation for FileOutputStream native calls. strange, isn’t it ?
JNIEXPORT void JNICALL
Java_java_io_FileOutputStream_write(JNIEnv *env, jobject this, jint byte, jboolean append) {
writeSingle(env, this, byte, append, fos_fd);
}
And the writeSingle
call is using a file descriptor returned by GET_FD()
:
void
writeSingle(JNIEnv *env, jobject this, jint byte, jboolean append, jfieldID fid) {
// Discard the 24 high-order bits of byte. See OutputStream#write(int)
char c = (char) byte;
jint n;
FD fd = GET_FD(this, fid);
And this macro just calls GetObjectField(..., IO_handle_fdID)
to retrieve … the “handle” member of the corresponding FileDescriptor class.
What ? A handle ? Not a file descriptor ?
Yep: a HANDLE
- and this was the root cause of all out problems.
Let me give you the reason: on Windows, a standard library handle is actually a kernel HANDLE
wrapped by the C library, nothing more.
It means that if you replace the standard library input/output streams, you’ll just replace the C library streams.
Here is the complete bug scenario on Windows after collecting all the clues:
dup2()
HANDLE
, and replacing by new onesHANDLE
are now totally meaningless (“dangling handles”)HANDLE
stored in its FileDescriptor
internalsSystem.out
calls are now sending data to nowhere (this is bad)HANDLE
System.out
calls are now sending data to a valid – yet unrelated – file (this is very very bad)The root cause is that Java is unaware that the low-level HANDLE
has changed.
The solution is simple: let’s tell him!:
For each System.out
and System.err
OutputStream
instances, we need to:
out
” member field of the FilterOutputStream
class and mark it public (see Field.setAccessible()
)FilterOutputStream
wrapping by calling fetching the “out
” member field recursivelyFileOutputStream
FileDescriptor
through FileOutputStream.getFD()
SharedSecrets.getJavaIOFileDescriptorAccess()
to set the HANDLE
(a long
) value to -1 temporarilyHANDLE
(a long
) through a native call to _get_osfhandle()
HANDLE
using the previous stepsThis is a bit cumbersome - but mixing native code and Java is always cumbersome, so you have to be used to it :)
The two issues were rather different:
dup2()
for HANDLEs), causing random corruptions on files or socketsBut in both cases, a simple standard stream mishandling was the root cause. But the consequences were nasty, the resolution painful, and the time spent unreasonable (the second bug took months to be investigated)
TL;DR: redirecting Stdio is Hard, and prone to nasty bugs if you are not careful enough!
]]>This code won’t behave correctly (ie. partial copy) on many glibc versions on Linux:
strncat(dest, source, SIZE_MAX);
This is a corner case, but this is a bug.
(hint: the correct behavior is to do strcat(dest, source)
equivalent)
It began with code cleanup.
In httrack, there is a lot of legacy (old, dirty) code laying around. Rewriting everything from scratch would probably be a sane decision, but I just don’t have the time (and the courage) to do that, so I’m cleaning up bits and bits when possible, attempting to improve the code quality when possible.
One of the terrible design decision was to use C strings everywhere, and associated strcpy
, strcat
etc. primitives. It was an unfortunate decision, because of the lack of flexibility (immutable capacity), and security issues (buffer overflows that might be introduced).
In this hypothetical case, we’re adding a string to a structure foo:
struct foo {
char filename[1024];
...
};
void append_filename(struct foo *foo, const char *filename) {
strcat(foo->filename, filename);
}
This code will overflow if the passed string added to the existing string overflows the filename
buffer.
The strlcat version exists on BSD, and is a reasonable solution to mitigate this problem. Unfortunately, some people do not like BSD, and decided that these functions will never be part of the glibc. (insert flame here)
Besides, switching to strlcat and its friends would involve checking every single occurrence of strcat, strcpy. etc., probably introducing new bugs!
I then decided to use a modified version of the dangerous libc primitives, using compile-time type checking:
/**
* Check whether 'VAR' is of type char[].
*/
#if (defined(__GNUC__) && !defined(__cplusplus))
/* Note: char[] and const char[] are compatible */
#define HTS_IS_CHAR_BUFFER(VAR) ( __builtin_types_compatible_p ( typeof (VAR), char[] ) )
#else
/* Note: a bit lame as char[8] won't be seen. */
#define HTS_IS_CHAR_BUFFER(VAR) ( sizeof(VAR) != sizeof(char*) )
#endif
#define HTS_IS_NOT_CHAR_BUFFER(VAR) ( ! HTS_IS_CHAR_BUFFER(VAR) )
/**
* Append at most N characters from "B" to "A".
* If "A" is a char[] variable then the size is assumed to be the capacity of this array.
*/
#define strncatbuff(A, B, N) \
( HTS_IS_NOT_CHAR_BUFFER(A) \
? strncat(A, B, N) \
: strncat_safe_(A, sizeof(A), B, \
HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B), N, \
"overflow while appending '" #B "' to '"#A"'", __FILE__, __LINE__) )
The idea was to sniff at compile-time the arguments passed to the modified strncat
version: when an opaque char*
is used, there’s nothing you can do. But when passing a regular char[...]
array, a secure version can be used. This is not perfect (especially because with non-GCC compilers, char[8] are seen as char*), but at least it allows you to quickly mitigate many potential buffer overflows without touching existing code.
But at one point I did this quite daring thing to create the secure strcat version:
#define strcatbuff(A, B) strncatbuff(A, B, (size_t) -1)
The (size_t) -1
value was simply passed as the boundary of strncatbuff
- at some point, we would be doing strncat(dest, source, (size_t) -1)
, and this seemed fine to me.
Soon after, bugs started to appear (strings were copied partially). It took me a bit of time (with the kind help of the bug reporter) to figure out that my daring macro was involved in this bug. This took me a while because the bug only appeared on Linux, on some glibc releases, and quite randomly.
I managed to have a reproducible minimalistic case:
/* strncat_size_t-1_testcase. */
/* gcc -Wall strncat_size_t-1_testcase.c -o strncat_size_t-1_testcase */
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <string.h>
#include <assert.h>
/* strncat_size_t-1_testcase "Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor" */
int main(int argc, char **argv) {
const size_t size = strlen(argv[1]);
char *const buffer = malloc(size + 1);
int success;
buffer[0] = '\0';
strncat(buffer, argv[1], SIZE_MAX);
success = strlen(buffer) == size;
if (!success) {
fprintf(stderr, "** result: '%s'\n", buffer);
}
assert(success);
return success ? EXIT_SUCCESS : EXIT_FAILURE;
}
If you build this program, and use it:
gcc -Wall strncat_size_t-1_testcase.c -o strncat_size_t-1_testcase
./strncat_size_t-1_testcase "Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor"
… then you’ll get an error. Or maybe you’ll get one. Randomly.
The only way to be sure was to carefully read the strncat
standard.
The strncat() function shall append not more than n bytes (a null byte and bytes that follow it are not appended) from the array pointed to by s2 to the end of the string pointed to by s1. The initial byte of s2 overwrites the null byte at the end of s1. A terminating null byte is always appended to the result. If copying takes place between objects that overlap, the behavior is undefined.
With the help of specialists in comp.unix.programmer
, several points must be noticed:
Therefore, strncat(..., ..., SIZE_MAX)
should behave as strcat(..., ...)
.
Yes, yes, this is a corner case. But this is a legit one.
The reason why not all glibc releases were impacted is simple: this is an optimization bug. (insert laughs from experienced developers)
The original strncat source is not used on x86-64 architectures: the optimized SSE2 version is used in place.
The code might have been introduced in 2013 in this commit which was supposed to introduce a “Faster strlen on x64” - but some existing files seems to have been merged at that time.
Unfortunately, the assembly source is uncommented, and I did not have the courage to dig deeper. But this source is buggy.
Funny Remark: By the way, I’m not even convinced that the assembly version is faster at all: code executing speed is probably irrelevant compared to L1/L2/L3/memory bandwidth issues.
First I obviously stopped using SIZE_MAX with strncat, and dispatch to strcat accordingly. This was a daring thing to use this corner case value anyway, prone to … corner-case bugs.
The second step was to fill a bug report in the glibc.
TL;DR: Beware when using corner cases, even when perfectly conforming to the standard. And never “optimize” code by rewriting obvious functions into a 300-lines of undocumented assembly!
]]>According to Wikipedia,
A coucal is one of about 30 species of birds in the cuckoo family. All of them belong in the subfamily Centropodinae and the genus Centropus. Unlike many Old World cuckoos, coucals are not brood parasites.
If you read my previous blog entry on HTTrack’s new hashtables, you probably already know that cuckoo hashing is a rather new (2001) hashing technique to be used for efficient hashtables.
The idea is to have a smarter open-addressing strategy based on kicking an existing entry if a collision occurs. This is why the name Cuckoo hashtable was chosen.
Note that the name is a bit misleading: a typical cuckoo will not only kick an existing foe, but will consequently kill it (nature, you scary!). Here, we only relocate it, reason why I choose the Coucal, a nicer cuckoo member who does not destroy existing entries - I mean, existing eggs!
This implementation has the following features:
This implementation has been tested since July 2013 in HTTrack, and so far, no bugs were reported (the only issues spotted were bugs in upstream code that was corrupting existing keys)
Please have a look at this library implementation: https://github.com/xroche/coucal
TL;DR: cuckoo hashtables are both efficient and smart!
]]>void
void
is a very strange animal. Basically void means nothing, emptiness. A function “returning” void actually does not return anything. On the machine code side, register(s) aimed to hold return values are simply unused in such case.
You won’t be surprised to hear that the size of void is undefined for this reason ; ie. sizeof(void)
is not supposed to exist, even if some compilers have rather dirty extensions that allow to set this size to 1 to do pointer arithmetics.
But what about void*
? A pointer to something that does not exist, what’s that ?
Well, this is the first trap: void*
is not really related to void
. The void*
type is used to store addresses of unspecified types ; ie. you can cast from/to the void*
type, for example when passing generic functions (such as read()
, write()
, memset()
..)
When I say cast, I don’t really mean cast, actually. You can view the pointer to void
as a kind of a supertype, and aliasing rules do not apply per se: casting int*
to void*
is legit and vice-versa. But casting the resulting void*
pointer to long*
, for example, is not, because aliasing rules would be violated by indirectly casting from int*
to long*
. Yes, aliasing rules are a whole lot of fun. We’ll see that later.
Here comes another trap: you can cast anything from/to void*
, except pointer to functions. Yep, in C, pointers to data and functions are not necessarily if the same size. Basically this means that the address space dedicated to data (data/bss segments) and the one dedicated to code (text segment) are not necessarily of the same size (you could have a 32-bit code pointers and 64-bit data pointers for example). Of course this is almost always the case, because the universe would otherwise collapse, and standard functions such as dlsym
would not work anymore.
To quote the holy standard:
The ISO C standard does not require that pointers to functions can be cast back and forth to pointers to data. Indeed, the ISO C standard does not require that an object of type void * can hold a pointer to a function. Implementations supporting the XSI extension, however, do require that an object of type void * can hold a pointer to a function.
Oh, yes.
int (*function)(char* s, int c) = my_function;
Function definitions are sometimes see as tricky by newcomers. They are NOT. Simply replace (*function)
on the left member (the parenthesis are there for operator priority reasons) by a function name, and you just have the classic function declaration.
char
Nice ?In comparison to void*
and function pointers, char
is the nice guy:
int*
safely)char
Is Generally SignedYes, generally, because this is not specified in the standard. This is actually annoying, because when handling strings, you may end up with negative characters for ASCII > 127, and negative values can be promoted to integers.
Here’s a buggy example:
/** Return the next character within a \0-terminated string, or EOF. **/
int my_read_char(const char *buffer, size_t *offs) {
if (buffer[*offs] != '\0') {
return buffer[*offs++]; /* here's the trap */
} else {
return EOF;
}
This function will return negative values for ASCII > 127, and especially the value -1 for ASCII 255 (0xFF), which is also the value of EOF.
You need to explicitly cast to a unsigned char to have a working version:
/** Return the next character within a \0-terminated string, or EOF. **/
int my_read_char(const char *buffer, size_t *offs) {
if (buffer[*offs] != '\0') {
return (unsigned char) buffer[*offs++];
} else {
return EOF;
}
T*
for const T*
, But Not T**
for const T**
The typical example is that:
char*
string may be implicitly cast into const char*
:
char *foo = strdup("Hello, world!");
const char *bar = foo; /* No need for a dirty cast. */
.. but a char**
array of string may NOT be implicitly cast into const char**
:
char *foo[2];
foo[0] = foo[1] = strdup("Hello, world!");
const char **bar = foo; /* Warning. */
The compiler will complaint:
1.c:6:22: warning: initialization from incompatible pointer type [enabled by default] const char *bar = foo; / Warning. */
The reason behind of the same reason why, in Java, you can not cast List<String>
into List<Object>
: a function taking the supertype array version may add an incompatible type without being noticed.
Here’s an example:
static void replace_the_first_element(const char **array) {
static const char *hello = "hello, world!";
array[0] = hello; /* replace first element by a constant string */
}
...
char *foo[2];
foo[0] = foo[1] = strdup("Hello, world!");
replace_the_first_element(foo);
In this example, without a cast warning, the function replace_the_first_element
would silently replace the first element of foo by a non-constant string, and the caller would have a constant string within the array when the function returns. The code is strictly equivalent to:
static const char *hello = "hello, world!";
char *foo[2];
foo[0] = hello; /* now you can see the problem */
I would say that we should be able to silently cast T**
into const T*const*
(an array of constant arrays of T), but anyway …
Quizz time: what the test
function below is supposed to print ? (don’t cheat!)
static void test(char foo[42]) {
char bar[42];
printf("foo: %zu bytes\n", sizeof(foo));
printf("bar: %zu bytes\n", sizeof(bar));
}
Well, you might be surprised to see:
foo: 8 bytes
bar: 42 bytes
Yes, an array of type T
in a function argument list is strictly equivalent of declaring the type as T*
. This is why nobody uses this confusing syntax within function argument list.
To quote the standard:
“A declaration of a parameter as ‘array of type’ shall be adjusted to ‘qualified pointer to type’, where the type qualifiers (if any) are those specified within the [ and ] of the array type derivation.”
This also imply that the passed object is passed by reference, not value.
… and this is totally inconsistent with the handling of structures as function argument, or as return type. Yes, yes, historical mistakes.
switch
can be seen as a dispatch function, and case
as label. Each switch
must have its case, but you are free to interleave loops of you want, such as in:
switch (step) {
while (1) {
case step_A:
case step_B:
...
}
}
See my previous An Unusual Case of Case (/switch) entry for more fun!
Let’s say I have a char foo[42]
and I want a pointer to this array in bar
. I can write:
What shall I use ? The first and third pointers are identical: they are char*
ones, the one you generally want to use: the pointer points to a char area of undefined size. The second one is a pointer to the array of 42 chars itself. You may then use:
The pointer to the array of 42 chars syntax is a bit weird for newcomers, but it is logical: *bar
is an array of 42 chars, so you just have to replace foo
by (*bar)
in the standard definition to have the correct syntax (the parenthesis are needed for operators priority reasons, otherwise you’ll declare an array of 42 pointer to char). (This trick is also helpful to understand the apparent obscure pointer to function syntax.)
The advantage of the second definition is that you still have a pointer to an array of 42 char: sizeof(*bar)
is 42, *bar
is of type char[]
(suitable for specific compiler optimizations and safe string operations)
struct { } foo;
this code is undefined in C: an empty structure is not officially supported. Some compilers such as GCC support them, and define the size of the structure as 0.
It is supported in C++ (because many classes do not have any member within) and in such case the sizeof
of such empty structure is … 1, because C++ needs a different pointer for every instance of a given class. Oh, and because you need to do pointer arithmetics also, and having a null size would cause issues.
char
Arrays, And Are Not Const (But They Really Are)Another corner case: string literals, such as "hello world"
are actually arrays of char
, with a terminating \0
. The two definitions are equivalent:
char *foo = "hi";
static const char hi[] = { 'h', 'i', '\0' };
char *foo = (char*) hi;
Note that sizeof("hi")
is equal to 3 (an array of 3 chars).
Oh, did you notice the (char*)
in the second example ? Isn’t it a bit dirty ?
Let’s have a look at two code samples:
int i = 0, j = 0;
foo(i++, j+=i, j++);
int i = 0, j = 0;
bar((i++, j+=i, j++));
The first sample calls a function named foo
taking three integers as arguments. Is has an undefined behavior: side-effects of i
and j
will be executed at an undefined time, and might even be optimized. You basically don’t have a clue of what foo
will receive as arguments.
The second sample calls a function named bar
taking one integer argument. It has a perfectly defined behavior. This integer is the result of the comma operator
i++, j+=i, j++
, which can be rewritten as the pseudo-C++-code:
static int my_comma(int &i, int &j) {
i++;
j+=i;
return j++;
}
...
int i = 0, j = 0;
bar(my_comma(i, j));
The comma operator evaluates each expression starting from the leftmost member to the rightmost member, and yields the last member as value. All other values are discarded (each expression should yield void
, actually). And between each member evaluation, a sequence point is committed: side-effects of each expression are committed for the next expression. This is why this expression has a perfectly defined behavior.
If
Two pretty nice feature of the “boolean and” (&&
) and the “boolean or” (||
) operators is that they:
if (foo[i++] != '\0' && bar[i++] != '\0')
is perfectly defined, as the post-increment of i will be committed before the evaluation of the right expression (ie. a sequence point exists between left and right operator evaluation)if (foo != NULL && *foo != '\0')
without fearing of a NULL pointer dereferencingchar buffer[1024] = ""
Not only is it a bit dirty to have large buffers on the stack (I must admit that some legacy code in httrack is filled with that, cough cough), but the C standard ensure that when initializing a structure or an array, missing elements shall be initialized to zero.
This basically means here that the first element of the array is explicitly initialized to zero (this is the terminating \0 of the empty string), and the 1023 other elements will be implicitly initialized to zero, too. The performance impact is the same as memset(buffer, 0, sizeof(buffer))
.
Use at least an explicit buffer[0] = '\0'
without any initializer when declaring the buffer.
strchr
(and Friends) DefinitionsDid you notice that ?
char *strchr(const char *s, int c);
char *strstr(const char *haystack, const char *needle);
...
All these function takes a const char*
and return a char*
. How can a function transform an array of constant chars to a non-constant one ?
Well, obviously this is impossible, and the only reason of this oddity was to offer a somewhat generic function that would work for both char*
and const char*
strings: accepting const char*
as argument will also accept char*
versions by silently casting it, and returning char*
will also allow to store the result in a const char*
for the same reason.
Unfortunately, this may lead to:
char *pos = strchr(my_const_string, ' ');
if (pos != NULL) {
*pos++ = '\0'; /* oops, I forgot that the string was constant, and the compiler did not warn me! */
/* kaboom here */
}
A sane decision would be to split all these string handling functions into const and non-const ones. In C++, you can have specialized versions.
Dividing an integer by zero leads to a floating point exception (Yes, this is not a floating point operation, but the error triggers one.).
But dividing the smallest signed integer by -1 also leads to the same floating point exception. This is a bit of a curiosity: when dividing by -1 the smallest integer (-2147483648
), the result should be 2147483648
, but is too large (by one) to be represented in an integer (the largest integer is 2147483647
), and the divide operation then triggers the same exception that the one triggered by the division by zero.
However, multiplying the smallest signed integer by -1 does not trigger anything because… well, because the multiply operation does not raises anything, even when overflowing.
Therefore, the first printf below will gently print a result, but the second one won’t, and the program will abort:
/* Note: using volatile to prevent the div to be optimized out */
volatile int int_min = INT_MIN;
volatile int minus_1 = -1;
int main(int argc, char **argv) {
printf("%d\n", int_min * minus_1);
printf("%d\n", int_min / minus_1);
return EXIT_SUCCESS;
}
By the way, what’s the first result ? Well, -2147483648
, because the 32-bit multiply operation of two 32-bit numbers provides a 64-bit result, which is 2147483648
(0x0000000080000000 in hexadecimal), and the lowest part is just truncated to the 32-bit 0x80000000
value, which is… -2147483648
in two’s complement arithmetics. Basically the overflow was silently ignored. Got it ?
Macros are often considered evil (and will even cause C++-fans to scream and have foam on their mouth), especially when hiding multiple argument evaluation:
#define min(a, b) ( (a) < (b) ? (a) : (b) )
This function will not behave well when used with something like min(++i, ++j)
: a second increment will be committed in either i or j…
Another difference between a function version and its macro version are the side effects, which are committed before and after calling a function, but not a macro.
You know that __FILE__
is a magic preprocessor variable holding the current source filename, and __LINE__
the line number at the time of the macro evaluation. This is quite convenient to print debugging:
#define PRINT_CURRENT_FILE_AND_LINE printf("at %s:%d\n", __FILE__, __LINE__)
...
int main(...) {
...
PRINT_CURRENT_FILE_AND_LINE; /* print some debug */
...
The __FILE__
and __LINE__
macros will be expanded to the location in the main() function, at expansion time.
You can use the expansion time behavior to convert the __LINE__
numerical token to a string one, for example using a double macro expansion:
#define L_(TOKEN) #TOKEN
#define L(TOKEN) L_(TOKEN)
#define FILE_AND_LINE __FILE__ ":" L(__LINE__)
int main(int argc, char **argv) {
printf("we are at: %s\n", FILE_AND_LINE);
return 0;
}
A call to FILE_AND_LINE
will form a string using the three substrings (in C, “foo” “bar” is equivalent to “foobar”). We need an intermediate L_()
macro because otherwise __LINE__
would not be expanded, and the resulting string would simply be … "__LINE__"
.
My headache is killing me, so I’m going to redirect you to the excellent article on cellperformance. But aliasing rules are very, very, very fun, trust me!
Do not hesitate to comment this article to complete it! There are other C-corner cases I probably didn’t mention…
TL;DR: corner cases are not reserved to C++!
]]>vtable
And Other Monsters
C++ inheritance, virtual functions, and virtual tables are sometimes sees as obscure beasts, something most people will not mess with.
Casting an object to another one or calling a virtual function are probably two of the most common operations you can think of in C++, so it is a shame that such simple operations are not fully understood by everyone.
I have been programming in C
for many years, and one of the benefits of C
is that there is no real magic: when you define a structure, you can see the layout in memory, you can feel the pointers being padded, and every other operations, such as casting a pointer to another one (something you should only do with great care, by the way) are just re-interpreting memory addresses from one type to another one or shifting addresses a bit. No magic. No surprises. Accessing the second member of a pointer array ? This is just basic pointer arithmetic (indirect move with shifting and index, something that even the venerable 68000
processor had if my memory is correct)
Some people go straight in C++
(or even in Java
) when they start learning programming - which is in my point of view a bad thing - and tend to lose their focus on what’s really happening behind the curtains (ie. inside the memory)
But things are generally much simpler that you thought.
Let’s first have a look at a very simple object. A C++ one:
class A {
private:
int id;
public:
A(): id(42) {
printf("yay, my address is %p, and my id is %d!\n", this, this->id);
}
};
This class contains only a constructor, and has one member (an integer). These two things are not of the same nature: variable members and functions are of different kind (yes, this is a really smart and insightful remark, I know).
To make things clearer, we can view this class as a C-structure, and its functions as global functions taking a (hidden) “this” member as first argument (the pointer to the structure itself). The instance members can be accessed implicitly, without the this->
pointer indirection, by the way: C++ will add the missing bits.
The code above can be rewritten in pure C as follows:
// this is the actual "class A" instance data layout!
struct A {
int id;
};
// this is the class A equivalent constructor (a global function)
void A_A(struct A* this) {
this->id = 42;
printf("yay, my address is %p, and my id is %d!\n", this, this->id);
}
… and, rather that just declaring:
A a;
… you’ll need to call the constructor manually:
struct A a;
A_A(&a); // call constructor manually!
Here it is! C++ in plain C! Of course a C++ compiler will hide everything behind the curtains when using regular classes, but the actual compiled code will be more or less the same.
If you add more members and/or more methods in this class, the principle is the same: extend the structure layout, and/or add more global functions that take an hidden this first argument.
Quizz: what is the size of a class A
object instance ? The answer is pretty simple: the class only contains one member (id
), which is an integer (32-bit here) ; the sizeof(A)
is therefore 4 (bytes). No magic.
Quizz: what will be the size of a class A
object instance if a regular member function is added ? The answer is pretty simple, too: its size won’t change, as it still contains only one member - only an additional global function will be added in the code.
No, no, not at all.
Let’s take a simple (yet representative) example: our previous A class, another B class with a single “age” member, and at last a final class C with a member “mode”, but inheriting from A and B.
class A {
public:
int id;
A(): id(42) {
printf("A: yay, my address is %p, my size is %zu, and my id is %d!\n",
this, sizeof(*this), this->id);
}
};
class B {
public:
int age;
B(): age(7) {
printf("B: yay, my address is %p, my size is %zu, and my age is %d!\n",
this, sizeof(*this), this->age);
}
};
class C: public A, public B {
public:
int mode;
C(): mode(-1) {
printf("C: yay, my address is %p, my size is %zu, my id, age and mode are %d, %d, %d!\n",
this, sizeof(*this), this->id, this->age, this->mode);
}
};
Instantiating an object from C (C c;
) will print something like:
A: yay, my address is 0x73620f9f5c20, my size is 4, and my id is 42!
B: yay, my address is 0x73620f9f5c24, my size is 4, and my age is 7!
C: yay, my address is 0x73620f9f5c20, my size is 12, my id, age and mode are 42, 7, -1!
The class C
constructor called first the upper inherited classes constructors (class A
constructor and class B
constructor), and therefore prints their information first. Here again, C++
hides everything behind curtains: the calls to base classes constructors are generated automatically, before your actual class C
constructor code.
You can note that the address seen by the constructors of A and C are identical, but not the one seen by B. How can it be ?
Let’s rewrite everything in plain C:
struct A {
int id;
};
// class A constructor
void A_A(struct A* this) {
this->id = 42;
printf("A: yay, my address is %p, my size is %zu, and my id is %d!\n",
this, sizeof(*this), this->id);
}
struct B {
int age;
};
// class B constructor
void B_B(struct B *this) {
this->age = 7;
printf("B: yay, my address is %p, my size is %zu, and my age is %d!\n",
this, sizeof(*this), this->age);
}
struct C {
struct A a;
struct B b;
int mode;
};
void C_C(struct C *this) {
// manually call base classes constructors first
A_A(&this->a);
B_B(&this->b);
// then, user code
this->mode = -1;
printf("C: yay, my address is %p, my size is %zu, my id, age and mode are %d, %d, %d!\n",
this, sizeof(*this), this->a.id, this->b.age, this->mode);
}
The struct C
layout is pretty straightforward: the first element is the struct A
members - the reason why constructors from class A and class C did see the same address, the second member is the member(s) of class B, and finally comes the member(s) of C.
In both C++ and C variants,
Casting will provide the same behavior, by the way:
static_cast<A*>(&c))
== &(&c)->a
== 0x73620f9f5c20static_cast<B*>(&c))
== &(&c)->b
== 0x73620f9f5c24static_cast<C*>(&c))
== &c
== 0x73620f9f5c20Well, let’s consider a simple example: simply add a virtual function print
to our previous sample, aiming to print the object class name and parameters. The idea is that even if we cast
the high-level class C
instance into A*
or B*
, we’ll see the same behavior.
class A {
public:
int id;
A(): id(42) {
printf("A: yay, my address is %p, my size is %zu, and my id is %d!\n",
this, sizeof(*this), this->id);
}
virtual void print() {
printf("I am A(%d)\n", id);
}
};
class B {
public:
int age;
B(): age(7) {
printf("B: yay, my address is %p, my size is %zu, and my age is %d!\n",
this, sizeof(*this), this->age);
}
virtual void print() {
printf("I am B(%d)\n", age);
}
};
class C: public A, public B {
public:
int mode;
C(): mode(-1) {
printf("C: yay, my address is %p, my size is %zu, my id, age and mode are %d, %d, %d!\n",
this, sizeof(*this), this->id, this->age, this->mode);
}
virtual void print() {
printf("I am C(%d, %d, %d)\n", id, age, mode);
}
};
If we create an instance of C
called c
(C c;
), the following calls will produce exactly the same output:
Where is the magic ? Let’s first have a look at the output, when creating this object:
A: yay, my address is 0x726fa25b7c00, my size is 16, and my id is 42!
B: yay, my address is 0x726fa25b7c10, my size is 16, and my age is 7!
C: yay, my address is 0x726fa25b7c00, my size is 32, my id, age and mode are 42, 7, -1!
We can see that A
and B
are not anymore 4 bytes, but 16. And C is 32 bytes! Somehow, the C++
compiler did add some bytes in our objects without telling us!
Let’s have a look at each object instance actual contents within each classes, by dumping data as pointers in constructors, to figure ou what has been added inside our beloved structures classes:
// simple dump function
static void print_object(const char *name, void *this_, size_t size) {
void **ugly = reinterpret_cast<void**>(this_);
size_t i;
printf("created %s at address %p of size %zu\n", name, this_, size);
for(i = 0 ; i < size / sizeof(void*) ; i++) {
printf(" pointer[%zu] == %p\n", i, ugly[i]);
}
}
And using print_object(__FUNCTION__, this, sizeof(*this));
inside each constructor.
Remember that the constructors are called as follows when creating an instance of C:
this
points to the beginning of the allocated C structure)this
starts at 16 bytes from the beginning of the allocated C structure)this
points to the beginning of the allocated C structure)Here’s the result:
created A at address 0x7082f962ccb0 of size 16
pointer[0] == 0x400f20
pointer[1] == 0x2a
created B at address 0x7082f962ccc0 of size 16
pointer[0] == 0x400f00
pointer[1] == 0x7
created C at address 0x7082f962ccb0 of size 32
pointer[0] == 0x400ed0
pointer[1] == 0x2a
pointer[2] == 0x400ee8
pointer[3] == 0xffffffff00000007
We can see that:
0x400f20
), and the id variable (0x2a
== 42
) padded to 8 bytes0x400f00
), and the age variable (7
) padded to 8 bytes0x2a
, and 0xffffffff00000007
(00000007
for the age variable, and ffffffff
for the mode variable: -1
is 0xffffffff
in two complement’s world) - and the two additional pointersSo it seems that the structure size change reason is that an hidden pointer member (8 bytes in the 64-bit world) has been added to the class instance - and the compiler probably padded the structure to 16 bytes, because you want to have 8-byte alignment when dealing with arrays: if you have an array of two objects, the second object hidden pointer has to be padded to 8 bytes, enforcing the structure to be a multiple of 8 bytes. The 4 bytes used for padding are unused, but useful to maintain the alignment.
Hey, wait! The two pointers are different, when printed inside the C constructor! Yes, good catch: the additional A pointer which was 0x400f20
inside the class A
constructor has been changed into 0x400ed0
inside the class C
constructor, and the additional B pointer which was 0x400f00
inside the class B
constructor has been changed into 0x400ee8
inside the class C
constructor.
Oh, and a last note: C is only 32 bytes.
class B
for the mode
variable (smart, isn’t it ?)Okay, so two magical pointers have been added because we defined a virtual function. What does it mean ?
Let’s go back to basic: how would you implement this class hierarchy in pure C ?
Here’s what you may do:
struct A {
int id;
};
// class A constructor
void A_A(struct A *this) {
this->id = 42;
print_object(__FUNCTION__, this, sizeof(*this));
}
void A_print(struct A *this) {
printf("I am A(%d)\n", this->id);
}
struct B {
int age;
};
// class B constructor
void B_B(struct B *this) {
this->age = 7;
print_object(__FUNCTION__, this, sizeof(*this));
}
void B_print(struct B *this) {
printf("I am B(%d)\n", this->age);
}
struct C {
struct A a;
struct B b;
int mode;
};
void C_C(struct C *this) {
// manually call base classes constructors first
A_A(&this->a);
B_B(&this->b);
// then, user code
this->mode = -1;
print_object(__FUNCTION__, this, sizeof(*this));
}
void C_print(struct C *this) {
printf("I am C(%d, %d, %d)\n", this->a.id, this->b.age, this->mode);
}
Okay, looks more or less what we had previously. But what about the virtual function ? We have to be sure that we call the right function, even if we only have a A* or a B* pointer (and we don’t know that this is actually a C object class behind).
A solution can be to add a function pointer for every virtual function in each classes, and initialize inside the constructor!
Something like this:
struct A {
void (*print)(struct A *this);
int id;
};
// class A constructor
void A_A(struct A *this) {
this->print = A_print;
this->id = 42;
print_object(__FUNCTION__, this, sizeof(*this));
}
struct B {
void (*print)(struct B *this);
int age;
};
// class B constructor
void B_B(struct B *this) {
this->print = B_print;
this->age = 7;
print_object(__FUNCTION__, this, sizeof(*this));
}
void B_print(struct B *this) {
printf("I am B(%d)\n", this->age);
}
struct C {
struct A a;
struct B b;
void (*print)(struct C *this);
int mode;
};
void C_C(struct C *this) {
// manually call base classes constructors first
A_A(&this->a);
B_B(&this->b);
// then, user code
this->print = C_print;
this->a.print = C_print; // we patch our own A->print function !!
this->b.print = C_print; // and we need to do it for B->print too!!
this->mode = -1;
print_object(__FUNCTION__, this, sizeof(*this));
}
void C_print(struct C *this) {
printf("I am C(%d, %d, %d)\n", this->a.id, this->b.age, this->mode);
}
What did we do here ?
print
print
after A and B constructors, and overrides the one inside A and BNeat, isn’t it ?
Now, whatever pointer you have, these three lines will produce identical behavior:
Hey, by the way, we do not need an additional pointer for C, because we already have the correct version in two locations: a.print
and b.print
. Let’s remove this useless pointer in C (and the useless initialization), and use:
There is a little problem, however, when using a C instance cast in A* or B*. When you call a->print(a)
, the a
pointer is of type A*. And therefore the assignment this->a.print = C_print
will produce a warning. Similarly, when you call b->print(b), the this
pointer will have a type of B*, and not only it is of the wrong type, but it has a different address, as c->b is located sixteen bytes after the beginning of C.
We need specialized versions with correct casts!
Here we go:
void C_print(struct C *this) {
printf("I am C(%d, %d, %d)\n", this->a.id, this->b.age, this->mode);
}
void CA_print(struct A *this) {
C_print((struct C*) this);
}
void CB_print(struct B *this) {
// we know that this points after the struct A layout
// so let's fix the pointer by ourselves!
size_t offset = (size_t) &((struct C*) NULL)->b;
C_print((struct C*) ( ( (char*) this ) - offset ) );
}
void C_C(struct C *this) {
// manually call base classes constructors first
A_A(&this->a);
B_B(&this->b);
// then, override functions
this->a.print = CA_print; // we patch our own function !!
this->b.print = CB_print; // and we need to do it for B too!!
// this->print = C_print; // useless
// then, user code
this->mode = -1;
print_object(__FUNCTION__, this, sizeof(*this));
}
With these adjustments, things are smooth:
c.a.print(&c.a)
==> I am C(42, 7, -1)c.b.print(&c.b)
==> I am C(42, 7, -1)c.print(&c)
==> I am C(42, 7, -1)Perfect, isn’t it ?
Not yet: what if we add more virtual functions in our structures ? This will increase the object size, and require more pointer initialization in constructors! What a waste of space and time: we are assigning the same pointers each time. Could we prepare the set of pointers for each classes in a static, global structure, and just initialize a pointer to these static data inside the constructors ? Neat, isn’t it ?
Here’s the modified version - let’s call the global structure tables containing specialized function pointers vtbl
, as in virtual table, because this is actually what they are: table for virtual functions!
// type for global virtual function table structure A
// this structure enumerates all virtual functions of A
struct A;
struct table_A {
void (*print)(struct A *this);
};
struct A {
const struct table_A *vtbl;
int id;
};
void A_print(struct A *this) {
printf("I am A(%d)\n", this->id);
}
// the global, static table data for A
// this structure enumerates all virtual functions of B
static const struct table_A table_A_for_A = { A_print };
void A_A(struct A *this) {
this->vtbl = &table_A_for_A;
this->id = 42;
print_object(__FUNCTION__, this, sizeof(*this));
}
// type for global virtual function table structure B
struct B;
struct table_B {
void (*print)(struct B *this);
};
struct B {
const struct table_B *vtbl;
int age;
};
void B_print(struct B *this) {
printf("I am B(%d)\n", this->age);
}
// the global, static table data for B
static const struct table_B table_B_for_B = { B_print };
void B_B(struct B *this) {
this->vtbl = &table_B_for_B;
this->age = 7;
print_object(__FUNCTION__, this, sizeof(*this));
}
struct C {
struct A a;
struct B b;
int mode;
};
void C_print(struct C *this) {
printf("I am C(%d, %d, %d)\n", this->a.id, this->b.age, this->mode);
}
void CA_print(struct A *this) {
C_print((struct C*) this);
}
void CB_print(struct B *this) {
// we know that this points after the struct A layout
// so let's fix the pointer by ourselves!
size_t offset = (size_t) &((struct C*) NULL)->b;
C_print((struct C*) ( ( (char*) this ) - offset ) );
}
// the global, static tables data for A and B structures, for C
static const struct table_A table_A_for_C = { CA_print };
static const struct table_B table_B_for_C = { CB_print };
void C_C(struct C *this) {
// manually call base classes constructors first
A_A(&this->a);
B_B(&this->b);
// Override virtual function pointers that have just been initialized by
// A and B constructors with our own version. This allows to override all
// possible virtual functions in base classes!
this->a.vtbl = &table_A_for_C;
this->b.vtbl = &table_B_for_C;
this->mode = -1;
print_object(__FUNCTION__, this, sizeof(*this));
}
Neat, isn’t it ?
Well, I’m telling you a secret: this is more or less what the C++ compiler does (yes, behind the curtains) when implementing virtual functions. The virtual function tables are called virtual tables, and are static structures inside the compiled code. And inside each corresponding class constructor, additional code is inserted just between base class initialization calls, and user code, to set all virtual table pointers corresponding to the class type.
Some additional remarks on this design:
if you call a virtual function inside a base class constructor, you will not call the top-level function version, because the virtual table pointer has not yet been initialized (it will be initialized after the end of the base class constructor)
you need one static structure containing all necessary stuff for a given class
you can put additional information on these virtual table structures, which can be useful!
getClassType()
virtual function… and this is exactly what most compiler do, actually: Runtime Type Information data is generally included as soon as a virtual table needs to be created for a given class (ie. you can use dynamic_cast<>())
Now you know what’s going on when you define a virtual function: rather than calling the function directly, an indirection through a virtual table is done.
This has some drawbacks:
a little function call impact on performances, because you need to read the actual function pointer in a static table
a potentially huge impact on performances because you can not inline anymore function calls
Let’s dig a little bit more, using GCC version of the virtual tables.
If you run your program under GDB and examine an instance of class C
, you’ll see:
(gdb) p c
$1 = {<A> = {_vptr.A = 0x400e70 <vtable for C+16>, id = 42}, <B> = {_vptr.B = 0x400e88 <vtable for C+40>, age = 7}, mode = -1}
Note the _vptr.A = 0x400e70
and _vptr.B = 0x400e88
: these are the virtual tables.
What is behind these pointers ? Recent GCC releases allows you to dig a bit into an object’s virtual table:
(gdb) info vtbl c
vtable for 'C' @ 0x400e70 (subobject @ 0x775bfbf98600):
[0]: 0x400b00 <C::print()>
vtable for 'B' @ 0x400e88 (subobject @ 0x775bfbf98610):
[0]: 0x400b34 <non-virtual thunk to C::print()>
Otherwise, you can easily guess that each vtables starts with the same kind of function array we created for our C version:
(gdb) p ((void**)0x400e70)[0]
$5 = (void *) 0x400b00 <C::print()>
(gdb) p ((void**)0x400e88)[0]
$9 = (void *) 0x400b34 <non-virtual thunk to C::print()>
Now you understand a bit more the concept or vtable, let’s see a dirty-but-fun hack (valid for GCC, possibly for Visual C++ too): how to override a specific object’s virtual function table, and rewire virtual functions at runtime
// Our patched version of C::printf (or A::printf)
static void our_function(C *thiz) {
printf("I am now a new C(%d, %d, %d)!\n", thiz->id, thiz->age, thiz->mode);
}
// Our patched version of B::printf
static void our_function_from_B(B *thiz) {
// This is the same boring arithmetic than before, except that
// we use a fake C instance located at a random (but non-NULL) address
// to compute the offset. Note that the actual instance address will
// never be dereferenced, so do not fear illegal accesses!.
C *dummy = reinterpret_cast<C*>(0x42);
B *dummy_B = static_cast<B*>(dummy);
char *dummy_ptr = reinterpret_cast<char*>(dummy);
char *dummy_B_ptr = reinterpret_cast<char*>(dummy_B);
size_t offset = dummy_B_ptr - dummy_ptr;
char *ptr = reinterpret_cast<char*>(thiz);
C *thiz_C = reinterpret_cast<C*>(ptr - offset);
our_function(thiz_C);
}
int main(int argc, char **argv) {
// create an instance of C
C c;
// Hack the c instance and override its print() function!
void *our_vtbl[1] = { reinterpret_cast<void*>(our_function) };
void *our_vtbl2[1] = { reinterpret_cast<void*>(our_function_from_B) };
// patch for A* and C*
*reinterpret_cast<void**>(static_cast<A*>(&c)) = our_vtbl;
// patch for B*
*reinterpret_cast<void**>(static_cast<B*>(&c)) = our_vtbl2;
// get the pointers (note: GCC will optimize direct calls to c.printf)
A *pa = &c;
B *pb = &c;
C *pc = &c;
// and call printf() ... what will be printed ?
pa->print();
pb->print();
pc->print();
return 0;
}
And yes, your object was hacked:
I am now a new C(42, 7, -1)!
I am now a new C(42, 7, -1)!
I am now a new C(42, 7, -1)!
Neat, isn’t it ? (yes, yes, not really portable though)
You may have a look at the Itanium C++ ABI, Virtual Table Layout description (GCC is using the Itanium ABI for vtables, or at least something very close)
For Windows coders, the Reversing Microsoft Visual C++ Part II page will be a goldmine, too
TL;DR: remember that C++ used to be enhanced C a long time ago, and most basic features can be easily implemented and understood through plain C code!
]]>It began with a simple macro: (approximative code)
#define ADD_BYTE(C) do { \
if (offset == capa) { \
if (capa < 16) { \
capa = 16; \
} else { \
capa <<= 1; \
} \
buffer = realloc(buffer, capa); \
assert(buffer != NULL); \
} \
buffer[offset++] = (C); \
} while(0)
For the C
-illiterates, this simple macro just adds the byte ‘C’ to the dynamically allocated buffer buffer
, whose capacity (in bytes) is capa
. The next position to be written is identified by offset
. The capacity is doubled each time the buffer is filled (starting with a minimum of 16 bytes).
We’re adding bytes in a dynamic buffer - this is one of the most common thing you can do in a given program (used in strings, in arrays, etc.)
The problem is to understand if the reallocation strategy is good enough. If you take a look at the complexity, assuming realloc
is O(N), you quickly conclude that adding a single byte is on average O(1), which is good.
But what is the worst-case complexity, when reallocating a buffer ?
Guy: Are you sure this is the good strategy ? It looks like you have a big performance issue if you extend the size of a large array - say, one gigabyte for example. Imagine the impact, especially if the buffer has to be moved from the swap.
Me: Hummm, I never really thought about this, but I think it’s fine. I’m pretty sure the system can handle that gently, actually.
Guy: I think a linked structure might be a better idea - maybe with the same exponential strategy, for example
Me: I’m pretty sure this is a waste of time
Guy: Okay, show me
Okay, now I need to back my arguments. As all programmers, I am lazy. I should say: as all programmers should be, I am lazy. Laziness is an incentive to be smarter and more efficient. You are too lazy to execute repetitive or boring tasks, and you need something smarter, something faster. This is sometimes called laziness. I prefer to call that efficiency.
Having a linked list of blocks to store a buffer would be a bit cumbersome. I didn’t say impossible - Impossible n’est pas français as we say sometimes (well, it did not end well for Napoléon, but nevermind) - but cumbersome. Copying a sub-array or storing it on disk would require some work, we probably would need an index of all arrays base pointers, get the low offset 2-based logarithm to get the initial block, and do some other boring stuff. Sheesh, I’m already tired.
I have the feeling that being lazy is the right thing to do here. Because for small memory blocks, we basically do not care, and for large blocks, the system’s virtual memory will gently take care of the issue.
Let’s have a look.
realloc()
, By The Way ?It is a regular function, standardized by POSIX, located in the C library. On linux, you can find it inside the libc.so
library, with its siblings malloc
and free
:
nm -D /lib/x86_64-linux-gnu/libc.so.6 | grep -E "T (malloc|realloc|free)$"
000000000007e450 T free
000000000007e030 T malloc
000000000007e4e0 T realloc
For the curious, “T
” stands for “text symbol” (the uppercase means that the symbol is public and visible) ; the text segment of a program being the code, the data segment being initialized data (variables), and the bss segment being uninitialized (well, actually initialized to zero) data (variables).
Allocating memory, reallocating memory and freeing memory is done through an allocator (thank you Captain Obvious!). There are plenty of them (one of the most common being the buddy allocator).
We can implement our own trivial allocator using the venerable sbrk
call, which basically expand the end of the data segment to have more space:
#include <sys/types.h>
#include <unistd.h>
#include <string.h>
void *malloc(size_t size) {
return sbrk(size);
}
void free(void *ptr) {
/* who cares ? */
}
/* Note: meant to grow only (yes, I am lazy) */
void *realloc(void *ptr, size_t size) {
void *nptr = malloc(size);
if (nptr == NULL) {
return NULL;
}
// 'old_size' is supposed to be known :)
// (for example put in a header before the actual block)
memcpy(nptr, ptr, old_size);
free(ptr);
return nptr;
}
[ Note: as someone noted on hacker news, the old_size address needs to be known here ; for example by using a header before the returned address. And no, we shouldn’t free the original block upon realloc failure :) ]
A real allocator would be a bit more clever, of course, with complex data structures aimed to limit the number of actual memcpy
calls.
We need to dig a bit inside the Glibc used on Linux to figure out what’s happening for large blocks.
Just download a recent glibc release, and have a look at the source tree.
There is an interesting directory out there: the malloc
directory. Just pick the malloc.c
file and open it in an editor.
/*
The main properties of the algorithms are:
* For large (>= 512 bytes) requests, it is a pure best-fit allocator,
with ties normally decided via FIFO (i.e. least recently used).
* For small (<= 64 bytes by default) requests, it is a caching
allocator, that maintains pools of quickly recycled chunks.
* In between, and for combinations of large and small requests, it does
the best it can trying to meet both goals at once.
* For very large requests (>= 128KB by default), it relies on system
memory mapping facilities, if supported.
For a longer but slightly out of date high-level description, see
http://gee.cs.oswego.edu/dl/html/malloc.html
*/
The interesting part (for us) is: For very large requests (>= 128KB by default), it relies on system memory mapping facilities, if supported..
The 128KB by default can actually be controlled by the mallopt
function:
M_MMAP_THRESHOLD For allocations greater than or equal to the limit specified (in bytes) by M_MMAP_THRESHOLD that can't be satisfied from the free list, the memory-allocation functions employ mmap(2) instead of increasing the program break using sbrk(2).
Okay, so as I said previously, the system’s virtual memory will handle everything when dealing with large blocks.
Basically, it means that:
malloc
will be trunked to anonymous mmap
free
will be trunked to munmap
realloc
will be trunked to mremap
(Darn, not yet in POSIX ? Guys ?)Okay, so we need to dig a little deeper inside mremap.
A man mremap
will tell us that:
mremap() uses the Linux page table scheme. mremap() changes the mapping between virtual addresses and memory pages. This can be used to implement a very efficient realloc(3).
Humm, very efficient realloc. How efficient ?
First, mmap
, munmap
, and mremap
do have definitions in the glibc. Well, pretty small entry points, actually:
nm -D /lib/x86_64-linux-gnu/libc.so.6 | grep -E "(mmap|munmap|mremap)$"
00000000000e4350 W mmap
00000000000e9080 W mremap
00000000000e4380 W munmap
Note that the default entry points are weak symbols in this case, which basically means that they can be overridden by someone else, during dynamic link ; eg. in this case:
ldd /tmp/sample
linux-vdso.so.1 (0x00007584a8aaa000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007584a86e6000)
/lib64/ld-linux-x86-64.so.2 (0x00007584a8aac000
… the symbols may be overridden by linux-vdso.so.1
, which is a magical library mapped in all programs on Linux to speedup some calls involving system calls.
Anyway, our symbols in the glibc are only trunks to kernel system calls (aka “syscalls”) whether they are in the glibc or inside the vdso ; see the sysdeps/unix/sysv/linux/mmap64.c
default implementations, for example:
void *
__mmap64 (void *addr, size_t len, int prot, int flags, int fd, off64_t offset)
{
// removed arguments check
void *result;
result = (void *)
INLINE_SYSCALL (mmap2, 6, addr,
len, prot, flags, fd,
(off_t) (offset >> page_shift));
return result;
}
Okay, so our initial question is not anymore glibc-related, but involves the Linux kernel.
Now download a recent kernel version and let’s have a quick look at what mremap
involves.
If you take a look at the The Linux Kernel Howto, you see that there is an interesting directory for us:
mm This directory contains all of the memory management code. The architecture specific memory management code lives down in arch/*/mm/, for example arch/i386/mm/fault.c.
Ah, nice, just what we needed to check!
The mm/mremap.c
file seems to be of some interest. And inside the file, you’ll discover the syscall entry point of the mremap function. It’s just here:
/*
* Expand (or shrink) an existing mapping, potentially moving it at the
* same time (controlled by the MREMAP_MAYMOVE flag and available VM space)
*
* MREMAP_FIXED option added 5-Dec-1999 by Benjamin LaHaise
* This option implies MREMAP_MAYMOVE.
*/
SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len,
unsigned long, new_len, unsigned long, flags,
unsigned long, new_addr)
{
This is basically where you enter in the kernel when invoking the corresponding syscall in your user code (through the glibc or through the corresponding vdso thunk).
If you read this function’s code, you first see various arguments checks, and trivial cases being handled (especially shrinking a mapped memory block, which is just freeing the pages on the tail).
The kernel then attempt to expand the mapped area by growing it (as would do an allocator in a C library with sbrk
), and successfully return if expanding the area was possible.
All these trivial cases are O(1) (even if entering in the kernel has a cost - a cost lower now that interrupts aren’t involved anymore, but this is still costly).
What about the worst case ?
/*
* We weren't able to just expand or shrink the area,
* we need to create a new one and move it..
*/
ret = -ENOMEM;
if (flags & MREMAP_MAYMOVE) {
unsigned long map_flags = 0;
if (vma->vm_flags & VM_MAYSHARE)
map_flags |= MAP_SHARED;
new_addr = get_unmapped_area(vma->vm_file, 0, new_len,
vma->vm_pgoff +
((addr - vma->vm_start) >> PAGE_SHIFT),
map_flags);
if (new_addr & ~PAGE_MASK) {
ret = new_addr;
goto out;
}
map_flags = vma->vm_flags;
ret = move_vma(vma, addr, old_len, new_len, new_addr);
if (!(ret & ~PAGE_MASK)) {
track_exec_limit(current->mm, addr, addr + old_len, 0UL);
track_exec_limit(current->mm, new_addr, new_addr + new_len, map_flags);
}
}
At first glance, the kernel is doing what we did in our trivial allocator:
But let’s have a closer look.
The move_vma
call follows:
static unsigned long move_vma(struct vm_area_struct *vma,
unsigned long old_addr, unsigned long old_len,
unsigned long new_len, unsigned long new_addr)
{
...
new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >> PAGE_SHIFT);
new_vma = copy_vma(&vma, new_addr, new_len, new_pgoff,
&need_rmap_locks);
if (!new_vma)
return -ENOMEM;
moved_len = move_page_tables(vma, old_addr, new_vma, new_addr, old_len,
need_rmap_locks);
There are two interesting calls here:
copy_vma
callmove_page_tables
callThe copy_vma
function is inside the sibling mm/mmap.c
file, and basically moves the internal kernel structure related to the mmap’ed block, which is a vm_area_struct
structure.
You can find its definition in include/linux/mm_types.h
; this small structure contains all information on the region: the start and end address, the mmap’ed file on disk if any, etc.
So moving this one will be cheap, O(1).
What about move_page_tables
?
The interesting part seems to lie within this loop:
...
for (; old_addr < old_end; old_addr += extent, new_addr += extent) {
...
move_ptes(vma, old_pmd, old_addr, old_addr + extent,
new_vma, new_pmd, new_addr, need_rmap_locks);
...
The code is a bit complicated (and lacks a bit some comments, too), but is mostly basic arithmetics and accounting.
The move_ptes
function contains itself the deepest inner loop:
...
for (; old_addr < old_end; old_pte++, old_addr += PAGE_SIZE,
new_pte++, new_addr += PAGE_SIZE) {
if (pte_none(*old_pte))
continue;
pte = ptep_get_and_clear(mm, old_addr, old_pte);
pte = move_pte(pte, new_vma->vm_page_prot, old_addr, new_addr);
...
set_pte_at(mm, new_addr, new_pte, pte);
}
...
What the hell are we doing in this loop ? We basically are moving the PTE, the Page Table Entries of the mmap’ed region somewhere else. These are basically long integers assigned to each pages.
We’re pretty done here: the kernel never actually touched a single bit of memory related to our allocated block, and just swapped few bits to relocate the entire region elsewhere.
We may argue that we are still O(N) technically, but
Therefore, we are O(N), but with a huge divisor.
Oh, by the way, did you know that O(N) was in many cases misleading ?
In this case, N can be at most 2^48 (the maximum virtual space size). Most machines only deal with few gigabytes of memory, at most 64GB for most architectures, that is, 2^36 bytes. A huge page is 2^21 bytes, leaving 2^15 operations at most for a mremap
(yes, that’s only 32768 operations).
Well, the worst case scenario is cheap in all cases, and this is what I suspected from the beginning.
Further read: do not hesitate to have a look at Gorman’s book Understanding the Linux Virtual Memory Manager.
TL;DR: do not worry, be lazy, and trust realloc
.
Well, pretty everywhere. As an open-source programmer maintaining a website, I need:
You’re not obliged to secure everything, of course, but most of the stuff above is pretty standard nowadays, and it’s hard not to fulfill all these requirements.
Debian involves thousands of volunteers across the world collaborating to build a complete operating system, and one key aspect is trust among developers. It means that each developer needs to pass certain validation steps to be accepted, and this includes demonstrating your technical skills and your ability to fit the organization spirit. Another requirement is to identify who is responsible for packaging a program (in my case, httrack), as Debian needs to clearly know who uploaded what.
This seems a bit odd to some people: in a virtual world where anonymity is often considered a requirement, and a right - especially in the open-source community - Debian asks people to identify themselves, and identify the work they do.
But the reason behind is trust and accountability: Debian (and all derivative distributions based on its code base) is used by people who would have a hard time using untrusted binaries. What if a backdoor is injected somewhere, for example ? How can we be sure that uploaded packages haven’t been compromised somehow ? Trust is not an option when privacy and security of users is involved (especially in countries where what you may write on twitter can lead you in jail)
What does it mean for a developer like me ?
gpg --gen-key
(well, I already had one)Oh, by the way, I’ll need to update my key - Debian will soon require to have more secure keys by discarding the old 1024 “Diffie” keys. This will involve key transition steps, including meeting folks at Debian again to re-sign my key.
If you do not want your users to have the creepy “The publisher could not be verified, are you sure you want to run this software ?” alert message when attempting to install your program, you need to sign at least the installer.
Microsoft provides a handful utility called signtool.exe
to sign executables to be distributed:
"C:\Program Files\Microsoft SDKs\Windows\v6.0A\Bin\signtool.exe" sign /f "certificate.pfx" /t http://timestamp.verisign.com/scripts/timstamp.dll myprogram.exe
The only problem is that we need an official certificate to do that. Want to have a free PKI like what we have on Debian ? NOPE guys, you need those expensive little pieces of numbers - basically a simple certificate will cost you a thousand dollars. Per year. Yep. Per year.
Fortunately for me, Certum is nice enough to provide free certificates to open-source developers. Thank you, guys!
We only have to create a signing request certificate:
openssl req -new -newkey rsa:2048 -keyout signing.pem -out signing.pem
And copy/paste the certificate request part to the provider.
We then need to convert the produced PEM certificate into a useable PFX one for signtool.exe
:
openssl pkcs12 -export -out certificate.pfx -inkey signing.pem -in certificate.pem
If you have an online store, or are selling something, having a secured version of your website is just an absolute requirement.
But, increasingly, https is becoming the de-facto standard (especially after the PRISM scandal, and especially considering that nothing will change on the government’s side) pretty everywhere. The upcoming HTTP/2.0 protocol may even make https a requirement for all sites.
Having a secured website is unfortunately conditioned to the broken and expensive HTTPS public key infrastructure.
Fortunately enough, the guys at StartSSL also offer free SSL certificates for everyone. These are “low” level certificates, but this is perfectly fine for me (no, I don’t want the golden-platinum certificate). Thank you too, guys!
Things are a bit complicated when actually setting up Apache.
First, each virtual host needs its certificate:
SSLCertificateFile /etc/apache2/ssl/certs/my-server.pem
We need to get the StartSSL certificate chain locally:
wget http://www.startssl.com/certs/sub.class1.server.ca.pem
wget http://www.startssl.com/certs/ca.pem -O startssl-ca.pem
wget --no-check-certificate https://www.startssl.com/certs/ca-bundle.pem -O startssl-ca-bundle.pem
Then, spend few hours trying to figure out how to mitigate the various known attacks against SSL. Darn. Sheesh. Please give me some feedback if you thing the setup below can be improved :)
# Enable SSL
SSLEngine on
# Disable compression (CRIME attack)
# "Available in httpd 2.2.24 and later"
# SSLCompression off
# Enable only secure protocols: TLSv1, but not SSLv2/SSLv3
SSLProtocol all -SSLv2 -SSLv3
# Perfect forward secrecy + BEAST mitigation
# See <https://community.qualys.com/blogs/securitylabs/2013/08/05/configuring-apache-nginx-and-openssl-for-forward-secrecy>
SSLCipherSuite EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!ADH:!SSLv2:+HIGH:-MEDIUM
# Pay attention to the order ciphers are specified
SSLHonorCipherOrder On
SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown
# Certificate global settings
SSLVerifyClient none
SSLCACertificateFile /etc/apache2/ssl/certs/startssl-ca.pem
SSLCertificateChainFile /etc/apache2/ssl/certs/sub.class1.server.ca.pem
You can check whether your setup is fine, using openssl as commandline tool:
openssl s_client -CApath /etc/ssl/cert -showcerts -connect www.example.com:443
And you can also use online tools to control used ciphers, options… if you manage to have 100% to all tests, congratulations!
Yes, you also need to pay a certificate for that. The SSL PKI is definitely broken and mostly a scam. Haha.
Fortunately enough, we may use a free certificate for that, too :)
Note that securing the SMTP transactions is not considered sufficient when exchanging sensitive information - use GPG, guys!
For sendmail, I ended up with something like this:
define(`CERT_DIR', `/etc/mail/certs')
define(`confCACERT_PATH', `CERT_DIR')
define(`confCACERT', `CERT_DIR/CAcert.pem')
define(`confSERVER_CERT', `CERT_DIR/MYcert.pem')
define(`confSERVER_KEY', `CERT_DIR/MYkey.pem')
define(`confCLIENT_CERT', `CERT_DIR/MYcert.pem')
define(`confCLIENT_KEY', `CERT_DIR/MYkey.pem')
TRUST_AUTH_MECH(`LOGIN PLAIN CRAM-MD5 DIGEST-MD5')
dnl Do not choke on missing/invalid client cert
define(`confTLS_SRV_OPTIONS', `V')
LOCAL_CONFIG
O CipherList=ALL
It took me a while to understand that the missing define('confTLS_SRV_OPTIONS', 'V')
line was the culprit for rejecting some mails…
One last check to ensure your server is correctly setup:
openssl s_client -CApath /etc/ssl/cert -showcerts -connect smtp.example.com:25 -starttls smtp
At last, securing a DNS server becomes increasingly important, too, especially when governments attempt to hijack it. DNSSEC is not yet mainstream unfortunately, and operating systems can not enforce its use yet (which means that evil dictatorships can still intercept public DNS and corrupt the replies).
DNSSEC seems a bit cumbersome to implement at first glance, but at least the system rely on existing DNS infrastructure, and will not cost you additional fees. You’ll find numerous useful FAQ and blogs explaining how to deploy this thing.
You first need to adjust a bit your own DNS server settings to handle DNSSEC. With BIND 9, this can be done through built-in features:
dnssec-enable yes;
dnssec-validation auto;
dnssec-lookaside auto;
managed-keys-directory "/var/cache/bind";
Then, you need a Key Signink Key (aka. “KSK”) for each domain you own. The key, to be renewed every two years or so, is meant to sign other keys. The idea behind is that this key is to be declared to the upstream registrar, through your own domain administration system, and the registrar will sign it, to certify that it is legit. This allows to build a trust chain up to the root DNSSEC authority, whose certificate is known by DNS resolvers. With this key, you’ll be able to create a shorter-lifespan key, the Zone Signing Keys (aka. “ZSK”), aimed to sign the various entries in your DNS zone. This shorter-lifespan key will be put in your DNS zone among other entries, as a DNSKEY entry (the record contains the public key side).
Here comes the KSK:
dnssec-keygen -r /dev/urandom -a RSASHA256 -b 4096 -n ZONE -f KSK httrack.com
And, at last, you need to create regularly (every two or three months) a new ZSK:
dnssec-keygen -K dnssec/zsk -r /dev/urandom -a RSASHA256 -b 2048 -n ZONE httrack.com
And do not forget to sign your zone each time you need to modify it, using the ZSK key:
dnssec-signzone -K dnssec/zsk -e +3024000 httrack.com
service bind9 reload
The .signed versions of the DNS zones will have to be used in place of the previous unsigned ones (named.conf
), of course.
The whole setup is not totally straightforward, and will probably cause headache to newcomers. Using online tools to verify the DNS signed records or to check the DNSSEC setup will be extremely helpful.
TL;DR: Security is sometimes painful, but this must not stop you. Your users are trusting you!
]]>There are countless books in computer science (and related domains), but it is sometimes hard to pick the right one, the one which provides an easy access to a domain, but at the same time helps you to embrace it.
Algorithms is a typical case, but not the only one: a good professor will not teach you more advanced algorithms, or more efficient ones. A good professor will make complicated things less complicated. An excellent one will turn them into simple things. A bad one will not only turn things even more complicated, but will also hinder any further reflexion, because he’ll convince you that this is not something for you (ie. you are probably too limited for that).
Books are more or less like professors: there are countless of badly written ones, but at the end very few excellent ones.
The motivation behind reading books can vary:
Let’s review what I enjoyed reading in the past few years.
Yes, this list is personal, non-exhaustive, with probably some questionable choices. Some people may want to complete it, or will disagree with several ones, but hey, I think it’s a good start anyway.
The C programming language by Brian Kernighan and Dennis Ritchie. Probably the most controversial one, but this book was the first C-language-related book I read, at a time Internet was not mainstream, and it was a great pleasure to switch from a gfa-basic/assembly developer to a C developer in only few weeks. I would probably not recommend the book nowadays, but I am really grateful for the (possibly imperfect) first C “professor” I had, and this book is one of the main reasons why I am writing C today.
Applied Cryptography by Bruce Schneier was also my first cryptography-related book and even if I’m still (very) far from being an expert in this field, it really helped me a lot understanding many of the concepts behind it. I would really recommend this book to anyone interested by cryptography (and I would say that everybody should be interested by cryptography, and not only programmers)
Algorithms by Robert Sedgewick and Kevin Wayne is without hesitation the best algorithm book I have ever read, and I would gladly recommend it to anyone - and by anyone I really mean anyone: beginners, students, confirmed developers… I was first a bit disappointed by the rather basic level of the first chapter (quickly reviewing fundamental concepts, including for loops, etc.), but this is a key point of this book: the progression is uniform, and before you even realize it, more complex things appear. I have been studying algorithms during my college years, and I must admit that some of them totally flew over my head ; including Dijkstra’s algorithm, something I painfully implemented the first time when I was still a student (and I did not really understand the underlying concepts at that time, I must admit - this was bruteforce coding for me). In Algorithms, Dijkstra is not something which appears from the void, but is within a logical difficulty progression - and this book taught me how simple Dijkstra’s algorithm was. Yes, really simple: a direct graph search through a priority queue, with edge relaxation - after reading few chapters, I guarantee that this is a really simple thing do to. More generally, very intimidating algorithms many developers are afraid to check out are explained in such way that not only it makes your fears disappear, but gives you the trust to explore further domains (did you know that red-black-trees were not as crazy as they seemed ?). Ah, and a last note: algorithms in this books are written in Java, but are easily adaptable to other languages (in my case, C and C++)
Hacker’s Delight by Henry S. Warren is a delicious book for bit-fiddling fans and more generally hackers (and by hackers I do not mean script kiddies trying to deface the NSA website, but hackers). Have you ever wondered how to count the number of leading ones in an integer without spending your time in a loop ? Or how to divide an integer number without actually using the division operation (slow) but only the multiply one ? This book is not only a collection of good recipes, it also explains the maths behind them. I strongly recommend this book to all advanced programmers, but also to anyone wanting to dive inside the numbers behind our computers.
Programming Pearls by Jon Bentley is a really enjoyable book on algorithms - and beyond algorithms - with personal and very insightful war stories, entertaining remarks, and more. I really recommend this little gem, which will teach you how simple and powerful a markov chain can be, for example. Oh, and Jon Bentley is also the author of the excellent long common strings data compression (which can be viewed as a “low frequency” compression algorithm, compared to classical “high frequency” algorithms such as zlib, which basically means that it can be piped to a secondary compression, with a combined ratio)
IPv6 Théorie et Pratique by G6 association is a French (yes, sorry guys) book which taught me everything on IPv6 at a time IPv6 was really not mainstream (well, it is still not mainstream today, cough cough cough…). It is very unfortunate O’Reilly France disappeared, but at least this book is still online, and I would recommend it too.
Oh yes, some videos can also be extremely helpful.
For a very good and non-experts oriented introduction to cryptography, I would strongly recommend the Gambling with Secrets series, and the RSA Encryption one. These small entertaining videos are really great (even for non-developers)
An excellent algorithm course by Robert Sedgewick and Kevin Wayne which is a good complement to the Algorithms book. Robert Sedgewick is an excellent professor, and the courses videos are intended for anyone (including beginners). Do not hesitate to enroll to this course, one more time: this is really worth it.
TL;DR: No, really, sometimes reading is the shortest path!
]]>As a programmer - especially as a C/C++ programmer - you might be reluctant to embrace the Unicode beast. You vaguely know that wchar.h
contains various functions to handle the dirty things, but it does not make things clearer. In Java, programmers are using String
without troubles, until they want to convert them from/to arrays of bytes (hey, how Java is supposed to handle supplemental characters ?)
Unicode is very simple in its essence, even if you can make things unreasonably complex (have you ever decomposed an Unicode character ?): this is basically the extension of the ASCII concept (assigning a number to a character) to fit more characters.
Yes, Unicode is rather simple if you keep things simple.
Let’s begin with some concepts:
Unicode: a consortium aimed to standardize all possible characters in use on Earth (even the most improbable languages - but this does not include Klingon), such as latin characters, Chinese, Arabic, etc., and assign them a number. Numbers from 0 to 127 are the standard 7-bit ASCII, and code points above are non-ascii characters, obviously (the first range assigned above being latin letters with accents)
UTF-8: a way to encode an Unicode character code into a sequence of bytes ; 7-bit ASCII is encoded “as is”, using 7-bit ASCII (ie. the code point for the letter “A”, which is the letter 65 in the Unicode world, is encoded into the ASCII byte “A”, which is also 65 in the ASCII world), and the rest is using a sequence of bytes in the range (in hex) 80
..FD
(FE
and FF
are never used, which is pretty cool, because this greatly helps making the difference between an UTF-8 stream and other encodings)
UCS-2: encoding of Unicode points as 16-bit wide characters - basically, an array of uint16_t
- note that you may only encode Unicode points from 0 to 65535 (something which is called the “Basic Multilingual Plane”, or BMP). This is generally used internally only - most people won’t store or send raw arrays of uint16_t
. You may have two flavors: UCS-2BE (big-endian flavor), or UCS-2LE (little-endian flavor), and a common trick to detect the endianness is to put a zero-width no-break space (sometimes called byte order mark) as first invisible character in a text stream, whose code point is FEFF
. The “reverse” code point FFFE
is invalid in Unicode, and should never appear anywhere. Oh, did I tell you that FE
and FF
never appear in an UTF-8 stream previously ? If you use the byte order mark, you’ll never confuse an UCS stream with an UTF-8 one.
UCS-4: encoding of Unicode points as 32-bit wide characters - basically, an array of uint32_t
. You also have two flavors: UCS-4BE (big-endian flavor), or UCS-4LE (little-endian flavor), and the byte-order-mark trick is also used in many places.
UTF-16: encoding of Unicode points as 16-bit wide characters (like UCS-2), except that Unicode points beyond the Basic Multilingual Plane are encoded as two 16-bit characters in the surrogate range. This is a bit wicked: we are encoding a character using two characters, and each of them are encoded as 16-bit characters. You can treat an UTF-16 stream as a regular UCS-2 stream - in this case, code points beyond the Basic Multilingual Plane will be seen as a pair of unknown-class characters (surrogates).
UTF-32: this is actually the synonym for UCS-4 (and is in my opinion an confusing naming: this is not a variable length encoding). Oh, and you are not supposed to have surrogates (used for UTF-16, see above) inside an UCS-4/UTF-32 stream.
Let’s have a look as UTF-8, which is generally the preferred encoding for storing and sending data (most RFC are using UTF-8 as default encoding nowadays).
The good news is that 7-bit ASCII is a subset of UTF-8: nothing will change on this side.
For the rest, UTF-8 is a very simple and smart encoding (Ken Thompson is behind this fine piece of work), and decoding it is not really hard.
If a character outside the 7-bit range is used, UTF-8 will encode it as a byte sequence. For example, the “é” letter (latin letter e with acute accent), which is in the Unicode world the point E9
(in hexadecimal - that is, 233 in decimal), will be encoded as bytes C3
and A9
(C3 A9
) (which is displayed as “é” - something you’ll learn to recognize and which generally is a sign of UTF-8 not being recognized by a client). The “姫” Chinese character (“princess”), which is in the Unicode world the point 59EB
(23019 in decimal), will be encoded as E5 A7 AB
(displayed as “姫” if your display client or browser is having troubles recognizing it).
To understand UTF-8, you just need to know one thing: for a given byte, visualize its 8 bits, and count the number of leading “1” (ie. the number of “1” before the first “0”). This is the only information you need to know whether this byte is a single 7-bit ASCII character, the beginning of an UTF-8 sequence, or an intermediate byte within an UTF-8 sequence. Oh, and if the byte is the beginning of a sequence, you also know how long it is: its length is precisely the number of leading “1”. Isn’t that magic ?
Yep, this is it.
Basically,
0
..7F
)This is an extremely good property of UTF-8: you can easily cut an UTF-8 string/skip characters/count characters without bothering about actual code points. Only bother about leading “1”’s.
Oh, by the way: did you know that the number of leading “1” could be computed without using a loop ? Take a look at the Hacker’s Delight recipes (do not hesitate to buy the book, it is worth it, and recipes are presented with their mathematical explanation - there is no pure magic out there)
Here’s the trick: (this is the number of leading zeros, but you probably have an idea on how to use this function to compute the number of leading ones)
/* Hacker's delight number of leading zeros. */
static unsigned int nlz8(unsigned char x) {
unsigned int b = 0;
if (x & 0xf0) {
x >>= 4;
} else {
b += 4;
}
if (x & 0x0c) {
x >>= 2;
} else {
b += 2;
}
if (! (x & 0x02) ) {
b++;
}
return b;
}
We know how to recognize UTF-8 sequences, and we even know how to get their length. How to encode Unicode points ?
Encoding is extremely simple. It could not be simpler, actually: visualize your Unicode point as a stream of bits (starting from the leading “1”). You’ll encode them in the remaining space, in the first leading character, and in the remaining sequence. The remaining space within sequence bytes is the space after the leading “1” and the “0” delimiter.
For example, if the leading byte defines a 4-byte sequence, then it will start with four “1”, and a “0” delimiter. Three bits remains available for encoding (the first three bits of the Unicode point number, starting from the leading “1”, will be encoded there). Continuing characters (in sequence) have a leading “1” followed by the “0” delimiter: six bits are available for encoding.
Let’s go back to the “姫” Chinese character. This character is 59EB
in the Unicode charts ; here’s how it is encoded as E5 A7 AB
(and sorry for my lame Chinese writing skills!):
As an exercise, you can now easily rewrite the getwchar()
function in an UTF-8 environment (locale).
Here’s my try (do not cheat! look at this solution after you tried!):
#include <stdio.h>
#include <stdlib.h>
#define UTF8_ERROR ( (int) (-2) )
/* Hacker's delight number of leading zeros. */
static unsigned int nlz8(unsigned char x) {
unsigned int b = 0;
if (x & 0xf0) {
x >>= 4;
} else {
b += 4;
}
if (x & 0x0c) {
x >>= 2;
} else {
b += 2;
}
if (! (x & 0x02) ) {
b++;
}
return b;
}
/* Length of an UTF-8 sequence. */
static size_t utf8_length(const char lead) {
const unsigned char f = (unsigned char) lead;
return nlz8(~f);
}
/* Equivalent to getwchar() on an UTF-8 locale. */
static int utf8_getchar() {
const int c = getchar();
const size_t len = utf8_length(c);
if (c < 0) { /* EOF */
return EOF;
} else if (len == 1) { /* ASCII */
return UTF8_ERROR;
} else if (len == 0) { /* Error (in-sequence) */
return c;
} else { /* UTF-8 */
unsigned int uc = c & ( (1 << (7 - len)) - 1 );
size_t i;
for(i = 0 ; i + 1 < len ; i++) {
const int c = getchar();
/* not EOF, and second bit shall always be cleared */
if (c != -1 && ( c >> 6 ) == 0x2) {
uc <<= 6;
uc |= (c & 0x3f);
} else if (c == -1) {
return EOF;
} else {
return UTF8_ERROR;
}
}
return (int) uc;
}
}
int main(int argc, char* argv[]) {
for(;;) {
const int c = utf8_getchar();
if (c == EOF) {
break;
} else if (c == UTF8_ERROR) {
fprintf(stderr, "* UTF-8 error\n");
} else {
printf("unicode character 0x%04x\n", c);
}
}
return EXIT_SUCCESS;
}
Note that the above code can be improved a little bit, by unrolling the for()
loop, and using few switch
.
TL;DR: Do not worry, and embrace UTF-8 and the Unicode world.
]]>Your GCC build flags. Yes, this is actually an interesting question! I have been building code at the various places where I work for many years, on different architectures, and tweaking the build flags has always been an important task.
I mean, you probably are too experimented to just use something like:
gcc -c fubar.c -o fubar.o
And you probably use -W -Wall
, or additional flags, to tune the compiler behavior.
The idea behind is not only to carefully optimize produced bytecode, but also to improve its quality, and its security. Compiler warnings are critical to spot many programming errors or lethal typos that would otherwise consume days of debugging (and a big pile of money!).
Many beginners are still using nowadays the default gcc
command without further tuning, and this always makes me cringe. Warning messages are not annoying, they are useful. And the minutes you spend fixing them (Note that I did not say hide, but fix) will spare you days or even months of nightmares.
Here are the flags we are using where I work:
gcc -pipe -m64 -ansi -fPIC -g -O3 -fno-exceptions -fstack-protector -Wl,-z,relro -Wl,-z,now -fvisibility=hidden -W -Wall -Wno-unused-parameter -Wno-unused-function -Wno-unused-label -Wpointer-arith -Wformat -Wreturn-type -Wsign-compare -Wmultichar -Wformat-nonliteral -Winit-self -Wuninitialized -Wno-deprecated -Wformat-security -Werror -c source.c -o dest.o
Within these fancy flags, I was among the craziest tuner. Here’s the summary of what I shamelessly added:
-pipe
I prefer NOT to use temporary files when possible, and use pipes, for example when compiling preprocessed code into intermediate assembly source (.S). This has no real impact on performances (temporary files are generally put on some kind of tmpfs filesystem), but this is a bit cleaner.
-fno-exceptions
I don’t like C++ exceptions. And we don’t use them where I work. So let’s remove the overhead generated by unwinding code (ie. code and data aimed to allow the runtime to “rollback” function calls in case an exception is thrown, eg. object destructors that need to be called, etc.). Bonus: it may also produce a bit faster code. What if an exception is thrown (eg. by the runtime or the STL) by the way ? Well, abort()
will be called instead, which is perfectly fine as thrown exceptions are either programming errors (out_of_range, bad_cast etc.) or critical conditions (new
throwing bad_alloc)
-fstack-protector -Wl,-z,relro -Wl,-z,now -Wformat-security
These flags are actually ripped from the Debian Hardening Wiki, and are aimed to detect stack smashing issues, have a read-only global offset table (preventing any attacks involving writing through the GOT), and various format strings suspicious usages.
-fvisibility=hidden
Ah yes, this one is cool. It changes the default extern symbol export mode in GCC to “hidden”. Basically, it means that “extern” symbols are visible by all units within a library, but not outside this library. If you ever wrote code on Windows environments, you probably remember those: _declspec(dllexport)
and _declspec(dllimport)
. Without them, symbols inside your DLL would not be exported (or imported). On POSIX systems, you export all extern
symbols, and they become visible in your .so library. But this default mode has several drawbacks
The last argument was actually the strongest one: the build would be broken several times per week because someone forgot to properly export the symbols and the code was building fine on Linux but not on Windows. For all these reasons, using -fvisibility=hidden
(and properly exporting symbols using __attribute__ ((visibility("default")))
) was a true relief!
-Wpointer-arith
We do not want to know the size of void
, and we won’t use it!
-Wformat-nonliteral
Banishing “non literal format strings” was actually a sane decision. Especially when the format string source was some kind of user generated input - we’re not in 1990 anymore, and security issues involving string format can not be ignored anymore.
-Winit-self
This is actually a joke in the C standard.
int i = i;
Yes, this code is perfectly valid, and won’t even produce a single warning by default. And, of course, the behavior is unspecified – the variable i
is left “initialized” with a garbage value. I have been bitten by this idiotic default behavior too often before turning on this specific warning. Shame on me.
-Werror
Haha. This one was painful to propagate through all of our code. It basically breaks the build once you have a warning somewhere. Yes, the code won’t build unless it is totally warning-free. Yes, people were a bit upset at me at the beginning :), and they were even more upset when committing code that would break on a new compiler version (but not on an older one). This was painful. This was hard. But I did not give up! And at the end, the overall impact was tremendously positive.
grep
‘ping thousands of lines of logs is not a cool thing to do, and nobody is going to do that)-Wno-*
flags) if needed!Oh, and on the linker side, I did a bit of tweaking too:
-Wl,-O1
Did you know that you also have an optimization flag for the linker ? Now you know!
-Wl,--discard-all
Discard all local symbols, for the sake of library size.
-Wl,--no-undefined
Yes, I don’t like missing (unresolved) symbols at link time, even if this is actually a feature. It makes the Windows build easier too, by preventing behavioral differences between Linux/Windows.
-Wl,--build-id=sha1
This adds a fancy “build identifier” in all produced modules, which has a deterministic value (ie. two builds with the same sources and build flags will produce the same identifier), allowing to control whether or not you managed to rebuild from scratch and produce the same binaries.
-rdynamic
(note: this actually passes the -export-dynamic
flag to ld
)
This linker option allows you to have more exported symbols within the dynamic symbol table, which is actually a nice thing when attempting to have readable backtraces!
That’s all, folks.
TL;DR: Take the time to read carefully and understand the man gcc
and man ld
pages. It is worth it.
Thanks to Andrew Hochhaus for changing -fvisibility=internal to -fvisibility=hidden. Thanks for the insightful remarks to all hacker news contributors, including additional flags and their advantages.
]]>Yes. Many people like me are developing on multiple platforms, including Windows, but do not necessarily work on Windows for their primary platform. Besides, you may log-in on a random Windows machine with a very minimalistic setup (say, a new virtual machine, for example), and do not want to have a full compiler installed. You may also use multiple versions of a compiler (playing with Visual C++ 2008, 2010, 2012, 2013..), and switching between them may become a real burden.
For these many reasons, having a standalone compiler, say, ready-to-use on a network drive, is often a very handy solution.
Technically, using a full Visual C++ install deported on an external location, without actually installing anything, is not really supported (I would say rather untested, though)
Fortunately enough, Visual C++ is nice and well-educated enough to accept being used in a standalone fashion ; at least for the commandline tools, such as cl.exe or link.exe. Yay!
The recipe is almost the same for all Visual C++ releases (tested from 2005 to 2013): blindly copying important directories in a safe place, adjusting some DLL’s. I will describe the rough necessary steps for 2013 and 2010 - you’ll need to adapt a bit to fit your needs, sometimes find a bit in subfolders to catch some DLL’s, but this should not be too cumbersome.
Start on a fresh machine. Preferably on a virtual machine, up-to-date (service packs). The machine can be trashed after creating the remote compiler, of course. Having a fresh install is quite handy to spot what has been installed, and to avoid being polluted by external DLL’s (such as external redistributables)
Install a regular Visual C++ Express version. Yes, I’m using the Express flavor, as it is simpler to use, and has probably less licensing issues than the “paid” release. You will miss some components, though, such as ATL and MFC (but who use them nowadays ? Err, I may have still some MFC code lying around by the way …)
Do not forget to install every possible updates, patches, service packs…
Source | Destination |
---|---|
vssrc\VC | VC |
vssrc\Common7\IDE 10.0\Common7\IDE\{mspdb*.dll} | VC\bin\ |
vssrc\VC\redist\x86\Microsoft.VC120.CRT\*.dll | VC\bin\ |
vssrc\VC\redist\x86\Microsoft.VC120.CRT\*.dll | VC\bin\amd64 |
vssrc\VC\bin\{mspdb*.*} | VC\bin\amd64 |
vssrc\VC\bin\msobj120.dll | VC\bin\amd64 |
programfiles\Microsoft SDKs\Windows\v7.1A\Bin | bin |
programfiles\Microsoft SDKs\Windows\v7.1A\Include | include |
programfiles\Microsoft SDKs\Windows\v7.1A\Lib | lib |
programfiles\MSBuild\12.0\Bin | NET\Framework |
programfiles\MSBuild\12.0\Bin\amd64 | NET\Framework64 |
Source | Destination |
---|---|
vssrc\VC | VC |
vssrc\Common7\IDE\msobj100.dll | VC\bin\ |
vssrc\Common7\IDE\mspdb100.dll | VC\bin\ |
vssrc\Common7\IDE\mspdbcore.dll | VC\bin\ |
vssrc\Common7\IDE\mspdbsrv.exe | VC\bin\ |
mssdk\v7.1\Redist\VC..\msvcr100.dll | VC\bin\ |
mssdk\v7.1\Redist\VC..\x64\msvcr100.dll | VC\bin\amd64 |
programfiles\Microsoft SDKs\Windows\v7.1\Bin | bin |
programfiles\Microsoft SDKs\Windows\v7.1\Include | include |
programfiles\Microsoft SDKs\Windows\v7.1\Lib | lib |
C:\Windows\Microsoft.NET\Framework | NET\Framework |
C:\Windows\Microsoft.NET\Framework64 | NET\Framework64 |
In all cases, you may also want to have Process Explorer and Dependency Walker within the same place, as they are really useful tools for a developer.
Watch out: 64-bit binaries may be placed in different subfolders:
Yes, Microsoft had some troubles using a single naming scheme for 64-bit :)
After that, you only need to set environment variables to have the 64-bit or 32-bit directories in your PATH in some kind of cmd script ; eg:
@SET VSINSTALLDIR=X:\data\my-standalone-compiler
@SET VCINSTALLDIR=%VSINSTALLDIR%\VC
@SET WindowsSdkDir=%VSINSTALLDIR%\
@set PATH=%VCINSTALLDIR%\BIN\x86_amd64;%VCINSTALLDIR%\BIN;%VSINSTALLDIR%\bin\x64;%PATH%
@set LIB=%VCINSTALLDIR%\LIB\amd64;%VSINSTALLDIR%\LIB\x64
@set INCLUDE=%VCINSTALLDIR%\INCLUDE;%VSINSTALLDIR%\INCLUDE
@set LIBPATH=%VCINSTALLDIR%\LIB\amd64;%VSINSTALLDIR%\LIB\x64
You may deploy Visual C++ redistributables (32-bit or 64-bit) depending on your needs.
After that, you’re done - enjoying the simplicity of having a ready-to-use build setup anywhere.
A last note: I’m not fan of .NET, but apparently some tools will cause you some headache if you are using this solution. You have been warned! :)
TL;DR: Who needs to install a compiler on every machine ?
]]>I am writing to follow up on an email regarding a possible sponsorship on HTTrack.com. …
I would like to offer you a partnership agreement to monetize your application HTTrack …
We are a software monetization platform and I’m contacting you to discuss possible partnership …
Sheesh.
The first time I got one of these email, I really didn’t know what it was all about. At first glance, I thought they wanted to offer some kind of ad-platform for httrack.com, or maybe some kind of sponsored links.
I generally always decline these offers, as I do not plan to put more ads on the main site (the only ad, placed on the download page, is to support bandwidth fees, as I’m hosting the binaries directly). Oh, and also because many of these ad providers are really terrible - do you really want to have ads for counterfeit clothes, fake enhancement pills, or pyramid-scheme business ? Me neither.
After several emails, I decided to ask for a bit more information on these “partnerships” and “opportunities” to understand what they really wanted.
It appears that what they were interesting in was to put a “download bar” (or a “software bundle”, or a “download optimizer”) in the httrack installer executable, and share benefits with me (well, at least this is what they said).
The only purpose of the search bar was to “enhance browser experience” and things like that (yes, really).
.. said nobody. Ever.
Do you have a search bar installed on your PC ? I mean, you have a browser, and you decide to install an additional bar to… search for things. Like with the regular bar.
No, I suppose. Me, neither. Nobody does. Even your grandma.
So, what’s the point ?
Great, isn’t it ?
Adware, spyware, crapware. Yes. And this was my first reaction, and then I politely declined.
But you can opt-out your users during install!
Oh great. So each time a user install the program and does not carefully look at the checkbox enabled by default, he’s going to install the bar. This is the kind of thing which infuriates me.
But you can opt-in your users during install, too!
I was still skeptical, but why not, after all ? People installing the program by clicking “next-next” will not be harmed. And after all, this is just a search bar, isn’t it ?
Well, no, this is wrong. So wrong.
First, if you take look at the reputation of all these companies, you’ll be horrified. Endless list of users trying to remove the infamous search bar, trying to figure out why they caught this virus, why they have pornographic pop-ups everywhere, etc.
Second, nobody will install this component opt-in.
More precisely:
What’s the point in allowing opt-in for these companies, by the way, is it does not generate any real revenue ?
The bleak truth is that in this case they are not providing you a marginal revenue. They are using your reputation to cover their bad one.
And this is the last reason why you should never, ever, make a deal with these people: as a developer, you have a reputation. You built something great (with few bugs, possibly) and you have a large user base trusting you. Do not betray them.
Unfortunately, some über-jerks found an even better solution: bundle your application without telling you, and put it on their smelly download site.
The consequences are extremely annoying
That is why I recently decided to start signing the Windows installer version, with an official trusted SSL key (registered to Xavier Roche). I will probably do the same for the program DLL’s soon. By the way, on Debian platforms, I have been signing the sources from the beginning, and this is a very good practice.
TL;DR: Search bars are crap, and you should always download programs on trusted sites.
]]>is my file properly sync’ed on disk ?
You probably know more or less how databases (or things that look like one) store their data on disk in a permanent and safe way. Or at least, you know the basic principles. Or not ?
There are a bunch of concepts that first must be understood: what is atomicity, consistency, and durability ? These concepts apply on databases (see ACID), but also on the underlying filesystem.
Atomicity: a write operation is fully executed at once, and is not interleaved with another one (if, for example, someone else is writing to the same location)
Atomicity is typically guaranteed in operations involving filename handling ; for example, for rename, “specification requires that the action of the function be atomic” - that is, when renaming a file from the old name to the new one, at no circumstances should you ever see the two files at the same time.
Consistency: integrity of data must be maintained when executing an operation, even in a crash event - for example, a power outage in the middle of a rename()
operation shall not leave the filesystem in a “weird” state, with the filename being unreachable because its metadata has been corrupted. (ie. either the operation is lost, or the operation is committed.)
Consistency is guaranteed on the filesystem level ; but you also need to have the same guarantee if you build a database on disk, for example by serializing/locking certain operations on a working area, and committing the transaction by changing some kind of generation number.
Durability: the write operation is durable, that is, unplugging the power cord (or a kernel panic, a crash…) shall not lose any data (hitting the hard disk with a hammer is however not covered!)
This is an important one - at a given point, you must ensure that the data is actually written on disk physically, preventing any loss of data in case of a sudden power outage, for example. This is absolutely critical when dealing with a client/server architecture: the client may have its connection or transaction aborted at any time without troubles (ie. the transaction will be retried later), but once the server acknowledges it, no event should ever cause it to be lost (think of responsibility in a commercial transaction, or a digital signature, for example). For this reason, having the data committed in the internal system or hard disk cache is NOT durable for obvious reasons (unless there is a guarantee that no such power outage could happen - if a battery is used on a RAID array, for example).
On POSIX systems, durability is achieved through sync operations (fsync()
, fdatasync()
, aio_fsync()
): “The fsync() function is intended to force a physical write of data from the buffer cache, and to assure that after a system crash or other failure that all data up to the time of the fsync() call is recorded on the disk.”. [Note: The difference between fsync() and fdatasync() is that the later does not necessarily update the meta-data associated with a file - such as the “last modified” date - but only the file data.]
Now that these concepts are a bit clearer, let’s go back to our filesystem!
If we want to simplify the concept, let’s consider the filesystem on POSIX platforms as a very simple flat storage manager, allowing to read/write data blobs and basic properties (such as the modified time) indexed by an integer number (hint: they sometimes call that the inode number).
For example, you may want to read the file #4242’s data. And later, write some data on file #1234.
To have a more convenient way to handle files (because “I need to send you the presentation number 155324” would not be really convenient in the real world), we use the filename/directory concepts. A file has a name, and it is contained within a directory structure. You may put files and directories in a directory, building a hierarchical structure. But everything rely on our previous basic data blobs to store both filename and the associated index.
As an example, reading the file foo/bar.txt
(ie. the file bar.txt
within the foo
directory) will require to access the data blob associated with the directory foo
. After parsing this opaque data blob, the system will fetch the entry for bar.txt
, and open the associated data blob. (And yes, there is obviously a root entry, storing references to first-level entries, allowing to access any file top-down)
If I now want to create a new file named foo/baz.txt
, it will require the system to access the data blob associated with the directory foo
, add an entry named baz.txt
with a new allocated index for the upcoming file, and write the updated directory blob back, and from this point, write to the newly allocated blob. The operation therefore involves two data structures: the directory entry, and the file itself.
Let’s go back to our database problem: what is the impact of having two data structures for our files ?
Atomicity and consistency of filenames are handled for us by the filesystem, so this is not really a bother.
What about durability ?
We know that fsync()
provides guarantees related to data and meta-data sync’ing. But if you look closer to the specification, the only data involved are the one related to the file itself - not its directory entry. The “metadata” concept involves modified time, access time etc. - not the directory entry itself.
It would be cumbersome for a filesystem to provide this guarantee, by the way: on POSIX systems, you can have an arbitrary number of directory links to a filename (or to another directory entry). The most common case is one, of course. But you may delete a file being used (the file entry will be removed by the system when the file is closed) - the very reason why erasing a log file which is flooding a filesystem is a futile and deadly action - in such case, the number of links will be zero. And you may also create as many hard-links as you want for a given file/directory entry.
Therefore, in theory, you may create a file, write some data, synchronize it, close the file, and see your precious file lost forever because of a power outage. Oh, the filesystem must guarantee consistency, of course, but not durability unless explicitly asked by the client - which means that a filesystem check may find your directory entry partially written, and decide to achieve consistency by taking the previous directory blob entry, wiping the unreferenced file entry (note: if you are “lucky” enough, the file will be expelled in lost+found
)
The filesystem can, of course, decide to be gentle, and commit all filename operations when fsync’ing. It may also, such as for ext3, commit everything when fsync’ing a file - causing the infamous and horrendous lags in firefox or thunderbird.
But if you need to have guarantees, and not just hope the filesystem “will be gentle”, and do not want to “trust the filesystem” (yes, someone actually told me that: you need to “trust the filesystem” - I swear it), you have to actually make sure that your filename entry is properly sync’ed on disk following the POSIX specification.
Oh, and by the way: according to POSIX, The fsync() function is intended to force a physical write of data from the buffer cache, and to assure that after a system crash or other failure that all data up to the time of the fsync() call is recorded on the disk.
But things are sometimes a bit obscure on the implementation side :
Linux/ext3: If the underlying hard disk has write caching enabled, then the data may not really be on permanent storage when fsync() / fdatasync() return. (do’h!)
Linux/ext4: The fsync() implementations in older kernels and lesser used filesystems does not know how to flush disk caches. (do’h!) - issue adressed quite recently
OSX: For applications that require tighter guarantees about the integrity of their data, Mac OS X provides the F_FULLFSYNC fcntl. The F_FULLFSYNC
fcntl asks the drive to flush all buffered data to permanent storage (hey, fsync was supposed to do that, no ? guys ?) (Edit: no, fsync is actually not required to do that - thanks for the clarification Florent!)
But we may assume that on Linux with ext4 (and OSX with proper flags ?) the system is properly propagating write barriers.
On Windows, using FlushFileBuffers()
is probably the way to go.
I told you that a filesystem was actually a bunch of flat data blobs with associated metadata, and that a file had actually two parts: its directory entry (let’s assume there is only one directory entry for the sake of simplicity), and its actual data. We already know how to sync the later one ; do we have a way to do the same for the directory container itself ?
On POSIX, you may actually open a directory as if you were opening a file (hint: a directory is a file that contains directory entries). It means that open()
may successfully open a directory entry. But on the other hand, you generally can not open a directory entry for writing (see POSIX remark regarding EISDIR
: The named file is a directory and oflag includes O_WRONLY or O_RDWR), and this is perfectly logical: by directly writing to the internal directory entry, you may be able to mess up with the directory structure, ruining the filesystem consistency.
But can we fsync() written data using a file descriptor opened only for reading ? The question is… yes, or at least “yes it should” - even POSIX group had editorial inconsistencies regarding fdatasync and aio_fsync(), leading to incorrect behavior on various implementations. And the reason it should execute the operation is because requesting the completion of a write operation does not have to require actual write access - which have already been checked and enforced.
On Windows… err, there is no clear answer. You can not call FlushFileBuffers()
on a directory handle as far as I can see.
Oh, a last funny note: how do you sync the content of a symbolic link (and its related meta-data), that is, the filename pointed by this link ? The answer is… you can’t. Nope. This is not possible with the current standard (hint: you can not open()
a symbolic link). Which means that if you handle some kind of database generation update based on symbolic links (ie. changing a “last-version” symlink to the latest built generation file), you have zero guarantee over durability.
Does it means that we need to call fsync()
twice, one on the file data, and one on its parent directory ? When you need to achieve durability, the answer is obviously yes. (Remember that file file/filename will be sync’ed on disk anyway by the operating system, so you do not actually need to do that for every single file - only for those you want to have a durability guarantee at a given time)
However, the question is causing some headache on the POSIX standard, and as a follow-up to the austin-group (ie. POSIX mailing-list) discussion, an editorial clarification request is still pending and is waiting for feedback from various implementors. (you may also have a look at the comp.unix.programmer discussion)
TL;DR: syncing a file is not as simple as it seems!
]]>I consider myself as a chef - yes, as in cooking. My interest in cuisine (I am French, I admit) has probably something to do with this comparison, but I have always found it adequate for multiple reasons.
The ingredients.
Of course, there’s always a simpler solution:
Some people will also find themselves unable to cook anything without specific instructions. Other will never follow any recipe, but will only write down rough steps of what they want to do.
But there is an even better reason for comparing coding and cooking: comparing the food industry with the IT industry.
There are basically two trends when you want to work in the food industry:
The fast food chains: neither food nor salaries are great, but it’s a good way to start if you know nothing about cooking, and/or do not want to be cooking for the rest of your life. If you are good enough, you’ll start to do more interesting things as flipping burgers, such as management, running your own business, etc. And no, you won’t be a chef, ever.
The restaurants: food is great (at least if this is a decent place), salaries CAN be great too (especially if you’re famous), but prepare yourself for hard work and pain. You’ll start peeling potatoes, and if you’re good enough, in few years you’ll become a useful aid inside a brigade. And maybe, one day, a great chef inside a three-star restaurant!
The two career paths are quite different, but most people understand that you may choose one or another depending on your motivation toward food. I mean, if you REALLY love food, you’ll try to become a chef. And nobody is going to consider this as a weird choice, or even less desirable. Most people will actually think the contrary.
In the IT industry, you may choose to work for a company focusing on management, business, and not at all on software engineering - think of contractors and integrators. You will attempt to stop coding as fast as possible, because, err, only looser still code after few years in these companies.
But for those who like to code, want to excel at finding new algorithms, discovering new technical challenges, the path is not quite the same, and the company profile radically different.
That’s why focusing on the good types of companies is absolutely vital for a newcomer in the IT industry. Working in small startups or in obscure software editors can be a great choice. You will learn, and you will learn by coding. And after few years, you’ll still code, design great algorithms, and lead incredible technical projects. And if you don’t feel like coding in few years, maybe you should choose the other path.
TL;DR: comparisons are always a bit lame, but, err, this one is not that bad. Achieve excellence in your code as a chef can in cooking!
]]>fopen()
are by default locked for deletion or renaming.
Ie. if you do this:
FILE *my_file = fopen("foo.bar", "wb");
...
then the foo.bar
file will be locked until the process closes the file (or dies). No renaming. No deleting.
This default feature is rather different from what we have on POSIX-like systems, and on Linux especially, where any file can basically be renamed or deleted while being opened. Once a file is being opened, its directory reference has basically no influence, except the implicit reference count (ie. deleting the file will only decrement the refcount, and closing the last handle will remove the underlying inodes)
This is actually a cool feature: you can have a file opened, and immediately unlinked, to have an anonymous file where you may execute read/write operations without bothering handling the cleanup if the process is killed, for example. You can also create a log file, redirect stdout/stderr, and later want to rotate the file, without being hit by permission denied
errors. This is especially true when all references to the file haven’t been closed yet - say, if you have stdio redirected to a log file, and if you have a child process spawned and logging in the same log file.
On, Windows, the infamous ERROR_ACCESS_DENIED
will kick you in the face as soon as you try to do the same.
There is actually a quite simple solution if you want to get a fopen()
semantic which is closer to the POSIX one: use the low-level CreateFile
function with the FILE_SHARE_DELETE
sharing mode.
The documentation says:
Hey, that’s exactly what we are looking for! But we want a nice FILE
pointer, not an horrible HANDLE
object. Is it possible ?
Yes!:
the HANDLE
object can be attached to a low-level C library handle through _open_osfhandle()
the low-level C library handle can be attached to a FILE*
object through _fdopen()
You just have to take some care about the passed fopen()
flags to open the file with proper flags, and your “compatible” function should be something like:
#include <stdio.h>
#include <fcntl.h>
#include <io.h>
#include <windows.h>
/**
* Opens the file whose name is the string pointed to by file and associates
* a stream with it.
**/
static FILE* fopen_unixlike(const char *file, const char *mode) {
DWORD dwDesiredAccess = 0;
DWORD dwCreationDisposition = 0;
const DWORD dwFlagsAndAttributes = FILE_ATTRIBUTE_NORMAL;
const DWORD dwShareMode = FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE;
size_t i;
HANDLE handle;
int fd;
/* Infer flags. */
for(i = 0; mode[i] != '\0'; i++) {
switch(mode[i]) {
case 'r':
dwDesiredAccess |= GENERIC_READ;
dwCreationDisposition = OPEN_EXISTING;
break;
case 'a':
dwDesiredAccess |= GENERIC_WRITE;
dwCreationDisposition = CREATE_NEW;
break;
case 'w':
dwDesiredAccess |= GENERIC_WRITE;
dwCreationDisposition = CREATE_ALWAYS;
break;
case '+':
dwDesiredAccess |= GENERIC_READ;
dwDesiredAccess |= GENERIC_WRITE;
break;
}
}
/* Create the file. */
handle = CreateFileA(file,
dwDesiredAccess,
dwShareMode,
NULL,
dwCreationDisposition,
dwFlagsAndAttributes,
NULL);
if (handle == INVALID_HANDLE_VALUE) {
return NULL;
}
/* Associates a C run-time file descriptor with a file HANDLE. */
fd = _open_osfhandle((intptr_t) handle, _O_BINARY);
if (fd == -1) {
CloseHandle(handle);
return NULL;
}
/* Associates a stream with a C run-time file descriptor. */
return _fdopen(fd, mode);
}
You can test this function using the example below:
/* Comment to test. */
#define fopen fopen_unixlike
int main(int argc, char* argv[]) {
FILE *fp = fopen("log.txt", "wb");
int i;
for(i = 0; i < 10; i++) {
if (fprintf(fp, "this is the log line #%d\r\n", i) > 0
&& fflush(fp) == 0) {
printf("successfully written line #%d, press a key\n", i);
_fgetchar();
} else {
fprintf(stderr, "error writting line #%d, press a key\n", i);
_fgetchar();
break;
}
}
return 0;
}
The “log.txt” file will be deletable and movable without troubles, and you may even successfully write to the deleted file.
TL;DR: I wish I had known sooner. Darn.
]]>The latest gem was this code snippet:
switch (step) {
while (1) {
case step_A:
case step_B:
...
}
}
No, there’s no mistake - there’s a while
interleaved between switch
and case
. Darn.
The fact is that switch
is actually a disguised dispatch of goto
(horror!) equivalent instructions, and case
a disguised label case. As you can jump from/to nested loops or conditions, the code is perfectly valid (and logical).
The above code can be rewritten as something like:
do {
if (step == step_A) {
goto A;
} else if (step == step_B) {
goto B;
} else {
goto E;
}
while (1) {
A:
B:
...
}
E:
} while(0);
This was confirmed by the specialists of comp.lang.c
, with interesting remarks:
void somewhat_fast_memcpy(char *to, const char *from, const size_t count) {
size_t n = (count + 7) / 8;
switch(count % 8) {
case 0:
do {
*to = *from++;
case 7: *to = *from++;
case 6: *to = *from++;
case 5: *to = *from++;
case 4: *to = *from++;
case 3: *to = *from++;
case 2: *to = *from++;
case 1: *to = *from++;
} while(--n > 0);
}
}
goto
equivalent ; such as uninitialized variables: (note that clang detect this case, but not my version of gcc) int i;
switch (step) {
for (i = 0; i < loops; i++) {
case step_A: // WARNING: uninitialized i
case step_B:
...
}
}
There’s one constraint you have to follow (please take your breath before reading): “If a switch statement has an associated case or default label within the scope of an identifier with a variably modified type, the entire switch statement shall be within the scope of that identifier.”. Basically, it means that if you are foolish enough to use this coding style, AND foolish enough to use variable-length-variables (such as dynamic arrays), then the compiler has the right to put a contract on you.
Abuse of interleaving of switch/case and other constructs may turn your code into a steaming pile of crap. Did I tell you that switch
/case
was somewhat equivalent to goto
?
TL;DR: take an aspirin and good nite.
]]>You have a nice application available on the Google Android Store and, as a developer, you have access to nice features giving you basic statistics (the number of downloads, Android version breakdown, etc.), reviews, and a Crashes & ANRs section allowing to audit user crash reports (and Application hangs - that’s the ANR thing).
This is a rather basic feature (you do not have any details on the user’s phone, Android version, etc. - only a Java stack trace extract), but at least it allows you to quickly spot mistakes (such as NullPointerException
, or in this case, a NumberFormatException
), and the report is pretty straightforward for users (they only have click on the “Report” button in case of crash)
But what if your application is using native code (through JNI) ? In such case, the application will just crash silently, giving the user no opportunity to report the bug to the upstream developer (you), which is not cool (because the bug will remain unspotted, unless users are nice enough to email you, and have the know-how to provide you useful technical details, such as the address of the crash, which is kind of rare)
A first step is obviously to be able to detect common native crashes (SIGSEGV
, SIGBUS
, etc.) using signal handlers. On POSIX systems, this can be achieved by using sigaction()
:
struct sigaction sa;
struct sigaction sa_old;
memset(&sa, 0, sizeof(sa));
sigemptyset(&sa.sa_mask);
sa.sa_sigaction = my_handler;
sa.sa_flags = SA_SIGINFO;
if (sigaction(sig, &sa, &sa_old) == 0) {
...
}
SIGSEGV
signal handler, running by default on the… same full stack, raising one more time the signal. Fortunately, you may register in any thread an alternative stack through the use of sigaltstack
, which basically reserve some space in case of emergency (ie. the system will switch the stack pointer to this one in case of trouble, letting you handler run on a “fresh” stack).stack_t stack;
memset(&stack, 0, sizeof(stack));
/* Reserver the system default stack size. We don't need that much by the way. */
stack.ss_size = SIGSTKSZ;
stack.ss_sp = malloc(stack.ss_size);
stack.ss_flags = 0;
/* Install alternate stack size. Be sure the memory region is valid until you revert it. */
if (stack.ss_sp != NULL && sigaltstack(&stack, NULL) == 0) {
...
}
SIGSEGV
might be regularly raised to address NullPointerException
or as normal JIT processing (ie. executable pages might be flagged with a “no access” protection, and filled by the JIT compiler through a signal handler) - you have to make sure the original signal handler is called first, before messing up with it. If the signal was not processed, the original signal handler will generally return, or will call abort()
(which is nice, because we have a last chance to catch it through a SIGABRT
handler)static void my_handler(const int code, siginfo_t *const si, void *const sc) {
/* Call previous handler. */
old_handler.sa_sigaction(code, si, sc);
...
}
pthread_getspecific()
to have a thread-specific context. Well, this is actually a dirty solution: pthread_getspecific()
is not an async-signal-safe function, which means that if you are using it on a signal handler, you may have to prepare for unforeseen consequences. (I fail to see what could go wrong with this specific function, however - this is just a peek in a thread-specific address array. But yes, yes, we’re playing with fire, don’t kick me!)static void my_handler(const int code, siginfo_t *const si, void *const sc) {
/* Call previous handler. */
old_handler.sa_sigaction(code, si, sc);
/* Get thread-specific context. */
my_struct *s = (my_struct*) pthread_getspecific(my_thread_var);
if (s != NULL) {
...
}
...
}
sigaction
callback is a pointer to a ucontext_t
context collecting register values (and various other processor-specific details). On x86-64 architectures, the program counter will typically be saved in uc_mcontext.gregs[REG_RIP]
; on ARM, uc_mcontext.arm_pc
. Unfortunately, on Android, the ucontext_t
structure is not defined in any system headers, and you’ll have to import one by yourself (I shamelessly copied the one from Richard Quirk). You also have to find out what was the binary where the program counter was actually running, to find out this code base address in memory, because a randomized address is not very useful for audit and debugging. The Linux-specific dladdr()
function is fortunately giving you this information, with useful other ones (namely the nearest symbol matching the address, and the module base address, to compute a relative offset address). (Note: you can also get this information on Linux by snooping in /proc/self/maps
, and checking the address ranges - it will at least provide you the base address)Dl_info info;
if (dladdr(addr, &info) != 0 && info.dli_fname != NULL) {
void * const nearest = info.dli_saddr;
const uintptr_t addr_relative =
((uintptr_t) addr - (uintptr_t) info.dli_fbase);
...
}
You have also the opportunity to catch a backtrace, with the same information, as long as you have a recent (ie. 4.1.1 or higher) Android version, using libcorkscrew
library features. This library is not available on older Android releases, and besides, we do not want to get a backtrace of the current stack, but a backtrace of the stack provided in the crash context. Fortunately, we can dynamically load the libcorkscrew.so
library to solve the first issue (using dlopen
and dlsym()
) and, for the second issue, import manually a nice function called unwind_backtrace_signal_arch
, which does exactly what we want:
/*
* Describes a single frame of a backtrace.
*/
typedef struct {
uintptr_t absolute_pc; /* absolute PC offset */
uintptr_t stack_top; /* top of stack for this frame */
size_t stack_size; /* size of this stack frame */
} backtrace_frame_t;
ssize_t unwind_backtrace_signal_arch(siginfo_t* siginfo, void* sigcontext,
const map_info_t* map_info_list,
backtrace_frame_t* backtrace, size_t ignore_depth, size_t max_depth);
(We also need to import acquire_my_map_info_list()
)
We can also, when corkscrew is there, use the advanced get_backtrace_symbols
function to resolve symbols and demangle them:
/*
* Describes the symbols associated with a backtrace frame.
*/
typedef struct {
uintptr_t relative_pc; /* relative frame PC offset from the start of the library, or the absolute PC if the library is unknown */
uintptr_t relative_symbol_addr; /* relative offset of the symbol from the start of the library or 0 if the library is unknown */
char* map_name; /* executable or library name, or NULL if unknown */
char* symbol_name; /* symbol name, or NULL if unknown */
char* demangled_name; /* demangled symbol name, or NULL if unknown */
} backtrace_symbol_t;
/*
* Gets the symbols for each frame of a backtrace.
* The symbols array must be big enough to hold one symbol record per frame.
* The symbols must later be freed using free_backtrace_symbols.
*/
void get_backtrace_symbols(const backtrace_frame_t* backtrace, size_t frames,
backtrace_symbol_t* backtrace_symbols);
RuntimeException
and unwind all Java frames, to have your final exception being reported through the Android framework. The only way to achieve that is by storing the exit point in your own code, using setjmp
(actually sigsetjmp
, because we’ll need to restore some masked signals), and using it in your signal handler to directly jump at the correct location. The sigsetjmp
/siglongjmp
functions are obviously not async-signal-safe (see the remark on Application Usage of sigaction()
), so this is a highly risky bet. Typically, if the crash happened in the middle of a malloc()
call (because, say, the linked list of free blocks has been corrupted), you may find yourself triggering another SIGSEGV
(which is the lesser of the evils) or worse, deadlocked, which is rather embarrassing because the user will have to find a way to kill the application by himself. For this reason, an alarm()
call will be the first operation executed in case of emergency (and yes, alarm()
is async-signal-safe - we can safely kill ourselves).static void my_handler(const int code, siginfo_t *const si, void *const sc) {
/* Call previous handler. */
old_handler.sa_sigaction(code, si, sc);
/* Trigger a time bomb. */
(void) alarm(30);
/* Get thread-specific context. */
my_struct *s = (my_struct*) pthread_getspecific(my_thread_var);
if (s != NULL) {
/* Store crash context for later. */
s->code = code;
s->si = *si;
s->uc = *(ucontext_t*) sc;
/* Jump back to initial location. */
siglongjmp(t->ctx, -1);
}
...
}
Yes, and it’s much nicer to have SIGSEGV
advertised through a clean stack:
FATAL EXCEPTION: AsyncTask #5
java.lang.RuntimeException: An error occured while executing doInBackground()
at android.os.AsyncTask$3.done(AsyncTask.java:299)
at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:352)
at java.util.concurrent.FutureTask.setException(FutureTask.java:219)
at java.util.concurrent.FutureTask.run(FutureTask.java:239)
at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:230)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1080)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
at java.lang.Thread.run(Thread.java:841)
Caused by: java.lang.Error: signal 11 (Address not mapped to object) at address 0x42 [at libhttrack.so:0xa024]
at com.httrack.android.jni.HTTrackLib.main(Native Method)
at com.httrack.android.HTTrackActivity$Runner.runInternal(HTTrackActivity.java:998)
at com.httrack.android.HTTrackActivity$Runner.doInBackground(HTTrackActivity.java:919)
at com.httrack.android.HTTrackActivity$Runner.doInBackground(HTTrackActivity.java:1)
at android.os.AsyncTask$2.call(AsyncTask.java:287)
at java.util.concurrent.FutureTask.run(FutureTask.java:234)
... 4 more
Caused by: java.lang.Error: signal 11 (Address not mapped to object) at address 0x42 [at libhttrack.so:0xa024]
at data.app_lib.com_httrack_android_2.libhttrack_so.0xa024(Native Method)
at data.app_lib.com_httrack_android_2.libhttrack_so.0x705fc(hts_main2:0x8f74:0)
at data.app_lib.com_httrack_android_2.libhtslibjni_so.0x4cc8(HTTrackLib_main:0xf8:0)
at data.app_lib.com_httrack_android_2.libhtslibjni_so.0x52d8(Java_com_httrack_android_jni_HTTrackLib_main:0x64:0)
at system.lib.libdvm_so.0x1dc4c(dvmPlatformInvoke:0x70:0)
at system.lib.libdvm_so.0x4dcab(dvmCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*):0x18a:0)
at system.lib.libdvm_so.0x385e1(dvmCheckCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*):0x8:0)
at system.lib.libdvm_so.0x4f699(dvmResolveNativeMethod(unsigned int const*, JValue*, Method const*, Thread*):0xb8:0)
at system.lib.libdvm_so.0x27060(Native Method)
at system.lib.libdvm_so.0x2b580(dvmInterpret(Thread*, Method const*, JValue*):0xb8:0)
at system.lib.libdvm_so.0x5fcbd(dvmCallMethodV(Thread*, Method const*, Object*, bool, JValue*, std::__va_list):0x124:0)
at system.lib.libdvm_so.0x5fce7(dvmCallMethod(Thread*, Method const*, Object*, JValue*, ...):0x14:0)
at system.lib.libdvm_so.0x54a6f(Native Method)
at system.lib.libc_so.0xca58(__thread_entry:0x48:0)
at system.lib.libc_so.0xcbd4(pthread_create:0xd0:0)
Bonus: produce a relative address, and get the filename and line number for FREE!
In the above stack, we know that the crash occurred somewhere inside the hts_main2
function, which is not very precise. We have actually another very useful information: the crash was spotted inside libhttrack.so
, at the relative address 0xa024
. This is a relative address, computed earlier with dladdr()
, which means that you can find out exactly the source location if you kept some debugging information. Most people do not want them, because it increases the binary size by a unreasonable factor (especially when running on small embedded devices with 3G+ connectivity priced above gold ingots levels), and thus either strip them silently, or keep another “debug” build.
You have an extremely simple alternative way: build once your libraries with all debugging symbols, including line numbers and macro information (-g3
), and instead of stripping them, split the debugging sections on a separate file (say, a .dbg
file). To let various tools such as gdb
or addr2line
behave gently, you have a way to “tell” them that the .so
actually has a debug .dbg
related file, through the .gnu_debuglink
ELF section.
Here’s what you typically need to do to split your library into a stripped version plus a debug symbol file:
# copy all debugging sections to dbg file
objcopy --only-keep-debug mylib.so mylib.dbg
# strip debug sections
objcopy --strip-debug mylib.so
# wipe any existing ELF .gnu_debuglink section if any
objcopy --remove-section .gnu_debuglink mylib.so
# set the .gnu_debuglink to the dbg file
objcopy --add-gnu-debuglink=mylib.dbg mylib.so
The nice .dbg
file can then be kept for debugging purpose:
cd /build-archives/httrack/armv7/3.47.99.35
./toolchains/arm-linux-androideabi-4.7/prebuilt/linux-x86_64/bin/arm-linux-androideabi-addr2line -C -f -e libhttrack.so 0xa024
fourty_two
src/htscoremain.c:111
A typical crash may have a great variety of causes (Captain Obvious to the rescue!). My own experience, though, shows that NULL pointer dereferencing, dangling pointers, and other isolated crash spots due to bad code logic are a very common cause of crashes. Yes, you will still have troubles when dealing with corrupted allocators, or when breaking in the middle of a async-signal-safe call (leaving mutexes locked, and more generally a dirty state that will hit you back later), but the alternate solution is to die immediately (the default behavior), so trying to do gentle emergency steps can not do any harm.
You can check out the “CoffeeCatch” library code on GitHub - you can either merge the .c
file in your project(s), or build it as a standalone library ; this tiny library has no exotic external dependencies. Make sure all your libraries are built with the -funwind-tables
compiler flag to produce frame unwind information for all functions (add if necessary LOCAL_CFLAGS := -funwind-tables
to your Android.mk
file). Note that -funwind-tables
does not produce significant data size overhead in normal situations - so rest well my friend.
The use is pretty straightforward - for JNI code, a simple macro COFFEE_TRY_JNI
, taking the JNIEnv
environment pointer as a first argument, and a code block as a second argument, will do the trick. For code outside JNI, you have to enclose protected code with COFFEE_TRY()/COFFEE_CATCH()/COFFEE_END() as in the example below.
In all cases, make sure the block is enclosed in a dedicated function without any local variables lying around (because the saved context does not include registers, AFAIK, especially volatile registers, which might be wiped before calling sigsetjmp
, and whose saved values are in unknown location in the stack).
The JNI flavor:
/** The potentially dangerous function. **/
jint call_dangerous_function(JNIEnv* env, jobject object) {
// ... do dangerous things!
return 42;
}
/** Protected function stub. **/
void foo_protected(JNIEnv* env, jobject object, jint *retcode) {
/* Try to call 'call_dangerous_function', and raise proper Java Error upon
* fatal error (SEGV, etc.). **/
COFFEE_TRY_JNI(env, *retcode = call_dangerous_function(env, object));
}
/** Regular JNI entry point. **/
jint Java_com_example_android_MyNative_foo(JNIEnv* env, jobject object) {
jint retcode = 0;
foo_protected(env, object, &retcode);
return retcode;
}
The standard flavor:
static __attribute__ ((noinline)) void demo(int *fault) {
COFFEE_TRY() {
recurse_madness(42);
*fault = 0;
} COFFEE_CATCH() {
const char*const message = coffeecatch_get_message();
snprintf(string_buffer, sizeof(string_buffer), "%s", message);
*fault = 1;
} COFFEE_END();
}
TL;DR: so far, no native crash has been reported on Android for HTTrack. But I’m ready to collect them in case of need :)
]]>The first official Android release of HTTrack has just been published on the Android store.
This was really fun to port HTTrack on this platform, actually. I never played too much with smartphones before (I usually don’t play at all with them, by the way, because everything I need to work or to connect to online resources is a real computer, not a smart-toy), but even if the httrack engine is 100% pure-C, the portage was not too difficult.
the SDK provided by Google is clean enough do to complicated things without too much hassle (such as cross-compiling ARM bytecode on x86 machines) and is available on most platforms (at least Windows and Linux)
the Eclipse-derivated developer environment (ADT) is rather well integrated, with all the nice features you would expect (including helping you tracking errors in obscure XML files, or starting gently device emulators)
you do not even need any smartphone to start ; you can emulate on almost any type of devices (smartphones, tablets…) using the provided emulator and OS images
the Android development API (in Java) is pretty straightforward, even when you start to mix native interface code (JNI)
To be clear, my first “hello, world” sample was created within minutes, and I then decided to directly start the httrack port.
I’m developing HTTrack on Linux, but I also use Windows as my main development platform, for practical reasons, and for the Windows release, which is currently the main “target” of HTTrack (and also, steam has more games on Windows, even if the Linux port is also quite excellent, but this is another story).
So I started by building the httrack engine code for android, of course. The first difficult task was to be able to build the original source code with the default build system (ie. Autotools/libtool). I could have created an Android mk file for that, but on the other hand I preferred to let libtool handle the dirty things (such as checking if v*printf was working, etc.) and have an unified build.
Another issue was the Android system itself: many common libraries, such as iconv
or openSSL
, are not available on this platform ; so I had to disable several features for the first releases.
After some tweaking, I finally got a working Makefile using something similar to:
./configure -host=arm-none-linux \
--prefix=${DEST_DIR} \
--enable-shared --disable-static \
--with-sysroot=${ANDROID_SDK}/platforms/android-5/arch-arm \
--with-zlib=${ANDROID_SDK}/platforms/android-5/arch-arm/usr \
--includedir=${ANDROID_SDK}/platforms/android-5/arch-arm/usr/include \
CC="${ANDROID_SDK}/toolchains/arm-linux-androideabi-4.6/prebuilt/linux-x86_64/bin/arm-linux-androideabi-gcc --sysroot=${ANDROID_SDK}/platforms/android-5/arch-arm -I${ANDROID_SDK}/platforms/android-5/arch-arm/usr/include -L${ANDROID_SDK}/platforms/android-5/arch-arm/usr/lib" \
CFLAGS="-DANDROID -D_ANDROID -DLIBICONV_PLUG -I${DEST_DIR}/include -L${DEST_DIR}/lib" \
&& sed -i -e 's/CPPFLAGS = .*/CPPFLAGS =/' Makefile \
&& sed -i -e 's/CPPFLAGS = .*/CPPFLAGS =/' src/Makefile \
&& find . -name Makefile -exec sed -i -e "s/\(.*LDFLAGS = \)-version-info .*/\1 -avoid-version/" {} \; \
&& make -j8 && make install DESTDIR=${DEST_DIR} \
&& make install
Yuk, yes, I know, this is a little verbose, but, believe me, this is not as horrible as it seems.
The good thing is that I got rather quickly a libhttrack.so
library containing everything I wanted (yay), without having two build systems. I was not too anxious, though: httrack is successfully built in a variety of platforms on Debian, including ARM architectures.
The second step was to create some kind of interface (GUI) with the Android libraries, because you can not have (easily) a command line interface to play with. The first version only had a very minimalistic screen to play with (mostly a “Start” button).
Later, I added more cosmetic features, such as Next/Previous-style navigation, option panel with tabs, and some nice idiot-proof features (detecting common obvious mistakes, such as empty project name, empty URL, etc.)
One of the recurring complaint about the Windows version is the relative ugliness of its GUI. I must admit that my UI-design-skills are far from being perfect, and the original Windows GUI version took an insanely large amount of time to be developed (especially for an ugly result - but hey, I’m not developing a photo gallery, I’m developing a system utility).
In comparison, the Android GUI was much easier to develop (some people will probably find it likewise ugly - okay - but at least I did not spend two months on it), and much nicer in term of extensibility (adding tabs, scrollviews, etc. is pretty straightforward).
As I said earlier, neither iconv nor OpenSSL are officially provided on Android platforms.
iconv is not provided at all (even in libc
)
OpenSSL is provided as binary-only (no include files in ther SDK)
iconv is the easier one to build ; Danilo Giulianelli wrote a very clear entry on how to build iconv on Android.
OpenSSL was more difficult, and I basically used the Guardian Project patched source at the beginning, only to discover that /system/lib/libcrypto.so
and /system/lib/libssl.so
are actually provided - you just need to get the include files, and hope the ABI is more or less stable. My experience with OpenSSL suggest that this should be true (I used to dynamically load OpenSSL in httrack on POSIX platforms - probing any available release - in a foolish attempt not to be tied to crazy exports regulations), and the rather small subset of OpenSSL functions I am using makes me confident that this… daring solution should not cause too much trouble.
In parallel, I started to write JNI interface code to be able to start the engine from the Java code.
This part was a bit more tricky, for a variety of reasons:
Therefore, if you do:
jclass myClass = (*env)->FindClass(env, name);
then you have to remember than myClass
is only a local object reference, and if you keep it in some global variable, you’re gonne have a very bad time (ie. random crashes).
Therefore, you basically need to create a global reference:
static jclass findClass(JNIEnv *env, const char *name) {
jclass localClass = (*env)->FindClass(env, name);
/* "Note however that the jclass is a class reference and must be protected
* with a call to NewGlobalRef " -- DARN! */
if (localClass != NULL) {
return (*env)->NewGlobalRef(env, localClass);
}
return NULL;
}
quite obvious, yes, but forget this obvious point and you are doomed!
Another detail, by the way: when you throw an exception, remember that the string passed has to be valid during the lifetime of the underlying String object. The small details are often the most annoying things…
You have also to make sure you don’t have too many local reference objects on the current stack, because the default one is pretty small (512 references at most). Each time you create an object (including strings), each time you get a member from a structure, you create an additional local reference which needs to be deleted as soon as possible when running in inner JNI code loops.
For example, converting a String[]
into a char*[]
should look like:
/* Create array */
size_t i;
for (i = 0; i < argc; i++) {
/* Note: a local reference is created here */
jstring str = (jstring)(*env)->GetObjectArrayElement(env, stringArray, i);
const char * const utf_string = (*env)->GetStringUTFChars(env, str, 0);
argv[i] = strdup(utf_string != NULL ? utf_string : "");
(*env)->ReleaseStringUTFChars(env, str, utf_string);
(*env)->DeleteLocalRef(env, str);
}
argv[i] = NULL;
Of course, you also have the solution to create a temporary local frame (something Java do when calling functions, for example) which will allow to wipe all created local references using PopLocalFrame
:
/* create a new local frame for local objects */
if ((*t->env)->PushLocalFrame(t->env, capacity) == 0) {
...
(void) (*t->env)->PopLocalFrame(t->env, NULL);
} else {
... throw something
}
/* Our own assert version. */
static void assert_failure(const char* exp, const char* file, int line) {
/* FIXME TODO: pass the getExternalStorageDirectory() in init. */
FILE *const dumpFile = fopen("/mnt/sdcard/Download/HTTrack/error.txt", "wb");
if (dumpFile != NULL) {
fprintf(dumpFile, "assertion '%s' failed at %s:%d\n", exp, file, line);
fclose(dumpFile);
}
abort();
}
#undef assert
#define assert(EXP) (void)( (EXP) || (assert_failure(#EXP, __FILE__, __LINE__), 0) )
I could also display some fancy message to the user directly, by calling the appropriate android java code…
LD_LIBRARY_PATH
ready with all your beloved libraries ; it means basically that you have to load all libraries and dependencies in reverse topological order, using System.loadLibrary()
calls in static initializer. I also realized that some libraries, such as OpenSSL
, are actually present on the system - wthout any headers being provided, though. Therefore, you have to dig a little to sort things out…The typical HTTrack library initializer is then looking like:
/** OpenSSL (for HTTPS); taken from the /system libraries **/
System.loadLibrary("crypto");
System.loadLibrary("ssl");
/** Iconv (Unicode). **/
System.loadLibrary("iconv");
/** HTTrack core engine. **/
System.loadLibrary("httrack");
/**
* HTTrack Java plugin (note: dlopen()'ed by HTTrack, has symbol
* dependencies to libhttrack.so).
**/
System.loadLibrary("htsjava");
/** HTTrack Android JNI layer. **/
System.loadLibrary("htslibjni");
Connecting all pieces together took a bit more time, too. On Android, you have to handle a variety of cases:
Permissions: this is probably one of the beginner’s first headache source. If you need to write to the external SDCard, for example, you will need to declare some special permission in your Android.xml
application file (in this case, android.permission.WRITE_EXTERNAL_STORAGE
) so that the system can let you do what you want. You won’t be able to connect to the outside world without android.permission.INTERNET
, either.
Orientation change: when an user is changing the screen orientation (by rotating the device), your Activity is by default killed, and recreated. This was a bit surprising for me, but fortunately you can put all working jobs on an isolated “fragment” container, and let the system know that you may gently save and restore the state of your GUI (Activity) if needed:
/* First attempt to reattach to an existing background job */
final FragmentManager fm = getSupportFragmentManager();
runner = (RunnerFragment) fm.findFragmentByTag(id);
/* Create a new "fragment" (our background job runner) */
if (runner == null) {
final FragmentManager fm = getSupportFragmentManager();
runner = new RunnerFragment();
runner.setParent(this);
fm.beginTransaction().add(runner, id).commit();
}
You also need to overrivde onSaveInstanceState
and onRestoreInstanceState
in your Activity to be able to retrieve your current context.
Your application can be killed at any time, for example if the user hits the “back” button to exit your main GUI. You have to take care of saving necessary settings, and stop running background jobs (especially native ones!) gently, and if necessary suggest to restart an interrupted job later.
Limited amount of resources: especially when playing with JNI, you have to take extreme care of how many objects, how many local references etc. you are using (see above remarks). You also have to take care on where you are storing data - writing directly on the SDcard root is considered a bad practice ; you also may not have any SDcard at all on the device, etc.
It took me basically two weeks to have a final release. Oh, I’m not saying that the first release is perfect, and you may experience bugs and problems, but the first tests were quite successful.
And it was a rather pleasant time - considering my total lack of experience on this field, the portage was rather straightforward, even with a project involving not trivial tasks (JNI, background jobs, etc.)
If you want to check out the code, feel free to browse it.
TL;DR: developing applications on Android, even including native code, is rather straightforward!
]]>Yes, MD5 is your friend. Believe me.
My first encounter with MD5 was around fifteen years ago, when I implemented the first hashtable in httrack. The code was not really glorious, but at least I quickly discovered something: if you design your own hashing function, you’re gonna have a bad time.
Here’s what I first implemented (please do not scream):
unsigned long int hash_for_dummies(const unsigned char* s) {
unsigned long sum;
unsigned int i;
for(i = 0, sum = 0 ; *s != '\0' ; i++) {
sum += 1;
sum += (unsigned int) *s;
sum *= (unsigned int) *(s++);
sum += i;
}
return sum;
}
Well, on several (I mean, on one of two) examples, it was kind of a cool hashing function actually. Of course, once being used inside my hashtable, things did not go so well, and on several sites with special URL patterns, the table exploded (in term of complexity).
Oh yes, I forgot to mention: the hashing function was meant to be used in a hashtable, and the keys being hashed were URLs. The funny thing with URLs is that you may have any pattern:
/session/?5209d5f396ca0d46
)/I-made-a-huge-mistake
)/page/1
, /page/2
, … /page/999999
)When hashing with a specific function, you may have very good results for one of specific pattern, or for several families, but time to time you discover that the hashing function betrayed you in specific cases.
Yes, yes, but you ruined the plot! Okay, that was expected, but yet, after discovering my poor hashing abilities, I decided to use the thermo-nuclear option: MD5, the cryptographic hashing function, which has been studied during long years, and which is considered an excellent solution for hashing anything.
And it was good. Hardly no collisions. Perfect distribution of keys. No nightmares about specific URL patterns that would ruin everything. MD5 was perfect.
Ah, yes, the cost. MD5 has a bad reputation of being over-costly. If you look at the internal loop, you will discover an intimidating block of 64 MD5STEP macros such as:
#define F1(x, y, z) (z ^ (x & (y ^ z)))
#define MD5STEP(f, w, x, y, z, data, s) \
( w += f(x, y, z) + data, w = w<<s | w>>(32-s), w += x )
...
MD5STEP(F1, a, b, c, d, in[0] + 0xd76aa478, 7);
MD5STEP(F1, d, a, b, c, in[1] + 0xe8c7b756, 12);
// ... during 64 lines
If you look closer, the internal loop function is actually processing 64 bytes at once (with final zero-padding) ; which mean that for each byte, a dozen simple (add
, xor
, shift…) operations is executed. That’s a lot, and not so much at the same time. Most operations can be executed in one cycle or less (pipelined) by a modern processor - and it has to be compared to more expensive operations (such as mul
) used in other hashes.
So yes, MD5 is probably slower. Well, actually it is:
MD5 | FNV-1 | Murmur 3 |
---|---|---|
6.0s | 4.40s | 1.54s |
(Tests executed on a 3801493504 bytes file, on a Xeon(R) CPU E5-1620 0 @ 3.60GHz)
Basically 50%
slower than some FNV-1 variant, and four times
slower than Murmur3.
No, actually not. Remember one thing: we’re not hashing the universe, we’re hashing small keys (No, keys are not supposed to be huge binary blobs). We have plenty of work to do after that (like, uh, storing the value in the hashtable, and probably many other interesting things with them), and the keys hashing is only a small subset of the code.
If you do more extensive tests, such as, inserting and reading to a hashtable, things are less clear:
FNV-1 | MD5 | Murmur |
---|---|---|
1m13s | 1m54 | 1m13s |
(Tests executed on 10000000 keys progressively, with keys of length between 80 and 100 bytes, with insert/overwrite/delete patterns.)
Yep, basically MD5 is only increasing marginally (well, yes, I call 30% “marginally” - I know everybody would love to earn marginally more) the overall hashtable cost.
Yes. And I don’t care. Remember the URLs pattern I was talking about previously ? Well, with MD5, I can sleep quietly. I don’t have to worry whether the hash function has been studied correctly. MD5 has. And even if on several tests Murmur has proven to be a reliable hashing function (with the same number of collisions than MD5, basically), I’m still reluctant in exchanging a little CPU cost with a bit of algorithmic uncertainty.
TL;DR: I stopped worrying and learned to love MD5.
Note: during my various tests, I used 64-bit hashing functions variants, with shift+mask to isolate the number of bits needed (ie. the hashtable bucket size was a power of two). The hashes were always processed by computing 128-bit hashing data, XOR-folded into a 64 bit version.
]]>