<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">

  <title><![CDATA[xavier roche's homework]]></title>
  <link href="https://blog.httrack.com/atom.xml" rel="self"/>
  <link href="https://blog.httrack.com/"/>
  <updated>2026-08-10T15:49:28+02:00</updated>
  <id>https://blog.httrack.com/</id>
  <author>
    <name><![CDATA[Xavier Roche]]></name>
    
  </author>
  <generator uri="http://octopress.org/">Octopress</generator>

  
  <entry>
    <title type="html"><![CDATA[Compiler-error-abuse-based large scale renaming in C++]]></title>
    <link href="https://blog.httrack.com/blog/2022/12/09/compiler-error-abuse-based-large-scale-renaming-in-c++/"/>
    <updated>2022-12-09T15:07:00+01:00</updated>
    <id>https://blog.httrack.com/blog/2022/12/09/compiler-error-abuse-based-large-scale-renaming-in-c++</id>
    <content type="html"><![CDATA[<p>Many developers hate compiler errors. This can become a daily pain, an endless stream of misery and frustration. But what if I told you compiler errors may help you refactor safely and quickly an entire codebase to rename class members and types ?</p>

<h2 id="a-story-of-renaming">A Story of Renaming</h2>

<p>At my company, we have been using string views for quite some time in our codebase - long before <code class="language-plaintext highlighter-rouge">std::string_view</code> was <em>cool</em> (that is, before C++17) - and we recently realized that some of our initial choices might not have been completely wise at that time.</p>

<p>Here’s what we used to write nearly ten years ago (lots of boilerplate removed):</p>

<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">MyFancyView</span>
<span class="p">{</span>
<span class="nl">public:</span>
    <span class="n">MyFancyView</span><span class="p">(</span><span class="k">const</span> <span class="kt">char</span><span class="o">*</span> <span class="n">ptr</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">ptrlen</span><span class="p">)</span> <span class="p">...</span>
<span class="p">...</span> <span class="n">lots</span> <span class="n">of</span> <span class="n">fancy</span> <span class="n">functions</span> <span class="n">out</span> <span class="n">there</span>

<span class="nl">public:</span>
    <span class="k">const</span> <span class="kt">char</span><span class="o">*</span> <span class="n">data</span><span class="p">;</span>	<span class="c1">// public data pointer</span>
    <span class="kt">size_t</span> <span class="n">len</span><span class="p">;</span>		    <span class="c1">// public length</span>
<span class="p">};</span>
</code></pre></div></div>

<p>First obvious issue with this primitive view are the public members <code class="language-plaintext highlighter-rouge">data</code> and <code class="language-plaintext highlighter-rouge">len</code>, something which is now frowned upon (for good reasons, as this tend to allow hacky changes in address and pointer). The other issue is that, if we want to be closer to the standard library API, accessors needs to be <code class="language-plaintext highlighter-rouge">data()</code> and <code class="language-plaintext highlighter-rouge">size()</code>.</p>

<p>The solution is rather straightforward: we need to rewrite thousands of lines of code using those direct members, using accessors, and sometimes change the code logic to use internal view logic - typically trimming views, taking a sub-view etc.
Simple but overwhelming: the operation needs to be carefully done, line by line.</p>

<p>Besides,</p>
<ul>
  <li>Using an editor to do the work <em>may</em> work, but it also may lead to errors. You may replace members in stuff that do not belong. You may also miss a lot of ugly legacy macros embedding the renamed members. The compiler is the only one knowing which type has to be replaced and where.</li>
  <li>Using tools such as <code class="language-plaintext highlighter-rouge">clang-rename</code> did not appear to help either, as they seem more convenient to replace one file at a time, with an offset provided by an editor typically.</li>
  <li>Using a straight replacement of all <code class="language-plaintext highlighter-rouge">data</code> into <code class="language-plaintext highlighter-rouge">data()</code> and <code class="language-plaintext highlighter-rouge">len</code> into <code class="language-plaintext highlighter-rouge">size()</code> in the codebase would lead to havoc: we have a lots of stuff out there with those names and this will never work.</li>
</ul>

<p>And there’s a last catch: programmers are lazy. There is no world where we’ll do that manually!</p>

<p>Oh, and there’s a perfectly valid excuse not to do it anyway: the risk of introducing subtle regressions is great, and the risk of tuning any code reviewer crazy is even greater.</p>

<p>So let’s start from the beginning: we’d like to rename those two members.</p>

<p>The strategy needs to be a bit <em>subtle</em>:</p>

<ol>
  <li>Rename the member definitions to an intermediate name that shall be <em>unique</em> (ie. in a unique pattern) in the codebase</li>
  <li>Identify all members references to be renamed and replace them with the intermediate pattern</li>
  <li>Then, we can start doing blind replacements in the codebase</li>
</ol>

<p>The first item has a pretty straightforward hack: let’s rename <code class="language-plaintext highlighter-rouge">data</code> into <code class="language-plaintext highlighter-rouge">dAtA</code>, and <code class="language-plaintext highlighter-rouge">len</code> into <code class="language-plaintext highlighter-rouge">lEn</code> in our definition file!</p>

<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code> public:
<span class="gd">-    const char* data{ nullptr };
-    size_t len{ 0 };
</span><span class="gi">+    const char* dAtA{ nullptr };
+    size_t lEn{ 0 };
</span> };
</code></pre></div></div>

<p>Why the weird <a href="https://en.wikipedia.org/wiki/Leet">l33t</a> case ? I’m glad you ask! Here’s the reason:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git <span class="nb">grep</span> <span class="nt">-E</span> <span class="s2">"(dAtA|lEn)"</span>
</code></pre></div></div>

<p>The command above did not yield any matches in our codebase. We have our perfect namings!</p>

<p>One more thing: you might have not realize it, but those two new namings do also have the same <em>length</em> as previously, so that replacing them won’t change a line size. You’ll see later why this is important.</p>

<p>What’s happening if we build the code without any changes ? Well, we’ll obviously have a couple of those errors before the compiler gives up on our own foolishness:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>../types/MixedFancy.h:84:16: error: no member named 'len' in 'MyFancyView'
    return lhs.len == rhs.size() &amp;&amp; (lhs.len == 0 || __builtin_memcmp(lhs.data, rhs.data(), lhs.len) == 0);
</code></pre></div></div>

<p>This message is actually cool. It provides four major information:</p>

<ul>
  <li>The error we seek to fix (no member named ‘len’)</li>
  <li>The file involved (types/MixedFancy.h)</li>
  <li>The line number involved (84)</li>
  <li>And even the column number involved (16)</li>
</ul>

<p>And remember those information are extracted from the compiler itself: we can rely on them.</p>

<p>What more could you even ask for ? A margarita ?</p>

<p>If we <em>grep</em> for <code class="language-plaintext highlighter-rouge">error: no member named</code>, then isolate the leftmost stuff (file:line:column), we almost are set to fix the error.</p>

<p>With this, if we extract the file, the line, and the column, we can now do our replacement:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">LANG</span><span class="o">=</span>C <span class="nb">sed</span> <span class="se">\</span>
    <span class="nt">-E</span> <span class="s2">"84s/^(.{15})len/</span><span class="se">\1</span><span class="s2">lEn/"</span> <span class="se">\</span>
    <span class="nt">-i</span> <span class="s2">"../types/MixedFancy.h"</span>
</code></pre></div></div>

<p>The leading <code class="language-plaintext highlighter-rouge">84</code> before sed <code class="language-plaintext highlighter-rouge">s</code> option is the line number, and we need to capture 15 (16 - 1) characters from the beginning of the line before column 16. Then, replace the captured part, the new name, and we’re all set. Using LANG=C is required as the compiler we use (<code class="language-plaintext highlighter-rouge">clang</code>) will provide offsets at byte level.</p>

<p>A last detail: we’re lazy (we’re merely <em>humans</em> after all), but the compiler shouldn’t be.
We need to tell the compiler not to worry if we have thousands of errors, by adding <code class="language-plaintext highlighter-rouge">-ferror-limit=0</code> in the cmake’s <code class="language-plaintext highlighter-rouge">cxx_flags</code>.
We also need to tell ninja not to be too lazy (by using <code class="language-plaintext highlighter-rouge">-k0</code>)</p>

<p>With all this, we can have the final renaming script main logic:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ninja <span class="nt">-k0</span> <span class="nt">-C</span> <span class="nb">.</span> 2&gt;&amp;1 |
  <span class="c"># capture compiler errors</span>
  <span class="nb">grep</span> <span class="s1">'error: no member named'</span> |
  <span class="c"># extract file, line, and column</span>
  <span class="nb">cut</span> <span class="nt">-f1-3</span> <span class="nt">-d</span><span class="s1">':'</span> |
  <span class="c"># sort all</span>
  <span class="nb">sort</span> <span class="nt">-u</span> |
  <span class="c"># only keep first file/line when several patterns exist on a single line</span>
  <span class="nb">sort</span> <span class="nt">-u</span> <span class="nt">-k1</span>,2 <span class="nt">-t</span><span class="s1">':'</span> |
<span class="c"># now bulk replace</span>
<span class="k">while </span><span class="nb">read </span>line<span class="p">;</span> <span class="k">do</span>
    <span class="c"># extract filename, line, column</span>
    <span class="nv">f</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$line</span><span class="s2">"</span>|cut <span class="nt">-f1</span> <span class="nt">-d</span><span class="s1">':'</span><span class="si">)</span><span class="s2">"</span>
    <span class="nv">l</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$line</span><span class="s2">"</span>|cut <span class="nt">-f2</span> <span class="nt">-d</span><span class="s1">':'</span><span class="si">)</span><span class="s2">"</span>
    <span class="nv">c</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$line</span><span class="s2">"</span>|cut <span class="nt">-f3</span> <span class="nt">-d</span><span class="s1">':'</span><span class="si">)</span><span class="s2">"</span>
    <span class="c"># replace at line l and column c either of the pattern</span>
    <span class="nv">LANG</span><span class="o">=</span>C <span class="nb">sed</span> <span class="nt">-E</span> <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$f</span><span class="s2">"</span> <span class="se">\</span>
        <span class="nt">-e</span> <span class="s2">"</span><span class="k">${</span><span class="nv">l</span><span class="k">}</span><span class="s2">s/^(.{</span><span class="k">$((</span>c-1<span class="k">))</span><span class="s2">})data/</span><span class="se">\1</span><span class="s2">dAtA/"</span> <span class="se">\</span>
        <span class="nt">-e</span> <span class="s2">"</span><span class="k">${</span><span class="nv">l</span><span class="k">}</span><span class="s2">s/^(.{</span><span class="k">$((</span>c-1<span class="k">))</span><span class="s2">})len/</span><span class="se">\1</span><span class="s2">lEn/"</span>
<span class="k">done</span>
</code></pre></div></div>

<p>It is <em>highly</em> suggested you split each step above until you’re confident enough, to be able to hunt for any mistakes.</p>

<p>A note on the fact we have the same name length for replaced versions:</p>

<div class="language-c++ highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">std</span><span class="o">::</span><span class="n">find</span><span class="p">(</span><span class="n">_elements</span><span class="p">.</span><span class="n">data</span><span class="p">,</span> <span class="n">_elements</span><span class="p">.</span><span class="n">data</span> <span class="o">+</span> <span class="n">_elements</span><span class="p">.</span><span class="n">size</span><span class="p">,</span> <span class="n">fun</span><span class="p">)</span>
</code></pre></div></div>

<p>This code above will trigger <em>two</em> warnings, on the same line, but on two different columns. If we were to replace with a name of a different length, this would completely mess up our script.</p>

<h2 id="going-further">Going Further</h2>

<p>That’s nice, but we also have a lot of stuff inside macros. I know, macros are bad, but legacy reasons…</p>

<p>Errors inside those macros will typically generate stacked error messages; such as in this case where we have two macros calling each other involved (in this case, we were renaming <code class="language-plaintext highlighter-rouge">MyFancyView</code> itself):</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>test/MyFancyTests.cpp:72:9: error: unknown type name 'MyFancyView'
        TEST_ENTITY("&amp;#12345;", 0x3039, false);
        ^
test/MyFancyTests.cpp:64:48: note: expanded from macro 'TEST_ENTITY'
#define TEST_ENTITY(name, value, onlyAlphaNum) _TEST_ENTITY(entities, name, value, onlyAlphaNum)
                                               ^
test/MyFancyTests.cpp:48:9: note: expanded from macro '_TEST_ENTITY'
...
</code></pre></div></div>

<p>The error location is actually on <code class="language-plaintext highlighter-rouge">MyFancyTests.cpp:48:9</code>, but its message is triggered at <code class="language-plaintext highlighter-rouge">MyFancyTests.cpp:72:9</code>.</p>

<p>To capture those cases, a multiline grep capture (<a href="https://stackoverflow.com/questions/3717772/regex-grep-for-multi-line-search-needed">as demonstrated on StackOverflow</a>) with <code class="language-plaintext highlighter-rouge">grep -Pzo</code> will do the magic:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">errMatch</span><span class="o">=</span><span class="s2">"(unknown type name|use of undeclared identifier|no type named|no member named)"</span>
<span class="nv">fromMacro</span><span class="o">=</span><span class="s2">"note: expanded from macro"</span>
<span class="nb">grep</span> <span class="nt">-aPzo</span> <span class="se">\</span>
		<span class="s2">"(?s)[^</span><span class="se">\n</span><span class="s2">]*: error: </span><span class="k">${</span><span class="nv">errMatch</span><span class="k">}</span><span class="s2"> '[A-Za-z0-9</span><span class="se">\.</span><span class="s2">_]*'[^</span><span class="se">\n</span><span class="s2">]*</span><span class="se">\n</span><span class="s2">([^</span><span class="se">\n</span><span class="s2">]*</span><span class="se">\n</span><span class="s2">[^</span><span class="se">\n</span><span class="s2">]*</span><span class="se">\n</span><span class="s2">)?[^</span><span class="se">\n</span><span class="s2">]*: </span><span class="k">${</span><span class="nv">fromMacro</span><span class="k">}</span><span class="s2">[^</span><span class="se">\n</span><span class="s2">]*(</span><span class="se">\n</span><span class="s2">([^</span><span class="se">\n</span><span class="s2">]*</span><span class="se">\n</span><span class="s2">[^</span><span class="se">\n</span><span class="s2">]*</span><span class="se">\n</span><span class="s2">)?[^</span><span class="se">\n</span><span class="s2">]*: </span><span class="k">${</span><span class="nv">fromMacro</span><span class="k">}</span><span class="s2">[^</span><span class="se">\n</span><span class="s2">]*)*"</span>
</code></pre></div></div>

<p>Then, with this capture, we need to cleanup null characters (emitted by the -z flag), keep only what we need (ie. filter only the desired symbols), and capture the four information we need.</p>

<h2 id="the-final-script">The Final Script</h2>

<p>The final script I used is the following. It was aimed for <code class="language-plaintext highlighter-rouge">clang</code>, but could be adapted for other compilers. Feel free to use it as inspiration, and happy renaming!</p>

<ul>
  <li>It enables <code class="language-plaintext highlighter-rouge">-ferror-limit=0</code></li>
  <li>It first patches files holding the class/members to be renamed</li>
  <li>Compute aLtErNaTe case intermediate patterns for replacement</li>
  <li>Check we don’t have hits on intermediate patterns</li>
  <li>Replace the files holding the definitions of the types we want to rename</li>
  <li>Loop until we don’t have any more errors
    <ul>
      <li>Capture errors and locate patterns to replace</li>
      <li>Replace patterns</li>
    </ul>
  </li>
  <li>Final name replacements from aLtErNaTe case to final versions</li>
  <li>Removal of idempotent types if we replaced a type by its final alias</li>
  <li>A bit of clang-tidy where we are at it!</li>
</ul>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>

<span class="nb">set</span> <span class="nt">-e</span>

<span class="c"># Source and build location</span>
<span class="nb">readonly </span><span class="nv">sources</span><span class="o">=</span><span class="nv">$HOME</span>/git/codebase
<span class="nb">readonly </span><span class="nv">build</span><span class="o">=</span><span class="k">${</span><span class="nv">sources</span><span class="k">}</span>/build
<span class="nb">readonly </span><span class="nv">headerFiles</span><span class="o">=(</span>MyFancyView.hpp<span class="o">)</span>

<span class="c"># List of replacements</span>
<span class="c"># TODO: put your own replacements here</span>
<span class="nb">declare</span> <span class="nt">-A</span> <span class="nv">replacements</span><span class="o">=(</span>
	<span class="o">[</span><span class="s2">"MyFancyView"</span><span class="o">]=</span><span class="s2">"StandardView"</span>
	<span class="o">[</span><span class="s2">"MyFancuString"</span><span class="o">]=</span><span class="s2">"U16StandardView"</span>
	<span class="o">[</span><span class="s2">"MyFancuUnicodeString"</span><span class="o">]=</span><span class="s2">"U32StandardView"</span>
	<span class="o">)</span>

<span class="c"># Function to return aLtErNaTe case version of $1</span>
altCase<span class="o">()</span> <span class="o">{</span>
	<span class="nb">local </span><span class="nv">result</span><span class="o">=</span>
	<span class="nb">local </span>i
	<span class="nb">local </span><span class="nv">up</span><span class="o">=</span>
	<span class="nb">local </span>c
	<span class="k">for </span>i <span class="k">in</span> <span class="si">$(</span><span class="nb">printf</span> <span class="s2">"%s"</span> <span class="s2">"</span><span class="nv">$1</span><span class="s2">"</span> | <span class="nb">sed</span> <span class="nt">-e</span> <span class="s1">'s/\(.\)/\1\n/g'</span><span class="si">)</span><span class="p">;</span> <span class="k">do
		if</span> <span class="o">[[</span> <span class="nt">-n</span> <span class="s2">"</span><span class="nv">$up</span><span class="s2">"</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then
			</span><span class="nv">c</span><span class="o">=</span><span class="si">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$i</span><span class="s2">"</span> | <span class="nb">tr</span> <span class="s1">'[:lower:]'</span> <span class="s1">'[:upper:]'</span><span class="si">)</span>
			<span class="nv">up</span><span class="o">=</span>
		<span class="k">else
			</span><span class="nv">c</span><span class="o">=</span><span class="si">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$i</span><span class="s2">"</span> | <span class="nb">tr</span> <span class="s1">'[:upper:]'</span> <span class="s1">'[:lower:]'</span><span class="si">)</span>
			<span class="nv">up</span><span class="o">=</span><span class="nb">yes
		</span><span class="k">fi
		</span><span class="nv">result</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">result</span><span class="k">}${</span><span class="nv">c</span><span class="k">}</span><span class="s2">"</span>
	<span class="k">done
	</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$result</span><span class="s2">"</span>
<span class="o">}</span>

<span class="c"># Print a message and fail</span>
die<span class="o">()</span> <span class="o">{</span>
	<span class="nb">echo</span> <span class="s2">"** FATAL: </span><span class="nv">$*</span><span class="s2">"</span> <span class="o">&gt;</span>&amp;2
	<span class="nb">exit </span>1
<span class="o">}</span>

<span class="c"># Emit info stuff</span>
info<span class="o">()</span> <span class="o">{</span>
	<span class="nb">echo</span> <span class="s2">"[ </span><span class="nv">$*</span><span class="s2"> ] ..."</span> <span class="o">&gt;</span>&amp;2
<span class="o">}</span>

<span class="c"># Rebuild logic; places errors in /tmp/errors</span>
<span class="c"># TODO: put your own (re)build logic here. It should build as much as possible, ignoring errors!</span>
rebuild<span class="o">()</span> <span class="o">{</span>
	<span class="c"># clean repository</span>
	git clean <span class="nt">-dxf</span>

	<span class="c"># rebuild (-k0 to continue despite of errors)</span>
	<span class="nb">rm</span> <span class="nt">-rf</span> <span class="s2">"build"</span> <span class="se">\</span>
		<span class="o">&amp;&amp;</span> <span class="nb">mkdir</span> <span class="nt">-p</span> <span class="s2">"build"</span> <span class="se">\</span>
		<span class="o">&amp;&amp;</span> <span class="o">(</span><span class="nb">cd</span> <span class="s2">"build"</span> <span class="o">&amp;&amp;</span> cmake <span class="nt">-GNinja</span> .<span class="o">)</span> <span class="se">\</span>
		<span class="o">&amp;&amp;</span> ninja <span class="nt">-C</span> <span class="s2">"build"</span> <span class="nt">-k0</span> <span class="se">\</span>
		| <span class="nb">tee</span> /tmp/errors
<span class="o">}</span>

<span class="c"># Code format logic, because we are worth it</span>
format<span class="o">()</span> <span class="o">{</span>
	<span class="c"># TODO: insert your code format logic here</span>
	info <span class="s2">"Cleanup clang-format"</span>
	git diff <span class="nt">-U0</span> <span class="nt">--no-prefix</span> <span class="nt">--no-color</span> | clang-format-diff-14 <span class="nt">-style</span> file <span class="nt">-i</span>
<span class="o">}</span>

<span class="c"># TODO: put your own logic here to add -ferror-limit=0 on the build</span>
info <span class="s2">"Set error limit to infinity"</span>
<span class="nb">sed</span> <span class="nt">-e</span> <span class="s1">'s/CFLAGS=/CFLAGS=-ferror-limit=0/'</span> <span class="nt">-i</span> CMakeLists.txt
git commit CMakeLists.txt <span class="nt">-m</span> <span class="s2">"chore: Add -ferror-limit=0"</span>

<span class="c"># Compute intermediate class names with mIxEd cAsE for line column stability</span>
<span class="nb">declare</span> <span class="nt">-A</span> intermediate
<span class="nv">srcAllowed</span><span class="o">=</span>
<span class="k">for </span>from <span class="k">in</span> <span class="s2">"</span><span class="k">${</span><span class="p">!replacements[@]</span><span class="k">}</span><span class="s2">"</span><span class="p">;</span> <span class="k">do
	</span><span class="nv">to</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">replacements</span><span class="p">[</span><span class="nv">$from</span><span class="p">]</span><span class="k">}</span><span class="s2">"</span>
	intermediate[<span class="nv">$from</span><span class="o">]=</span><span class="si">$(</span>altCase <span class="s2">"</span><span class="nv">$from</span><span class="s2">"</span><span class="si">)</span>
	<span class="k">if</span> <span class="o">[[</span> <span class="nt">-n</span> <span class="s2">"</span><span class="nv">$srcAllowed</span><span class="s2">"</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then
		</span><span class="nv">srcAllowed</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">srcAllowed</span><span class="k">}</span><span class="s2">|"</span>
	<span class="k">fi
	</span><span class="nv">srcAllowed</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">srcAllowed</span><span class="k">}${</span><span class="nv">from</span><span class="k">}</span><span class="s2">"</span>
<span class="k">done

</span>info <span class="s2">"Check we don't have hits on intermediate patterns"</span>
<span class="k">for </span>from <span class="k">in</span> <span class="s2">"</span><span class="k">${</span><span class="p">!intermediate[@]</span><span class="k">}</span><span class="s2">"</span><span class="p">;</span> <span class="k">do
	</span><span class="nv">to</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">intermediate</span><span class="p">[</span><span class="nv">$from</span><span class="p">]</span><span class="k">}</span><span class="s2">"</span>
	<span class="o">!</span> git <span class="nb">grep</span> <span class="nt">-q</span> <span class="s2">"</span><span class="nv">$to</span><span class="s2">"</span> <span class="o">||</span> die <span class="s2">"Oops: </span><span class="nv">$to</span><span class="s2"> did hit; please change the logic"</span>
<span class="k">done

</span>info <span class="s2">"Do all intermediate replacements in types"</span>
<span class="k">for </span>from <span class="k">in</span> <span class="s2">"</span><span class="k">${</span><span class="p">!intermediate[@]</span><span class="k">}</span><span class="s2">"</span><span class="p">;</span> <span class="k">do
	</span><span class="nv">to</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">intermediate</span><span class="p">[</span><span class="nv">$from</span><span class="p">]</span><span class="k">}</span><span class="s2">"</span>
	<span class="nb">echo</span> <span class="s2">"</span><span class="nv">$from</span><span class="s2"> → </span><span class="nv">$to</span><span class="s2">"</span>

	<span class="c"># Let's replace directly concerned files first except includes</span>
    <span class="c"># TODO: put your own logic here to specify the file(s) holding the types to be renamed to patch</span>
	<span class="k">for </span>i <span class="k">in</span> <span class="k">${</span><span class="nv">headerFiles</span><span class="p">[@]</span><span class="k">}</span><span class="p">;</span> <span class="k">do
		</span><span class="nb">sed</span> <span class="nt">-e</span> <span class="s2">"s/</span><span class="se">\b</span><span class="k">${</span><span class="nv">from</span><span class="k">}</span><span class="se">\b</span><span class="s2">/</span><span class="k">${</span><span class="nv">to</span><span class="k">}</span><span class="s2">/g"</span> <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$i</span><span class="s2">"</span>
		<span class="nb">sed</span> <span class="nt">-e</span> <span class="s2">"s%</span><span class="k">${</span><span class="nv">to</span><span class="k">}</span><span class="se">\.</span><span class="s2">h</span><span class="se">\"</span><span class="s2">%</span><span class="k">${</span><span class="nv">from</span><span class="k">}</span><span class="se">\.</span><span class="s2">h</span><span class="se">\"</span><span class="s2">%g"</span> <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$i</span><span class="s2">"</span>
	<span class="k">done
done

</span>info <span class="s2">"Do all intermediate forward declaration replacements"</span>
<span class="k">for </span>from <span class="k">in</span> <span class="s2">"</span><span class="k">${</span><span class="p">!intermediate[@]</span><span class="k">}</span><span class="s2">"</span><span class="p">;</span> <span class="k">do
	</span><span class="nv">to</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">intermediate</span><span class="p">[</span><span class="nv">$from</span><span class="p">]</span><span class="k">}</span><span class="s2">"</span>
	<span class="nb">echo</span> <span class="s2">"</span><span class="nv">$from</span><span class="s2"> → </span><span class="nv">$to</span><span class="s2">"</span>

	<span class="k">for </span>i <span class="k">in</span> <span class="si">$(</span>git <span class="nb">grep</span> <span class="nt">-E</span> <span class="s2">"^class </span><span class="k">${</span><span class="nv">from</span><span class="k">}</span><span class="s2">;"</span> | <span class="nb">cut</span> <span class="nt">-f1</span> <span class="nt">-d</span><span class="s1">':'</span> | <span class="nb">sort</span> <span class="nt">-u</span><span class="si">)</span><span class="p">;</span> <span class="k">do
		</span><span class="nb">sed</span> <span class="nt">-e</span> <span class="s2">"s/^class </span><span class="k">${</span><span class="nv">from</span><span class="k">}</span><span class="s2">;</span><span class="nv">$/</span><span class="s2">class </span><span class="k">${</span><span class="nv">to</span><span class="k">}</span><span class="s2">;/g"</span> <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$i</span><span class="s2">"</span>
	<span class="k">done
done</span>

<span class="c"># Let's loop until we converge without errors</span>
<span class="nv">step</span><span class="o">=</span>1
<span class="k">while </span><span class="nb">true</span><span class="p">;</span> <span class="k">do
	</span>info <span class="s2">"Rebuild #</span><span class="nv">$step</span><span class="s2">"</span>
	rebuild

	<span class="c"># Let the magic begin: fetch error information and replace exact pattern at the exact places</span>
	<span class="nv">errMatch</span><span class="o">=</span><span class="s2">"(unknown type name|use of undeclared identifier|no type named|no member named)"</span>
	<span class="nv">fromMacro</span><span class="o">=</span><span class="s2">"note: expanded from macro"</span>

	<span class="c"># First handle inlined macros. Trickier, as this is a multiline capture. (LANG=C because way faster)</span>
	<span class="c">#</span>
	<span class="c"># /code/library/utils/MyFancyTools.cpp:60:45: error: use of undeclared identifier 'UnicodeChar'</span>
	<span class="c">#     constexpr UnicodeChar replacementChar = UNICODE_REPLACEMENT_CHAR;</span>
	<span class="c">#                                             ^</span>
	<span class="c"># /code/library/unicode/unicode.h:24:36: note: expanded from macro 'UNICODE_REPLACEMENT_CHAR'</span>
	<span class="c">#</span>
	<span class="c"># -or- with nested macros:</span>
	<span class="c">#</span>
	<span class="c"># /code/test/MyFancyTests.cpp:72:9: error: unknown type name 'AString'</span>
	<span class="c">#         TEST_ENTITY("&amp;#12345;", 0x3039, false);</span>
	<span class="c">#         ^</span>
	<span class="c"># /code/test/MyFancyTests.cpp:64:48: note: expanded from macro 'TEST_ENTITY'</span>
	<span class="c"># #define TEST_ENTITY(name, value, onlyAlphaNum) _TEST_ENTITY(entities, name, value, onlyAlphaNum)</span>
	<span class="c">#                                                ^</span>
	<span class="c"># /code/test/MyFancyTests.cpp:48:9: note: expanded from macro '_TEST_ENTITY'</span>
	<span class="nv">LANG</span><span class="o">=</span>C <span class="se">\</span>
	<span class="nb">grep</span> <span class="nt">-aPzo</span> <span class="se">\</span>
		<span class="s2">"(?s)[^</span><span class="se">\n</span><span class="s2">]*: error: </span><span class="k">${</span><span class="nv">errMatch</span><span class="k">}</span><span class="s2"> '[A-Za-z0-9</span><span class="se">\.</span><span class="s2">_]*'[^</span><span class="se">\n</span><span class="s2">]*</span><span class="se">\n</span><span class="s2">([^</span><span class="se">\n</span><span class="s2">]*</span><span class="se">\n</span><span class="s2">[^</span><span class="se">\n</span><span class="s2">]*</span><span class="se">\n</span><span class="s2">)?[^</span><span class="se">\n</span><span class="s2">]*: </span><span class="k">${</span><span class="nv">fromMacro</span><span class="k">}</span><span class="s2">[^</span><span class="se">\n</span><span class="s2">]*(</span><span class="se">\n</span><span class="s2">([^</span><span class="se">\n</span><span class="s2">]*</span><span class="se">\n</span><span class="s2">[^</span><span class="se">\n</span><span class="s2">]*</span><span class="se">\n</span><span class="s2">)?[^</span><span class="se">\n</span><span class="s2">]*: </span><span class="k">${</span><span class="nv">fromMacro</span><span class="k">}</span><span class="s2">[^</span><span class="se">\n</span><span class="s2">]*)*"</span> <span class="se">\</span>
			/tmp/errors |
		<span class="nb">tr</span> <span class="s1">'\0'</span> <span class="s1">'\n'</span> |
		<span class="nb">grep</span> <span class="nt">-aE</span> <span class="s2">"(error: </span><span class="k">${</span><span class="nv">errMatch</span><span class="k">}</span><span class="s2">|</span><span class="k">${</span><span class="nv">fromMacro</span><span class="k">}</span><span class="s2">)"</span> |
		<span class="nb">sed</span> <span class="se">\</span>
			<span class="nt">-e</span> <span class="s2">"s/^.*error: [^']* '</span><span class="se">\(</span><span class="s2">[^']*</span><span class="se">\)</span><span class="s2">'[^</span><span class="se">\n</span><span class="s2">]*</span><span class="nv">$/</span><span class="s2">%</span><span class="se">\1</span><span class="s2">/"</span> <span class="se">\</span>
			<span class="nt">-e</span> <span class="s2">"s/^</span><span class="se">\(</span><span class="s2">[^:]*</span><span class="se">\)</span><span class="s2">:</span><span class="se">\(</span><span class="s2">[^:]*</span><span class="se">\)</span><span class="s2">:</span><span class="se">\(</span><span class="s2">[^:]*</span><span class="se">\)</span><span class="s2">: </span><span class="k">${</span><span class="nv">fromMacro</span><span class="k">}</span><span class="s2">.*/</span><span class="se">\1\t\2\t\3</span><span class="s2">/"</span> |
		<span class="nb">tr</span> <span class="s1">'\n%'</span> <span class="s1">'\t\n'</span> |
		<span class="nb">sed</span> <span class="nt">-E</span> <span class="s1">'s/^([^\t]*)\t([^\t]*\t[^\t]*\t[^\t]*\t)*([^\t]*)\t([^\t]*)\t([^\t]*)\t$/\3;\4;\5;\1/g'</span> |
		<span class="nb">grep</span> <span class="nt">-aE</span> <span class="s2">";</span><span class="k">${</span><span class="nv">srcAllowed</span><span class="k">}</span><span class="s2">;"</span> |
		<span class="nb">tr</span> <span class="s1">';'</span> <span class="s1">'\t'</span> |
		<span class="nb">grep</span> <span class="nt">-avE</span> <span class="s1">'^$'</span> |
		<span class="nb">sort</span> <span class="nt">-u</span> |
		<span class="k">while </span><span class="nb">read</span> <span class="nt">-r</span> elt<span class="p">;</span> <span class="k">do
			</span><span class="nv">file</span><span class="o">=</span><span class="si">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$elt</span><span class="s2">"</span> | <span class="nb">cut</span> <span class="nt">-f1</span><span class="si">)</span>
			<span class="nv">line</span><span class="o">=</span><span class="si">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$elt</span><span class="s2">"</span> | <span class="nb">cut</span> <span class="nt">-f2</span><span class="si">)</span>
			<span class="nv">col</span><span class="o">=</span><span class="si">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$elt</span><span class="s2">"</span> | <span class="nb">cut</span> <span class="nt">-f3</span><span class="si">)</span>
			<span class="nv">from</span><span class="o">=</span><span class="si">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$elt</span><span class="s2">"</span> | <span class="nb">cut</span> <span class="nt">-f4</span><span class="si">)</span>
			<span class="nv">to</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">intermediate</span><span class="p">[</span><span class="nv">$from</span><span class="p">]</span><span class="k">}</span><span class="s2">"</span> <span class="o">||</span> die <span class="s2">"Can't find </span><span class="nv">$from</span><span class="s2"> in intermediate"</span>
			<span class="nb">printf</span> <span class="s2">"</span><span class="se">\r\3</span><span class="s2">3[K</span><span class="se">\r</span><span class="s2">%s %s %s %s %s"</span> <span class="s2">"</span><span class="nv">$file</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$line</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$col</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$from</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$to</span><span class="s2">"</span>

			<span class="c"># Note: We need LANG=C as columns are apparently expressed in byte offset</span>
			<span class="nv">LANG</span><span class="o">=</span>C <span class="nb">sed</span> <span class="nt">-E</span> <span class="s2">"</span><span class="k">${</span><span class="nv">line</span><span class="k">}</span><span class="s2">s/^(.{</span><span class="k">$((</span>col-1<span class="k">))</span><span class="s2">})</span><span class="k">${</span><span class="nv">from</span><span class="k">}</span><span class="s2">/</span><span class="se">\1</span><span class="k">${</span><span class="nv">to</span><span class="k">}</span><span class="s2">/"</span> <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$file</span><span class="s2">"</span>
		<span class="k">done</span>

	<span class="c"># Then fetch direct code error information and replace exact pattern at the exact places</span>
	<span class="c">#</span>
	<span class="c"># /code/library/FooWriter.h:35:25: error: unknown type name 'MyFancyView'</span>
	<span class="c"># /code/library/FooWriter.h:431:21: error: use of undeclared identifier 'MyFancyView'</span>
	info <span class="s2">"Execute replacements"</span>
	<span class="nb">grep</span> <span class="nt">-a</span> <span class="s2">" error: "</span> /tmp/errors |
		<span class="nb">grep</span> <span class="nt">-aE</span> <span class="s2">"</span><span class="k">${</span><span class="nv">errMatch</span><span class="k">}</span><span class="s2">"</span> |
		<span class="nb">sed</span> <span class="nt">-e</span> <span class="s2">"s/^</span><span class="se">\(</span><span class="s2">[^:]*</span><span class="se">\)</span><span class="s2">:</span><span class="se">\(</span><span class="s2">[^:]*</span><span class="se">\)</span><span class="s2">:</span><span class="se">\(</span><span class="s2">[^:]*</span><span class="se">\)</span><span class="s2">:[^']*'</span><span class="se">\(</span><span class="s2">[^']*</span><span class="se">\)</span><span class="s2">'.*/</span><span class="se">\1\t\2\t\3\t\4</span><span class="s2">/"</span> |
		<span class="nb">grep</span> <span class="nt">-aE</span> <span class="s2">"</span><span class="se">\b</span><span class="s2">(</span><span class="k">${</span><span class="nv">srcAllowed</span><span class="k">}</span><span class="s2">)</span><span class="se">\b</span><span class="s2">"</span> |
		<span class="nb">sort</span> <span class="nt">-u</span> |
		<span class="k">while </span><span class="nb">read</span> <span class="nt">-r</span> elt<span class="p">;</span> <span class="k">do
			</span><span class="nv">file</span><span class="o">=</span><span class="si">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$elt</span><span class="s2">"</span> | <span class="nb">cut</span> <span class="nt">-f1</span><span class="si">)</span>
			<span class="nv">line</span><span class="o">=</span><span class="si">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$elt</span><span class="s2">"</span> | <span class="nb">cut</span> <span class="nt">-f2</span><span class="si">)</span>
			<span class="nv">col</span><span class="o">=</span><span class="si">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$elt</span><span class="s2">"</span> | <span class="nb">cut</span> <span class="nt">-f3</span><span class="si">)</span>
			<span class="nv">from</span><span class="o">=</span><span class="si">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$elt</span><span class="s2">"</span> | <span class="nb">cut</span> <span class="nt">-f4</span><span class="si">)</span>
			<span class="nv">to</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">intermediate</span><span class="p">[</span><span class="nv">$from</span><span class="p">]</span><span class="k">}</span><span class="s2">"</span> <span class="o">||</span> die <span class="s2">"Can't find </span><span class="nv">$from</span><span class="s2"> in intermediate"</span>
			<span class="nb">printf</span> <span class="s2">"</span><span class="se">\r\3</span><span class="s2">3[K</span><span class="se">\r</span><span class="s2">%s %s %s %s %s"</span> <span class="s2">"</span><span class="nv">$file</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$line</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$col</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$from</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$to</span><span class="s2">"</span>

			<span class="c"># Note: We need LANG=C as columns are apparently expressed in byte offset</span>
			<span class="nv">LANG</span><span class="o">=</span>C <span class="nb">sed</span> <span class="nt">-E</span> <span class="s2">"</span><span class="k">${</span><span class="nv">line</span><span class="k">}</span><span class="s2">s/^(.{</span><span class="k">$((</span>col-1<span class="k">))</span><span class="s2">})</span><span class="k">${</span><span class="nv">from</span><span class="k">}</span><span class="s2">/</span><span class="se">\1</span><span class="k">${</span><span class="nv">to</span><span class="k">}</span><span class="s2">/"</span> <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$file</span><span class="s2">"</span>
		<span class="k">done</span>

	<span class="c"># If no changes, bail out</span>
	<span class="k">if </span>git diff-index <span class="nt">--quiet</span> HEAD <span class="nt">--</span><span class="p">;</span> <span class="k">then
		</span><span class="nb">break
	</span><span class="k">fi</span>

	<span class="c"># Format code</span>
	format

	<span class="c"># Commit</span>
	<span class="k">if</span> <span class="o">[[</span> <span class="nv">$step</span> <span class="o">==</span> 1 <span class="o">]]</span><span class="p">;</span> <span class="k">then
		</span>info <span class="s2">"Commit result"</span>
		git commit <span class="nt">-a</span> <span class="nt">-m</span> <span class="s2">"chore: Automated replacements (</span><span class="nv">$0</span><span class="s2">)"</span>
	<span class="k">else
		</span>info <span class="s2">"Merging commit result"</span>
		git commit <span class="nt">-a</span> <span class="nt">--amend</span> <span class="nt">--no-edit</span>
	<span class="k">fi

	</span><span class="nv">step</span><span class="o">=</span><span class="k">$((</span>step+1<span class="k">))</span>
<span class="k">done

</span>info <span class="s2">"Final name replacements"</span>
<span class="k">for </span>key <span class="k">in</span> <span class="s2">"</span><span class="k">${</span><span class="p">!replacements[@]</span><span class="k">}</span><span class="s2">"</span><span class="p">;</span> <span class="k">do
	</span><span class="nv">from</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">intermediate</span><span class="p">[</span><span class="nv">$key</span><span class="p">]</span><span class="k">}</span><span class="s2">"</span>
	<span class="nv">to</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">replacements</span><span class="p">[</span><span class="nv">$key</span><span class="p">]</span><span class="k">}</span><span class="s2">"</span>
	<span class="nb">echo</span> <span class="s2">"</span><span class="nv">$from</span><span class="s2"> → </span><span class="nv">$to</span><span class="s2">"</span>

	<span class="k">for </span>i <span class="k">in</span> <span class="si">$(</span>git <span class="nb">grep</span> <span class="s2">"</span><span class="nv">$from</span><span class="s2">"</span> | <span class="nb">cut</span> <span class="nt">-f1</span> <span class="nt">-d</span><span class="s1">':'</span> | <span class="nb">sort</span> <span class="nt">-u</span><span class="si">)</span><span class="p">;</span> <span class="k">do
		</span><span class="nb">sed</span> <span class="nt">-e</span> <span class="s2">"s/</span><span class="se">\b</span><span class="k">${</span><span class="nv">from</span><span class="k">}</span><span class="se">\b</span><span class="s2">/</span><span class="nv">$to</span><span class="s2">/g"</span> <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$i</span><span class="s2">"</span>
	<span class="k">done
done

</span>info <span class="s2">"Remove idempotent types (using X=X)"</span>
<span class="k">for </span>from <span class="k">in</span> <span class="s2">"</span><span class="k">${</span><span class="p">!replacements[@]</span><span class="k">}</span><span class="s2">"</span><span class="p">;</span> <span class="k">do
	</span><span class="nv">to</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">replacements</span><span class="p">[</span><span class="nv">$from</span><span class="p">]</span><span class="k">}</span><span class="s2">"</span>
	<span class="nb">echo</span> <span class="s2">"</span><span class="nv">$from</span><span class="s2"> → </span><span class="nv">$to</span><span class="s2">"</span>

	<span class="k">for </span>i <span class="k">in</span> <span class="si">$(</span>git <span class="nb">grep</span> <span class="nt">-E</span> <span class="s2">"^using </span><span class="k">${</span><span class="nv">to</span><span class="k">}</span><span class="s2"> = </span><span class="k">${</span><span class="nv">to</span><span class="k">}</span><span class="s2">;"</span> | <span class="nb">cut</span> <span class="nt">-f1</span> <span class="nt">-d</span><span class="s1">':'</span><span class="si">)</span><span class="p">;</span> <span class="k">do
		</span><span class="nb">sed</span> <span class="nt">-E</span> <span class="s2">"/^using </span><span class="k">${</span><span class="nv">to</span><span class="k">}</span><span class="s2"> = </span><span class="k">${</span><span class="nv">to</span><span class="k">}</span><span class="s2">;/d"</span> <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$i</span><span class="s2">"</span>
	<span class="k">done
done</span>

<span class="c"># Format code</span>
format

info <span class="s2">"Merging commit result"</span>
git commit <span class="nt">-a</span> <span class="nt">--amend</span> <span class="nt">--no-edit</span>

info <span class="s2">"Final build"</span>
rebuild
</code></pre></div></div>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Redirecting Stdio is Hard]]></title>
    <link href="https://blog.httrack.com/blog/2014/12/20/redirecting-stdio-is-hard/"/>
    <updated>2014-12-20T14:04:00+01:00</updated>
    <id>https://blog.httrack.com/blog/2014/12/20/redirecting-stdio-is-hard</id>
    <content type="html"><![CDATA[<h2 id="the-two-nasty-bugs">The Two Nasty Bugs</h2>

<p><em>The nasty bugs</em></p>

<p>You know what it is like: you never saw them, but some people did, and these people are reliable, so there must be something wrong. But nobody on the engineering team managed to reproduce the issues. And the issues are random, and only happen rarely. <em>Very</em> rarely, such as once in a month for a given customer, or even once in a year for another one. And you know what: these bugs are horrible, because it seems you’ll never catch them. They hide, and silently trigger when you are asleep.</p>

<p><strong>First bug report</strong>: <em>Can not run an external crash reporter on Linux</em></p>

<p>This was the nicest one, actually, and it took only few days to investigate it. Basically, in case of crash, a <em>crash reporter</em> was supposed to be ran, and send useful information (stack traces et al.). Unfortunately the crash reporter appeared not to work correctly. It did not start. For an unknown reason (no error messages)</p>

<p>Fortunately, this bug was reproducible internally.</p>

<p><strong>Second bug report</strong>: <em>I see log files in my database on Windows</em></p>

<p>Yes, <em>log files in my database</em>. Or more precisely <em>log files in my database binary files</em>. I was a bit surprised, too. It seems that, randomly, some log files that were supposed to be sent to <em>the logs</em> were redirected to another random file, including database ones.</p>

<p>Obviously, the database was <em>not</em> happy after that.</p>

<p>That was the report after months of investigation: the first ones were just the usual <em>there was a random crash and I have no clue why - I scratched everything and now it works</em>.</p>

<p>These two issues were actually related, and the root cause was the way file descriptors are handled.</p>

<p>Oh, what are file descriptors, but he way ?</p>

<h2 id="the-three-little-descriptors">The Three Little Descriptors</h2>

<p>All C programmers know the three standard library input/output files: <code class="language-plaintext highlighter-rouge">stdin</code>, <code class="language-plaintext highlighter-rouge">stdout</code>, <code class="language-plaintext highlighter-rouge">stderr</code>. These are the default <em>standard</em> input, output, and error streams. The first one typically receives the console input (keyboard), the later ones allows to display output.</p>

<p>If you look at the <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/stdin.html">stdin</a> manual page, you will see that</p>
<blockquote>
  <p>A file with associated buffering is called a stream and is declared to be a pointer to a defined type FILE. (…) At program start-up, three streams shall be predefined and need not be opened explicitly: standard input (for reading conventional input), standard output (for writing conventional output), and standard error (for writing diagnostic output).</p>
</blockquote>

<p>Basically, these <code class="language-plaintext highlighter-rouge">FILE</code> streams are encapsulating a <em>file descriptor</em>, that is, a lower-level input or output resource (which is generally identified by an integer, ie. an <code class="language-plaintext highlighter-rouge">int</code>), and provide additional features, such as buffering, and offer a set of advanced functions, such as <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/fprintf.html"><code class="language-plaintext highlighter-rouge">fprintf</code></a> to print formatted output.</p>

<p>By default, the input/output streams are connected to the console (Linux, or DOS), but you may redirect them easily - this feature is especially known in the Linux/Un*x world:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">ls</span> <span class="nt">-l</span> /tmp/foo <span class="o">&gt;</span>/tmp/list 2&gt;/tmp/errorlog
</code></pre></div></div>

<p>In this very basic shell script example, the <code class="language-plaintext highlighter-rouge">ls -l /tmp/foo</code> command standard output is redirected to the file <code class="language-plaintext highlighter-rouge">/tmp/list</code>, and the standard error output is redirected to <code class="language-plaintext highlighter-rouge">/tmp/errorlog</code>. Note that the <code class="language-plaintext highlighter-rouge">&gt;/tmp/list</code> can also be written as <code class="language-plaintext highlighter-rouge">1&gt;/tmp/list</code>: the ‘1’ and ‘2’ refer to the file descriptors 1 and 2, which are the standard output and error file descriptors. The standard input is the file descriptor 0:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">read </span>A 0&lt;/tmp/bar
</code></pre></div></div>

<p>Why are the standard input/output streams identified by 0, 1 and 2 ? Well, this is just a convention - and, by default, file descriptors are allocated increasingly, and the first three files opened (this is done by the system) are these streams.</p>

<p>We can redirect the standard streams easily using shell commands - but can we do that <em>programmatically</em>, especially for the output streams ?</p>

<h2 id="redirection-to-a-file">Redirection to a File</h2>

<p>We have two output file descriptors (1 and 2), and we want to <em>capture</em> them in a file - say, a <em>log file</em>. This is quite convenient, especially for server applications: log files can then be rotated, inspected…</p>

<p>The standard way to reassign a file descriptor is through the <a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/dup.html"><code class="language-plaintext highlighter-rouge">dup2</code></a> standard library function. This function takes an opened file descriptor, and clone it to the second file descriptor provided. You can also extract the file descriptor beneath a <code class="language-plaintext highlighter-rouge">FILE</code> stream through the <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/fileno.html"><code class="language-plaintext highlighter-rouge">fileno</code></a> standard library function: here’s a simple code to redirect the standard output to a file:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#include</span> <span class="cpf">&lt;stdio.h&gt;</span><span class="cp">
#include</span> <span class="cpf">&lt;stdlib.h&gt;</span><span class="cp">
#include</span> <span class="cpf">&lt;unistd.h&gt;</span><span class="cp">
#include</span> <span class="cpf">&lt;assert.h&gt;</span><span class="cp">
</span>
<span class="cm">/** Redirect stdout and stderr. Returns 0 upon success. **/</span>
<span class="k">static</span> <span class="kt">int</span> <span class="nf">redirect_std_out_err</span><span class="p">(</span><span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">log_name</span><span class="p">)</span> <span class="p">{</span>
  <span class="kt">int</span> <span class="n">failure</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>

  <span class="cm">/* Open log FILE stream. */</span>
  <span class="kt">FILE</span> <span class="o">*</span><span class="k">const</span> <span class="n">fp</span> <span class="o">=</span> <span class="n">fopen</span><span class="p">(</span><span class="n">log_name</span><span class="p">,</span> <span class="s">"wb"</span><span class="p">);</span>
  <span class="k">if</span> <span class="p">(</span><span class="n">fp</span> <span class="o">!=</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>

    <span class="cm">/* Extract log FILE fd, and standard stdout/stderr ones. */</span>
    <span class="k">const</span> <span class="kt">int</span> <span class="n">log_fd</span> <span class="o">=</span> <span class="n">fileno</span><span class="p">(</span><span class="n">fp</span><span class="p">);</span>
    <span class="k">const</span> <span class="kt">int</span> <span class="n">stdout_fd</span> <span class="o">=</span> <span class="n">fileno</span><span class="p">(</span><span class="n">stdout</span><span class="p">);</span>
    <span class="k">const</span> <span class="kt">int</span> <span class="n">stderr_fd</span> <span class="o">=</span> <span class="n">fileno</span><span class="p">(</span><span class="n">stderr</span><span class="p">);</span>
    <span class="k">if</span> <span class="p">(</span><span class="n">log_fd</span> <span class="o">!=</span> <span class="o">-</span><span class="mi">1</span> <span class="o">&amp;&amp;</span> <span class="n">stdout_fd</span> <span class="o">!=</span> <span class="o">-</span><span class="mi">1</span> <span class="o">&amp;&amp;</span> <span class="n">stderr_fd</span> <span class="o">!=</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>

      <span class="cm">/* Clone log fd to standard stdout/stderr ones. */</span>
      <span class="k">if</span> <span class="p">(</span><span class="n">dup2</span><span class="p">(</span><span class="n">log_fd</span><span class="p">,</span> <span class="n">stdout_fd</span><span class="p">)</span> <span class="o">!=</span> <span class="o">-</span><span class="mi">1</span> <span class="o">&amp;&amp;</span> <span class="n">dup2</span><span class="p">(</span><span class="n">log_fd</span><span class="p">,</span> <span class="n">stderr_fd</span><span class="p">)</span> <span class="o">!=</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">failure</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
      <span class="p">}</span>
    <span class="p">}</span>

    <span class="cm">/* Close initial file. We do not need it, as we cloned it. */</span>
    <span class="n">fclose</span><span class="p">(</span><span class="n">fp</span><span class="p">);</span>
  <span class="p">}</span>
  <span class="k">return</span> <span class="n">failure</span><span class="p">;</span>
<span class="p">}</span>

<span class="kt">int</span> <span class="nf">main</span><span class="p">(</span><span class="kt">int</span> <span class="n">argc</span><span class="p">,</span> <span class="kt">char</span> <span class="o">**</span><span class="n">argv</span><span class="p">)</span> <span class="p">{</span>
  <span class="cm">/* Redirect stdout and stderr to a single log file */</span>
  <span class="k">if</span> <span class="p">(</span><span class="n">argc</span> <span class="o">!=</span> <span class="mi">2</span> <span class="o">||</span> <span class="n">redirect_std_out_err</span><span class="p">(</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]))</span> <span class="p">{</span>
    <span class="k">return</span> <span class="n">EXIT_FAILURE</span><span class="p">;</span>
  <span class="p">}</span>

  <span class="cm">/* Print output to both streams */</span>
  <span class="n">printf</span><span class="p">(</span><span class="s">"Hey, stdout is redirected!</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
  <span class="n">fprintf</span><span class="p">(</span><span class="n">stderr</span><span class="p">,</span> <span class="s">"Hey, stderr is also redirected!</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>

  <span class="k">return</span> <span class="n">EXIT_SUCCESS</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>

<p>And that’s it! Simple, isn’t it ?</p>

<h2 id="here-comes-the-problems">Here Comes The Problems</h2>

<p>Redirection is what we typically do in my company - I even modified some of the involved code. But sending output to a log file has its drawbacks. Especially regarding log file size: the file is growing, and we need to clear it time to time, or the disk will eventually fill. This is generally done through a “rotate” operation, that is:</p>

<ul>
  <li>closing the current log file</li>
  <li>renaming it with some timestamp</li>
  <li>opening a fresh one</li>
  <li>cleanup old logs if necessary</li>
</ul>

<p>File descriptors, more generally, are special resources in most operating systems, as they are <em>inherited</em> when spawning a process. By default, all file descriptors are transmitted to spawned processes (on POSIX systems, <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html"><code class="language-plaintext highlighter-rouge">fork</code></a> clone them, and <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html"><code class="language-plaintext highlighter-rouge">exec</code></a> functions keep them opened by default), and this is a pain: pipes and sockets are also inherited. This may lead to the following case:</p>

<ul>
  <li>main program opens a socket to a remote location</li>
  <li>spawning a process, which inherits this socket (file descriptor) too</li>
  <li>main program sens commands to remote location and closes the socket</li>
  <li>spawned program still lives, and still has the socket file descriptor copy opened</li>
  <li>remote location does not “sees” the client close() and is stuck</li>
</ul>

<p>The thing is that spawned processed do not give a damn about already opened file descriptors (except for stdio). But the standard is broken for historical reasons, very unfortunately.</p>

<p>The solution is painful: you need to flag all possible opened file descriptors with the <a href="http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/fcntl.h.html"><code class="language-plaintext highlighter-rouge">FD_CLOEXEC</code></a> flag set. This flag explicitly tells the system that the file descriptor <em>shall not be inherited</em> when spawning a process:</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">open(..., ...)</code> =&gt; <code class="language-plaintext highlighter-rouge">open(..., ... | O_CLOEXEC)</code></li>
  <li><code class="language-plaintext highlighter-rouge">fopen(..., ...)</code> =&gt; <code class="language-plaintext highlighter-rouge">open(..., ... | O_CLOEXEC)</code> + <code class="language-plaintext highlighter-rouge">fdopen(...)</code> OR <code class="language-plaintext highlighter-rouge">fopen(..., ... + "e")</code></li>
  <li><code class="language-plaintext highlighter-rouge">dup(...)</code> =&gt; <code class="language-plaintext highlighter-rouge">fcntl(..., F_DUPFD_CLOEXEC, (long) 0)</code></li>
  <li><code class="language-plaintext highlighter-rouge">dup2(..., ...)</code> =&gt; <code class="language-plaintext highlighter-rouge">dup3(..., ..., O_CLOEXEC)</code></li>
  <li><code class="language-plaintext highlighter-rouge">opendir(...)</code> =&gt; <code class="language-plaintext highlighter-rouge">open(..., O_RDONLY | O_DIRECTORY | O_CLOEXEC)</code> + <code class="language-plaintext highlighter-rouge">fdopendir(...)</code></li>
  <li><code class="language-plaintext highlighter-rouge">pipe(...)</code> =&gt; <code class="language-plaintext highlighter-rouge">pipe2(..., O_CLOEXEC)</code></li>
  <li><code class="language-plaintext highlighter-rouge">acept(..., ..., ...)</code> =&gt; <code class="language-plaintext highlighter-rouge">accept4(..., ..., ..., O_CLOEXEC)</code></li>
  <li><code class="language-plaintext highlighter-rouge">socket(..., ..., ...)</code> =&gt; <code class="language-plaintext highlighter-rouge">socket(..., ... | SOCK_CLOEXEC, ...)</code></li>
  <li>and so on…</li>
</ul>

<p>Some of these functions are non-standard (such as <a href="http://linux.die.net/man/2/accept4"><code class="language-plaintext highlighter-rouge">accept4</code></a>) unfortunately. Some other will not be available (Windows). Some old kernels (Linux) will sometimes not support these features - and some old glibc will not work either (especially for the “e” flag). Yes, this is a real pain.</p>

<p>Oh, one last thing: do not dare to use <code class="language-plaintext highlighter-rouge">fcntl(fd, F_SETFD, ...)</code> to clear the <code class="language-plaintext highlighter-rouge">FD_CLOEXEC</code> flag: this is insanely hopeless in a multithreaded environment. Can you guess <em>why</em> ?</p>

<h2 id="so-what-was-that-all-about-">So What Was That All About ?</h2>

<p>Okay, back yo our nasty bugs!</p>

<ol>
  <li>The first one was quite easy to investigate.</li>
</ol>

<p>Here are the investigation steps, ordered temporally:</p>

<ul>
  <li>“can not run an external crash reporter on Linux”; ie. can not <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html"><code class="language-plaintext highlighter-rouge">fork()</code></a>/<a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html"><code class="language-plaintext highlighter-rouge">execv()</code></a> the external crash reporter in a signal handler, with a status code returned by <a href="http://pubs.opengroup.org/onlinepubs/7908799/xsh/wait.html"><code class="language-plaintext highlighter-rouge">waitpid()</code></a> reporting a <em>signal 6 caught during launch</em> (ie. the launched program called <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/abort.html"><code class="language-plaintext highlighter-rouge">abort()</code></a>)</li>
  <li>we can not run the crash reporter <em>outside</em> the signal handler either</li>
  <li>we can not run the crash reporter using <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/system.html"><code class="language-plaintext highlighter-rouge">system()</code></a> either</li>
  <li>we can not run “/bin/echo foo” through <code class="language-plaintext highlighter-rouge">system()</code> either; the command returns the status “return code 1”!</li>
  <li>… BUT we CAN run “/bin/echo -n” through <code class="language-plaintext highlighter-rouge">system()</code>!</li>
</ul>

<p>At this stage, it is clear that the issue is related to the <em>output file descriptors</em>.</p>

<p>Further investigation showed that:</p>

<ul>
  <li>we redirected output to logs, opening the log file with the FD_CLOEXEC flag set, AND using the Linux-specific <a href="http://linux.die.net/man/2/dup3"><code class="language-plaintext highlighter-rouge">dup3()</code></a> call</li>
  <li>as a result or <code class="language-plaintext highlighter-rouge">dup3()</code>, standard streams had their FD_CLOEXEC flag set</li>
  <li>spawning an external program led to have the standard streams closed</li>
  <li>attempting to write on the closed standard streams led to either a <em>broken pipe</em> signal, or an assertion failure depending on the program involved</li>
</ul>

<p>The solution was simply to use the default <code class="language-plaintext highlighter-rouge">dup2()</code> function for <em>standard streams</em>. These are the only streams we want to be inherited, after all!</p>

<ol>
  <li>The second bug took a bit more time</li>
</ol>

<p>The biggest issue is that the bug could not be reproduced at first internally. Until our Q&amp;A team managed to reproduce it (these guys are really good at finding bugs, I must admit) three days in a row.</p>

<p>The logs reports were strange, and totally unrelated. But one log was kind of interesting:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[2014/11/28-10:59:08.514] [info] ...
[2014/11/28-10:59:08.659] [info] ...
[Fatal Error] :2:999961: XML document structures must start and end within the same entity.
</code></pre></div></div>

<p>The error was a corrupted produced XML document, with another unrelated log inside it, in the middle of the structure. The final error did not have any timestamp, but the previous one was <em>just before midnight</em> (GMT). Humm, a midnight bug ? What are the chances for this being a coincidence ?</p>

<p>I then realized that midnight was the trigger time for a number of maintenance operations, including… log rotation.</p>

<p>But after thoughtfully inspecting the code, everything looked fine. And the bug never popped up on Linux, it was <em>Windows-specific</em> (or at least, much frequent on this operating system)</p>

<p>But there was another clue: all weird logs were actually logs coming from the Java virtual machine (either through <a href="http://logging.apache.org/log4j/">Log4j</a>, or through <code class="language-plaintext highlighter-rouge">System.out.println()</code> calls)</p>

<p>When using <code class="language-plaintext highlighter-rouge">System.out</code> (or <code class="language-plaintext highlighter-rouge">System.err</code>) in Java, we are actually using a <a href="http://docs.oracle.com/javase/7/docs/api/java/io/PrintStream.html"><code class="language-plaintext highlighter-rouge">PrintStream</code></a>, which is an object wrapped in various <a href="http://docs.oracle.com/javase/7/docs/api/java/io/FilterOutputStream.html"><code class="language-plaintext highlighter-rouge">FilterOutputStream</code></a> objects for buffering, the final instance being a <a href="http://docs.oracle.com/javase/7/docs/api/java/io/FileOutputStream.html"><code class="language-plaintext highlighter-rouge">FileOutputStream</code></a>, which is, according to the documentation an:</p>

<blockquote>
  <p>output stream for writing data to a File or to a FileDescriptor</p>
</blockquote>

<p>And, in our case, the <a href="http://docs.oracle.com/javase/7/docs/api/java/io/FileDescriptor.html"><code class="language-plaintext highlighter-rouge">FileDescriptor</code></a> involved was <code class="language-plaintext highlighter-rouge">FileDescriptor.out</code> and <code class="language-plaintext highlighter-rouge">FileDescriptor.err</code>, that is, <em>Java handles to the standard output and error streams</em>.</p>

<p>Fortunately, the sources of the <a href="http://openjdk.java.net/">Open JDK</a> are public, and it is possible to dig into the <em>real</em> code beneath, that is, the <em>native code</em>:</p>

<ul>
  <li>./jdk/src/share/classes/java/io/FileOutputStream.java</li>
  <li>./jdk/src/windows/native/java/io/FileOutputStream_md.c</li>
  <li>./jdk/src/solaris/native/java/io/FileOutputStream_md.c</li>
</ul>

<p>Hummm… there is a Windows-specific implementation for FileOutputStream native calls. strange, isn’t it ?</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">JNIEXPORT</span> <span class="kt">void</span> <span class="n">JNICALL</span>
<span class="nf">Java_java_io_FileOutputStream_write</span><span class="p">(</span><span class="n">JNIEnv</span> <span class="o">*</span><span class="n">env</span><span class="p">,</span> <span class="n">jobject</span> <span class="n">this</span><span class="p">,</span> <span class="n">jint</span> <span class="n">byte</span><span class="p">,</span> <span class="n">jboolean</span> <span class="n">append</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">writeSingle</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">this</span><span class="p">,</span> <span class="n">byte</span><span class="p">,</span> <span class="n">append</span><span class="p">,</span> <span class="n">fos_fd</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>

<p>And the <code class="language-plaintext highlighter-rouge">writeSingle</code> call is using a file descriptor returned by <code class="language-plaintext highlighter-rouge">GET_FD()</code>:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">void</span>
<span class="nf">writeSingle</span><span class="p">(</span><span class="n">JNIEnv</span> <span class="o">*</span><span class="n">env</span><span class="p">,</span> <span class="n">jobject</span> <span class="n">this</span><span class="p">,</span> <span class="n">jint</span> <span class="n">byte</span><span class="p">,</span> <span class="n">jboolean</span> <span class="n">append</span><span class="p">,</span> <span class="n">jfieldID</span> <span class="n">fid</span><span class="p">)</span> <span class="p">{</span>
    <span class="c1">// Discard the 24 high-order bits of byte. See OutputStream#write(int)</span>
    <span class="kt">char</span> <span class="n">c</span> <span class="o">=</span> <span class="p">(</span><span class="kt">char</span><span class="p">)</span> <span class="n">byte</span><span class="p">;</span>
    <span class="n">jint</span> <span class="n">n</span><span class="p">;</span>
    <span class="n">FD</span> <span class="n">fd</span> <span class="o">=</span> <span class="n">GET_FD</span><span class="p">(</span><span class="n">this</span><span class="p">,</span> <span class="n">fid</span><span class="p">);</span>
</code></pre></div></div>

<p>And this macro just calls <code class="language-plaintext highlighter-rouge">GetObjectField(..., IO_handle_fdID)</code> to retrieve … the “handle” member of the corresponding FileDescriptor class.</p>

<p><strong>What</strong> ? A <em>handle</em> ? Not a <em>file descriptor</em> ?</p>

<p>Yep: a <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/ms724457%28v=vs.85%29.aspx"><code class="language-plaintext highlighter-rouge">HANDLE</code></a> - and this was the root cause of all out problems.</p>

<p>Let me give you the reason: on Windows, a standard library handle is actually a kernel <code class="language-plaintext highlighter-rouge">HANDLE</code> wrapped by the C library, nothing more.</p>

<p>It means that if you replace the standard library input/output streams, you’ll just replace the <em>C library</em> streams.</p>

<p>Here is the complete bug scenario on Windows after collecting all the clues:</p>

<ul>
  <li>log rotation function is called at low level, and replaces the standard output and error streams 1 and 2 using <code class="language-plaintext highlighter-rouge">dup2()</code></li>
  <li>the Visual C++ C library <em>replaces</em> the streams, by closing the underlying existing <code class="language-plaintext highlighter-rouge">HANDLE</code>, and replacing by <em>new ones</em></li>
  <li>the old <code class="language-plaintext highlighter-rouge">HANDLE</code> are now totally meaningless (“dangling handles”)</li>
  <li>Java has no knowledge of the replacement, and still has the <em>old</em> <code class="language-plaintext highlighter-rouge">HANDLE</code> stored in its <code class="language-plaintext highlighter-rouge">FileDescriptor</code> internals</li>
  <li>Java <code class="language-plaintext highlighter-rouge">System.out</code> calls are now sending data to nowhere (this is bad)</li>
  <li>someone opens a new file, and Windows recycle the <em>old</em> <code class="language-plaintext highlighter-rouge">HANDLE</code></li>
  <li>Java <code class="language-plaintext highlighter-rouge">System.out</code> calls are now sending data to <strong>a valid – yet unrelated – file</strong> (this is very very bad)</li>
  <li>data corruption ensues (and not profit)</li>
</ul>

<p>The root cause is that <em>Java is unaware that the low-level <code class="language-plaintext highlighter-rouge">HANDLE</code> has changed</em>.</p>

<p>The solution is simple: let’s tell him!:</p>

<p>For each <code class="language-plaintext highlighter-rouge">System.out</code> and <code class="language-plaintext highlighter-rouge">System.err</code> <code class="language-plaintext highlighter-rouge">OutputStream</code> instances, we need to:</p>

<ul>
  <li>get the “<code class="language-plaintext highlighter-rouge">out</code>” member field of the <code class="language-plaintext highlighter-rouge">FilterOutputStream</code> class and mark it public (see <code class="language-plaintext highlighter-rouge">Field.setAccessible()</code>)</li>
  <li>unwrap all <code class="language-plaintext highlighter-rouge">FilterOutputStream</code> wrapping by calling fetching the “<code class="language-plaintext highlighter-rouge">out</code>” member field recursively</li>
  <li>cast the resulting instance to a <code class="language-plaintext highlighter-rouge">FileOutputStream</code></li>
  <li>fetch the <code class="language-plaintext highlighter-rouge">FileDescriptor</code> through <code class="language-plaintext highlighter-rouge">FileOutputStream.getFD()</code></li>
  <li>use <code class="language-plaintext highlighter-rouge">SharedSecrets.getJavaIOFileDescriptorAccess()</code> to set the <code class="language-plaintext highlighter-rouge">HANDLE</code> (a <code class="language-plaintext highlighter-rouge">long</code>) value to -1 temporarily</li>
  <li>rotate the logs</li>
  <li>fetch the new <code class="language-plaintext highlighter-rouge">HANDLE</code> (a <code class="language-plaintext highlighter-rouge">long</code>) through a native call to <a href="http://msdn.microsoft.com/en-us/library/ks2530z6.aspx"><code class="language-plaintext highlighter-rouge">_get_osfhandle()</code></a></li>
  <li>set the new <code class="language-plaintext highlighter-rouge">HANDLE</code> using the previous steps</li>
</ul>

<p>This is a bit cumbersome - but mixing native code and Java is <em>always</em> cumbersome, so you have to be used to it :)</p>

<h2 id="epilogue">Epilogue</h2>

<p>The two issues were rather different:</p>

<ul>
  <li>one was a mistake created when attempting to fix the insane default handle inheritance on both POSIX and Windows API, causing external programs not to run anymore</li>
  <li>one was a design bug between the Visual C++ library and the JVM (ie. there is no <code class="language-plaintext highlighter-rouge">dup2()</code> for HANDLEs), causing random corruptions on files or sockets</li>
</ul>

<p>But in both cases, a simple standard stream mishandling was the root cause. But the consequences were nasty, the resolution painful, and the time spent unreasonable (the second bug took months to be investigated)</p>

<p><strong>TL;DR</strong>: redirecting Stdio is Hard, and prone to nasty bugs if you are not careful enough!</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[I Found A Bug In strncat()]]></title>
    <link href="https://blog.httrack.com/blog/2014/08/31/i-found-a-bug-in-strncat/"/>
    <updated>2014-08-31T09:55:00+02:00</updated>
    <id>https://blog.httrack.com/blog/2014/08/31/i-found-a-bug-in-strncat</id>
    <content type="html"><![CDATA[<h2 id="yes-a-bug">Yes, A Bug</h2>

<p>This code won’t behave correctly (ie. partial copy) on many glibc versions on Linux:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">strncat</span><span class="p">(</span><span class="n">dest</span><span class="p">,</span> <span class="n">source</span><span class="p">,</span> <span class="n">SIZE_MAX</span><span class="p">);</span>
</code></pre></div></div>

<p>This is a corner case, but this is a bug.</p>

<p>(hint: the correct behavior is to do <code class="language-plaintext highlighter-rouge">strcat(dest, source)</code> equivalent)</p>

<h2 id="a-secure-version-of-strcat">A Secure Version Of Strcat</h2>

<p>It began with code cleanup.</p>

<p>In httrack, there is a lot of legacy (old, dirty) code laying around. Rewriting everything from scratch would probably be a sane decision, but I just don’t have the time (and the courage) to do that, so I’m cleaning up bits and bits when possible, attempting to improve the code quality when possible.</p>

<p>One of the terrible design decision was to use C strings everywhere, and associated <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/strcpy.html"><code class="language-plaintext highlighter-rouge">strcpy</code></a>, <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/strcat.html"><code class="language-plaintext highlighter-rouge">strcat</code></a> etc. primitives. It was an unfortunate decision, because of the lack of flexibility (immutable capacity), and security issues (buffer overflows that might be introduced).</p>

<p>In this hypothetical case, we’re adding a string to a structure foo:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">struct</span> <span class="n">foo</span> <span class="p">{</span>
  <span class="kt">char</span> <span class="n">filename</span><span class="p">[</span><span class="mi">1024</span><span class="p">];</span>
<span class="p">...</span>
<span class="p">};</span>

<span class="kt">void</span> <span class="nf">append_filename</span><span class="p">(</span><span class="k">struct</span> <span class="n">foo</span> <span class="o">*</span><span class="n">foo</span><span class="p">,</span> <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">filename</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">strcat</span><span class="p">(</span><span class="n">foo</span><span class="o">-&gt;</span><span class="n">filename</span><span class="p">,</span> <span class="n">filename</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>

<p>This code will overflow if the passed string <em>added to</em> the existing string overflows the <code class="language-plaintext highlighter-rouge">filename</code> buffer.</p>

<p>The <a href="http://www.openbsd.org/cgi-bin/man.cgi?query=strlcat">strlcat</a> version exists on BSD, and is a reasonable solution to mitigate this problem. Unfortunately, <a href="https://www.sourceware.org/ml/libc-alpha/2007-05/msg00028.html">some people</a> do not like BSD, and decided that these functions will never be part of the glibc. (insert flame here)</p>

<p>Besides, switching to strlcat and its friends would involve checking every single occurrence of strcat, strcpy. etc., probably introducing new bugs!</p>

<p>I then decided to use a modified version of the <em>dangerous</em> libc primitives, using compile-time type checking:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/**
 * Check whether 'VAR' is of type char[].
 */</span>
<span class="cp">#if (defined(__GNUC__) &amp;&amp; !defined(__cplusplus))
</span><span class="cm">/* Note: char[] and const char[] are compatible */</span>
<span class="cp">#define HTS_IS_CHAR_BUFFER(VAR) ( __builtin_types_compatible_p ( typeof (VAR), char[] ) )
#else
</span><span class="cm">/* Note: a bit lame as char[8] won't be seen. */</span>
<span class="cp">#define HTS_IS_CHAR_BUFFER(VAR) ( sizeof(VAR) != sizeof(char*) )
#endif
#define HTS_IS_NOT_CHAR_BUFFER(VAR) ( ! HTS_IS_CHAR_BUFFER(VAR) )
</span>
<span class="cm">/**
 * Append at most N characters from "B" to "A".
 * If "A" is a char[] variable then the size is assumed to be the capacity of this array.
 */</span>
<span class="cp">#define strncatbuff(A, B, N) \
  ( HTS_IS_NOT_CHAR_BUFFER(A) \
  ? strncat(A, B, N) \
  : strncat_safe_(A, sizeof(A), B, \
  HTS_IS_NOT_CHAR_BUFFER(B) ? (size_t) -1 : sizeof(B), N, \
  "overflow while appending '" #B "' to '"#A"'", __FILE__, __LINE__) )
</span></code></pre></div></div>

<p>The idea was to <em>sniff</em> at compile-time the arguments passed to the modified <code class="language-plaintext highlighter-rouge">strncat</code> version: when an opaque <code class="language-plaintext highlighter-rouge">char*</code> is used, there’s nothing you can do. But when passing a regular <code class="language-plaintext highlighter-rouge">char[...]</code> array, a secure version can be used. This is not perfect (especially because with non-GCC compilers, char[8] are seen as char*), but at least it allows you to quickly mitigate many potential buffer overflows without touching existing code.</p>

<p>But at one point I did this quite daring thing to create the secure strcat version:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#define strcatbuff(A, B) strncatbuff(A, B, (size_t) -1)
</span></code></pre></div></div>

<p>The <code class="language-plaintext highlighter-rouge">(size_t) -1</code> value was simply passed as the boundary of <code class="language-plaintext highlighter-rouge">strncatbuff</code> - at some point, we would be doing <code class="language-plaintext highlighter-rouge">strncat(dest, source, (size_t) -1)</code>, and this seemed fine to me.</p>

<p>Soon after, <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1123625">bugs</a> started to appear (strings were copied partially). It took me a bit of time (with the kind help of the bug reporter) to figure out that my daring macro was involved in this bug. This took me a while because the bug only appeared on Linux, on some glibc releases, and quite randomly.</p>

<p>I managed to have a reproducible minimalistic case:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/* strncat_size_t-1_testcase. */</span>

<span class="cm">/* gcc -Wall strncat_size_t-1_testcase.c -o strncat_size_t-1_testcase */</span>
<span class="cp">#include</span> <span class="cpf">&lt;stdio.h&gt;</span><span class="cp">
#include</span> <span class="cpf">&lt;stdlib.h&gt;</span><span class="cp">
#include</span> <span class="cpf">&lt;stdint.h&gt;</span><span class="cp">
#include</span> <span class="cpf">&lt;string.h&gt;</span><span class="cp">
#include</span> <span class="cpf">&lt;assert.h&gt;</span><span class="cp">
</span>
<span class="cm">/* strncat_size_t-1_testcase "Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor" */</span>
<span class="kt">int</span> <span class="nf">main</span><span class="p">(</span><span class="kt">int</span> <span class="n">argc</span><span class="p">,</span> <span class="kt">char</span> <span class="o">**</span><span class="n">argv</span><span class="p">)</span> <span class="p">{</span>
  <span class="k">const</span> <span class="kt">size_t</span> <span class="n">size</span> <span class="o">=</span> <span class="n">strlen</span><span class="p">(</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]);</span>
  <span class="kt">char</span> <span class="o">*</span><span class="k">const</span> <span class="n">buffer</span> <span class="o">=</span> <span class="n">malloc</span><span class="p">(</span><span class="n">size</span> <span class="o">+</span> <span class="mi">1</span><span class="p">);</span>
  <span class="kt">int</span> <span class="n">success</span><span class="p">;</span>
  <span class="n">buffer</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="sc">'\0'</span><span class="p">;</span>
  <span class="n">strncat</span><span class="p">(</span><span class="n">buffer</span><span class="p">,</span> <span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">SIZE_MAX</span><span class="p">);</span>
  <span class="n">success</span> <span class="o">=</span> <span class="n">strlen</span><span class="p">(</span><span class="n">buffer</span><span class="p">)</span> <span class="o">==</span> <span class="n">size</span><span class="p">;</span>
  <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">success</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">fprintf</span><span class="p">(</span><span class="n">stderr</span><span class="p">,</span> <span class="s">"** result: '%s'</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">buffer</span><span class="p">);</span>
  <span class="p">}</span>
  <span class="n">assert</span><span class="p">(</span><span class="n">success</span><span class="p">);</span>
  <span class="k">return</span> <span class="n">success</span> <span class="o">?</span> <span class="n">EXIT_SUCCESS</span> <span class="o">:</span> <span class="n">EXIT_FAILURE</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>

<p>If you build this program, and use it:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcc -Wall strncat_size_t-1_testcase.c -o strncat_size_t-1_testcase
./strncat_size_t-1_testcase "Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor"
</code></pre></div></div>

<p>… then you’ll get an error. Or maybe you’ll get one. Randomly.</p>

<h2 id="this-is-not-a-glibc-bug-but-a-misuse-of-strncat">This Is Not A glibc Bug, But A Misuse Of Strncat</h2>

<p>The only way to be sure was to carefully read the <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/strncat.html"><code class="language-plaintext highlighter-rouge">strncat</code></a> standard.</p>

<blockquote>
  <p>The strncat() function shall append not more than n bytes (a null byte and bytes that follow it are not appended) from the array pointed to by s2 to the end of the string pointed to by s1. The initial byte of s2 overwrites the null byte at the end of s1. A terminating null byte is always appended to the result. If copying takes place between objects that overlap, the behavior is undefined.</p>
</blockquote>

<p>With the <a href="https://groups.google.com/forum/#!topic/comp.unix.programmer/qKMC4A_itLs">help</a> of specialists in <code class="language-plaintext highlighter-rouge">comp.unix.programmer</code>, several points must be noticed:</p>

<ul>
  <li>strncat is <em>not</em> a secure version of strcat (strlcat is), and the third argument is <em>not</em> the destination capacity</li>
  <li>using (size_t) -1 as third argument is actually using <a href="http://pubs.opengroup.org/onlinepubs/009695399/basedefs/stdint.h.html">SIZE_MAX</a></li>
  <li>the third argument is an <em>additional</em> limit and can have any value</li>
</ul>

<p>Therefore, <code class="language-plaintext highlighter-rouge">strncat(..., ..., SIZE_MAX)</code> <em>should</em> behave as <code class="language-plaintext highlighter-rouge">strcat(..., ...)</code>.</p>

<p>Yes, yes, this is a corner case. But this is a <em>legit</em> one.</p>

<h2 id="where-is-the-bug-">Where Is The Bug ?</h2>

<p>The reason why not all glibc releases were impacted is simple: this is an <em>optimization</em> bug. (insert laughs from experienced developers)</p>

<p>The <a href="https://sourceware.org/git/?p=glibc.git;a=blob;f=string/strncat.c;h=7ac44561bdfa8a82267a9fea73c0cd7d6e2a12df;hb=HEAD">original strncat</a> source is not used on x86-64 architectures: the <a href="https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86_64/multiarch/strcat-sse2-unaligned.S;h=dc782f2c2370915f74673ff8184e5f2eaf2795db;hb=HEAD">optimized SSE2</a> version is used in place.</p>

<p>The code might have been introduced in 2013 <a href="https://sourceware.org/git/?p=glibc.git;a=commit;f=sysdeps/x86_64/multiarch/strcat-sse2-unaligned.S;h=37bb363f03d75e5e6f2ca45f2c686a3a0167797e">in this commit</a> which was <em>supposed</em> to introduce a “Faster strlen on x64” - but some existing files seems to have been merged at that time.</p>

<p>Unfortunately, the assembly source is uncommented, and I did not have the courage to dig deeper. But this source is buggy.</p>

<p><strong>Funny Remark</strong>: By the way, I’m not even convinced that the assembly version is <em>faster</em> at all: code executing speed is probably irrelevant compared to L1/L2/L3/memory bandwidth issues.</p>

<h2 id="whats-next-">What’S Next ?</h2>

<p>First I obviously stopped using SIZE_MAX with strncat, and dispatch to strcat accordingly. This was a daring thing to use this corner case value anyway, prone to … corner-case bugs.</p>

<p>The second step was to <a href="https://sourceware.org/bugzilla/show_bug.cgi?id=17279">fill a bug report</a> in the glibc.</p>

<p><strong>TL;DR</strong>: Beware when using corner cases, even when perfectly conforming to the standard. And never “optimize” code by rewriting obvious functions into a 300-lines of undocumented assembly!</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Coucal, Cuckoo-hashing-based hashtable with stash area C library]]></title>
    <link href="https://blog.httrack.com/blog/2014/06/14/coucal/"/>
    <updated>2014-06-14T13:30:00+02:00</updated>
    <id>https://blog.httrack.com/blog/2014/06/14/coucal</id>
    <content type="html"><![CDATA[<p><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/8/8d/Greater_Coucal_%28Centropus_sinensis%29_in_Hyderabad_W_IMG_8962.jpg/250px-Greater_Coucal_%28Centropus_sinensis%29_in_Hyderabad_W_IMG_8962.jpg" alt="Greater Coucal Centropus sinensis" title="Yes, that's me! Centropus sinensis" /></p>

<h2 id="coucal-">Coucal ?</h2>

<p>According to <a href="https://en.wikipedia.org/wiki/Coucal">Wikipedia</a>,</p>
<blockquote>
  <p>A coucal is one of about 30 species of birds in the cuckoo family. All of them belong in the subfamily Centropodinae and the genus Centropus. Unlike many Old World cuckoos, coucals are not brood parasites.</p>
</blockquote>

<p>If you read my <a href="http://blog.httrack.com/blog/2013/07/12/so-i-made-a-hashtable/">previous blog entry</a> on <a href="http://www.httrack.com/">HTTrack</a>’s new hashtables, you probably already know that <em>cuckoo hashing</em> is a rather new (2001) hashing technique to be used for efficient hashtables.</p>

<p>The idea is to have a smarter <a href="https://en.wikipedia.org/wiki/Open_addressing"><em>open-addressing</em></a> strategy based on <em>kicking</em> an existing entry if a collision occurs. This is why the name <em>Cuckoo</em> hashtable was chosen.</p>

<p>Note that the name is a bit misleading: a typical cuckoo will not only <em>kick</em> an existing foe, but will consequently <em>kill it</em> (nature, you scary!). Here, we only <em>relocate</em> it, reason why I choose the <strong>Coucal</strong>, a nicer cuckoo member who does not destroy existing entries - I mean, existing eggs!</p>

<p>This implementation has the following features:</p>

<ul>
  <li>guaranteed constant time (Θ(1)) lookup</li>
  <li>guaranteed constant time (Θ(1)) delete or replace</li>
  <li>average constant time (O(1)) insert</li>
  <li>one large memory chunk for table (and one for the key pool)</li>
  <li>simple enumeration</li>
</ul>

<p>This implementation has been tested since July 2013 in HTTrack, and so far, no bugs were reported (the only issues spotted were bugs in upstream code that was corrupting existing keys)</p>

<p>Please have a look at this library implementation: <a href="https://github.com/xroche/coucal">https://github.com/xroche/coucal</a></p>

<p><strong>TL;DR</strong>: cuckoo hashtables are both efficient and smart!</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[C Corner Cases (and Funny Things)]]></title>
    <link href="https://blog.httrack.com/blog/2014/05/30/c-corner-cases-and-funny-things/"/>
    <updated>2014-05-30T11:20:00+02:00</updated>
    <id>https://blog.httrack.com/blog/2014/05/30/c-corner-cases-and-funny-things</id>
    <content type="html"><![CDATA[<h2 id="enter-the-void">Enter The <code class="language-plaintext highlighter-rouge">void</code></h2>

<p><code class="language-plaintext highlighter-rouge">void</code> is a very strange animal. Basically <em>void</em> means <em>nothing</em>, <em>emptiness</em>. A function “returning” void actually <em>does not return anything</em>. On the machine code side, register(s) aimed to hold return values are simply unused in such case.</p>

<p>You won’t be surprised to hear that the <em>size of void</em> is <strong>undefined</strong> for this reason ; ie. <code class="language-plaintext highlighter-rouge">sizeof(void)</code> is not supposed to exist, even if some compilers have rather dirty extensions that allow to set this size to 1 to do pointer arithmetics.</p>

<p>But what about <code class="language-plaintext highlighter-rouge">void*</code> ? A pointer to something that does not exist, what’s that ?</p>

<p>Well, this is the first trap: <code class="language-plaintext highlighter-rouge">void*</code> is not really related to <code class="language-plaintext highlighter-rouge">void</code>. The <code class="language-plaintext highlighter-rouge">void*</code> type is used to store addresses of unspecified types ; ie. you can cast from/to the <code class="language-plaintext highlighter-rouge">void*</code> type, for example when passing generic functions (such as <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/read.html"><code class="language-plaintext highlighter-rouge">read()</code></a>, <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/write.html"><code class="language-plaintext highlighter-rouge">write()</code></a>, <a href="http://pubs.opengroup.org/onlinepubs/7908799/xsh/memset.html"><code class="language-plaintext highlighter-rouge">memset()</code></a>..)</p>

<p>When I say <em>cast</em>, I don’t really mean cast, actually. You can view the pointer to <code class="language-plaintext highlighter-rouge">void</code> as a kind of a supertype, and aliasing rules do not apply <em>per se</em>: casting <code class="language-plaintext highlighter-rouge">int*</code> to <code class="language-plaintext highlighter-rouge">void*</code> is legit and vice-versa. But casting the resulting <code class="language-plaintext highlighter-rouge">void*</code> pointer to <code class="language-plaintext highlighter-rouge">long*</code>, for example, is <strong>not</strong>, because aliasing rules would be violated by indirectly casting from <code class="language-plaintext highlighter-rouge">int*</code> to <code class="language-plaintext highlighter-rouge">long*</code>. Yes, aliasing rules are a whole lot of fun. We’ll see that later.</p>

<h2 id="functions-are-not-data">Functions Are Not Data</h2>

<p>Here comes another trap: you can cast anything from/to <code class="language-plaintext highlighter-rouge">void*</code>, except <em>pointer to functions</em>. Yep, in C, pointers to data and functions are not <em>necessarily</em> if the same size. Basically this means that the address space dedicated to data (data/bss segments) and the one dedicated to code (text segment) are not <em>necessarily</em> of the same size (you could have a 32-bit code pointers and 64-bit data pointers for example). Of course this is <em>almost always</em> the case, because the universe would otherwise collapse, and <em>standard</em> functions such as <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/dlsym.html"><code class="language-plaintext highlighter-rouge">dlsym</code></a> would not work anymore.</p>

<p>To quote the holy standard:</p>

<blockquote>
  <p>The ISO C standard does not require that pointers to functions can be cast back and forth to pointers to data. Indeed, the ISO C standard does not require that an object of type void * can hold a pointer to a function. Implementations supporting the XSI extension, however, do require that an object of type void * can hold a pointer to a function.</p>
</blockquote>

<h2 id="function-pointer-definition">Function Pointer Definition</h2>

<p>Oh, yes.</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  <span class="kt">int</span> <span class="p">(</span><span class="o">*</span><span class="n">function</span><span class="p">)(</span><span class="kt">char</span><span class="o">*</span> <span class="n">s</span><span class="p">,</span> <span class="kt">int</span> <span class="n">c</span><span class="p">)</span> <span class="o">=</span> <span class="n">my_function</span><span class="p">;</span>
</code></pre></div></div>

<p>Function definitions are sometimes see as <em>tricky</em> by newcomers. They are NOT. Simply replace <code class="language-plaintext highlighter-rouge">(*function)</code> on the left member (the parenthesis are there for operator priority reasons) by a function name, and you just have the classic function declaration.</p>

<h2 id="is-char-nice-">Is <code class="language-plaintext highlighter-rouge">char</code> Nice ?</h2>

<p>In comparison to <code class="language-plaintext highlighter-rouge">void*</code> and function pointers, <code class="language-plaintext highlighter-rouge">char</code> is the nice guy:</p>

<ul>
  <li>its size is 1 <em>by definition</em>, which makes the pointer variant suitable for <em>pointer arithmetics</em></li>
  <li>it can be also cast from and to any other type without violating aliasing rules (still <em>per se</em>, re-casting to a third incompatible type would still violate aliasing rules)</li>
  <li>it can be dereferenced after a cast (ie. you can actually <em>reads the bytes within an <code class="language-plaintext highlighter-rouge">int*</code> safely</em>)</li>
</ul>

<h2 id="but-char-is-generally-signed">But <code class="language-plaintext highlighter-rouge">char</code> Is Generally Signed</h2>

<p>Yes, <em>generally</em>, because this is not specified in the standard. This is actually annoying, because when handling strings, you may end up with <em>negative</em> characters for ASCII &gt; 127, and negative values can be promoted to integers.</p>

<p>Here’s a buggy example:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/** Return the next character within a \0-terminated string, or EOF. **/</span>
<span class="kt">int</span> <span class="nf">my_read_char</span><span class="p">(</span><span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">buffer</span><span class="p">,</span> <span class="kt">size_t</span> <span class="o">*</span><span class="n">offs</span><span class="p">)</span> <span class="p">{</span>
  <span class="k">if</span> <span class="p">(</span><span class="n">buffer</span><span class="p">[</span><span class="o">*</span><span class="n">offs</span><span class="p">]</span> <span class="o">!=</span> <span class="sc">'\0'</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">return</span> <span class="n">buffer</span><span class="p">[</span><span class="o">*</span><span class="n">offs</span><span class="o">++</span><span class="p">];</span>  <span class="cm">/* here's the trap */</span>
  <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
    <span class="k">return</span> <span class="n">EOF</span><span class="p">;</span>
  <span class="p">}</span>
</code></pre></div></div>

<p>This function will return negative values for ASCII &gt; 127, and <em>especially</em> the value <strong>-1</strong> for ASCII 255 (0xFF), which is <em>also</em> the value of EOF.</p>

<p>You need to explicitly cast to a unsigned char to have a working version:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/** Return the next character within a \0-terminated string, or EOF. **/</span>
<span class="kt">int</span> <span class="nf">my_read_char</span><span class="p">(</span><span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">buffer</span><span class="p">,</span> <span class="kt">size_t</span> <span class="o">*</span><span class="n">offs</span><span class="p">)</span> <span class="p">{</span>
  <span class="k">if</span> <span class="p">(</span><span class="n">buffer</span><span class="p">[</span><span class="o">*</span><span class="n">offs</span><span class="p">]</span> <span class="o">!=</span> <span class="sc">'\0'</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">return</span> <span class="p">(</span><span class="kt">unsigned</span> <span class="kt">char</span><span class="p">)</span> <span class="n">buffer</span><span class="p">[</span><span class="o">*</span><span class="n">offs</span><span class="o">++</span><span class="p">];</span>
  <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
    <span class="k">return</span> <span class="n">EOF</span><span class="p">;</span>
  <span class="p">}</span>
</code></pre></div></div>

<h2 id="you-may-use-t-for-const-t-but-not-t-for-const-t">You May Use <code class="language-plaintext highlighter-rouge">T*</code> for <code class="language-plaintext highlighter-rouge">const T*</code>, But Not <code class="language-plaintext highlighter-rouge">T**</code> for <code class="language-plaintext highlighter-rouge">const T**</code></h2>

<p>The typical example is that:</p>

<ul>
  <li>a <code class="language-plaintext highlighter-rouge">char*</code> string may be implicitly cast into <code class="language-plaintext highlighter-rouge">const char*</code>:
    <div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  <span class="kt">char</span> <span class="o">*</span><span class="n">foo</span> <span class="o">=</span> <span class="n">strdup</span><span class="p">(</span><span class="s">"Hello, world!"</span><span class="p">);</span>
  <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">bar</span> <span class="o">=</span> <span class="n">foo</span><span class="p">;</span>  <span class="cm">/* No need for a dirty cast. */</span>
</code></pre></div>    </div>
  </li>
</ul>

<p>.. but a <code class="language-plaintext highlighter-rouge">char**</code> array of string may NOT be implicitly cast into <code class="language-plaintext highlighter-rouge">const char**</code>:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  <span class="kt">char</span> <span class="o">*</span><span class="n">foo</span><span class="p">[</span><span class="mi">2</span><span class="p">];</span>
  <span class="n">foo</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="n">foo</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">strdup</span><span class="p">(</span><span class="s">"Hello, world!"</span><span class="p">);</span>
  <span class="k">const</span> <span class="kt">char</span> <span class="o">**</span><span class="n">bar</span> <span class="o">=</span> <span class="n">foo</span><span class="p">;</span>  <span class="cm">/* Warning. */</span>
</code></pre></div></div>

<p>The compiler will complaint:</p>

<blockquote>
  <p>1.c:6:22: warning: initialization from incompatible pointer type [enabled by default]
   const char <em>*bar = foo;  /</em> Warning. */</p>
</blockquote>

<p>The reason behind of the same reason why, in Java, you can not cast <code class="language-plaintext highlighter-rouge">List&lt;String&gt;</code> into <code class="language-plaintext highlighter-rouge">List&lt;Object&gt;</code>: a function taking the supertype array version <em>may</em> add an incompatible type without being noticed.</p>

<p>Here’s an example:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">static</span> <span class="kt">void</span> <span class="nf">replace_the_first_element</span><span class="p">(</span><span class="k">const</span> <span class="kt">char</span> <span class="o">**</span><span class="n">array</span><span class="p">)</span> <span class="p">{</span>
  <span class="k">static</span> <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">hello</span> <span class="o">=</span> <span class="s">"hello, world!"</span><span class="p">;</span>
  <span class="n">array</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="n">hello</span><span class="p">;</span>  <span class="cm">/* replace first element by a constant string */</span>
<span class="p">}</span>
<span class="p">...</span>
  <span class="kt">char</span> <span class="o">*</span><span class="n">foo</span><span class="p">[</span><span class="mi">2</span><span class="p">];</span>
  <span class="n">foo</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="n">foo</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">strdup</span><span class="p">(</span><span class="s">"Hello, world!"</span><span class="p">);</span>
  <span class="n">replace_the_first_element</span><span class="p">(</span><span class="n">foo</span><span class="p">);</span>
</code></pre></div></div>

<p>In this example, without a cast warning, the function <code class="language-plaintext highlighter-rouge">replace_the_first_element</code> would silently replace the first element of foo by a <em>non-constant</em> string, and the caller would have a constant string within the array when the function returns. The code is strictly equivalent to:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  <span class="k">static</span> <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">hello</span> <span class="o">=</span> <span class="s">"hello, world!"</span><span class="p">;</span>
  <span class="kt">char</span> <span class="o">*</span><span class="n">foo</span><span class="p">[</span><span class="mi">2</span><span class="p">];</span>
  <span class="n">foo</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="n">hello</span><span class="p">;</span>  <span class="cm">/* now you can see the problem */</span>
</code></pre></div></div>

<p>I would say that we <em>should</em> be able to silently cast <code class="language-plaintext highlighter-rouge">T**</code> into <code class="language-plaintext highlighter-rouge">const T*const*</code> (an array of constant arrays of T), but anyway …</p>

<h2 id="an-array-is-not-always-an-array">An Array Is Not Always An Array</h2>

<p>Quizz time: what the <code class="language-plaintext highlighter-rouge">test</code> function below is supposed to print ? (<em>don’t cheat!</em>)</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">static</span> <span class="kt">void</span> <span class="nf">test</span><span class="p">(</span><span class="kt">char</span> <span class="n">foo</span><span class="p">[</span><span class="mi">42</span><span class="p">])</span> <span class="p">{</span>
  <span class="kt">char</span> <span class="n">bar</span><span class="p">[</span><span class="mi">42</span><span class="p">];</span>
  <span class="n">printf</span><span class="p">(</span><span class="s">"foo: %zu bytes</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">foo</span><span class="p">));</span>
  <span class="n">printf</span><span class="p">(</span><span class="s">"bar: %zu bytes</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">bar</span><span class="p">));</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Well, you might be surprised to see:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>foo: 8 bytes
bar: 42 bytes
</code></pre></div></div>

<p>Yes, an <em>array of type <code class="language-plaintext highlighter-rouge">T</code></em> in a function argument list is <em>strictly</em> equivalent of declaring the type as <code class="language-plaintext highlighter-rouge">T*</code>. This is why nobody uses this confusing syntax within function argument list.</p>

<p>To quote the standard:</p>

<blockquote>
  <p>“A declaration of a parameter as ‘array of type’ shall be adjusted to ‘qualified pointer to type’, where the type qualifiers (if any) are those specified within the [ and ] of the array type derivation.”</p>
</blockquote>

<p>This also imply that the passed object is passed by <em>reference</em>, not <em>value</em>.</p>

<p>… and this is totally inconsistent with the handling of structures as function argument, or as return type. Yes, yes, historical mistakes.</p>

<h2 id="switch-and-case-are-hacks">Switch And Case Are Hacks</h2>

<p><code class="language-plaintext highlighter-rouge">switch</code> can be seen as a dispatch function, and <code class="language-plaintext highlighter-rouge">case</code> as label. Each <code class="language-plaintext highlighter-rouge">switch</code> must have its case, but you are free to interleave loops of you want, such as in:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">switch</span> <span class="p">(</span><span class="n">step</span><span class="p">)</span> <span class="p">{</span>
  <span class="k">while</span> <span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">case</span> <span class="n">step_A</span><span class="p">:</span>
    <span class="k">case</span> <span class="n">step_B</span><span class="p">:</span>
      <span class="p">...</span>
   <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p>See my previous <a href="http://blog.httrack.com/blog/2013/09/16/an-unusual-case-of-case-slash-switch/"><em>An Unusual Case of Case (/switch)</em></a> entry for more fun!</p>

<h2 id="pointer-to-arrays">Pointer To Arrays</h2>

<p>Let’s say I have a <code class="language-plaintext highlighter-rouge">char foo[42]</code> and I want a pointer to this array in <code class="language-plaintext highlighter-rouge">bar</code>. I can write:</p>

<ul>
  <li>bar = foo</li>
  <li>bar = &amp;foo</li>
  <li>bar = &amp;foo[0]</li>
</ul>

<p>What shall I use ? The first and third pointers are identical: they are <code class="language-plaintext highlighter-rouge">char*</code> ones, the one you generally want to use: the pointer points to a char area of undefined size. The second one is a pointer to the array of 42 chars itself. You may then use:</p>

<ul>
  <li>char *bar = foo;</li>
  <li>char (*bar)[42] = &amp;foo;</li>
  <li>char *bar = &amp;foo[0];</li>
</ul>

<p>The <em>pointer to the array of 42 chars</em> syntax is a bit weird for newcomers, but it is logical: <code class="language-plaintext highlighter-rouge">*bar</code> is an array of 42 chars, so you just have to replace <code class="language-plaintext highlighter-rouge">foo</code> by <code class="language-plaintext highlighter-rouge">(*bar)</code> in the standard definition to have the correct syntax (the parenthesis are needed for operators priority reasons, otherwise you’ll declare an array of 42 pointer to char). (This trick is also helpful to understand the apparent obscure <em>pointer to function</em> syntax.)</p>

<p>The advantage of the second definition is that you <em>still have a pointer to an array of 42 char</em>: <code class="language-plaintext highlighter-rouge">sizeof(*bar)</code> is 42, <code class="language-plaintext highlighter-rouge">*bar</code> is of type <code class="language-plaintext highlighter-rouge">char[]</code> (suitable for specific compiler optimizations and safe string operations)</p>

<h2 id="empty-structure">Empty Structure</h2>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">struct</span> <span class="p">{</span> <span class="p">}</span> <span class="n">foo</span><span class="p">;</span>
</code></pre></div></div>

<p>this code is <em>undefined</em> in C: an empty structure is not officially supported. Some compilers such as GCC support them, and define the size of the structure as 0.</p>

<p>It is supported in C++ (because many classes <em>do not have any member within</em>) <strong>and</strong> in such case the <code class="language-plaintext highlighter-rouge">sizeof</code> of such empty structure is … 1, because C++ <em>needs a different pointer for every instance of a given class</em>. Oh, and because you need to do pointer arithmetics also, and having a null size would cause issues.</p>

<h2 id="strings-are-char-arrays-and-are-not-const-but-they-really-are">Strings Are <code class="language-plaintext highlighter-rouge">char</code> Arrays, And Are Not Const (But They Really Are)</h2>

<p>Another corner case: string literals, such as <code class="language-plaintext highlighter-rouge">"hello world"</code> are actually arrays of <code class="language-plaintext highlighter-rouge">char</code>, with a terminating <code class="language-plaintext highlighter-rouge">\0</code>. The two definitions are equivalent:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">char</span> <span class="o">*</span><span class="n">foo</span> <span class="o">=</span> <span class="s">"hi"</span><span class="p">;</span>
</code></pre></div></div>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">static</span> <span class="k">const</span> <span class="kt">char</span> <span class="n">hi</span><span class="p">[]</span> <span class="o">=</span> <span class="p">{</span> <span class="sc">'h'</span><span class="p">,</span> <span class="sc">'i'</span><span class="p">,</span> <span class="sc">'\0'</span> <span class="p">};</span>
<span class="kt">char</span> <span class="o">*</span><span class="n">foo</span> <span class="o">=</span> <span class="p">(</span><span class="kt">char</span><span class="o">*</span><span class="p">)</span> <span class="n">hi</span><span class="p">;</span>
</code></pre></div></div>

<p>Note that <code class="language-plaintext highlighter-rouge">sizeof("hi")</code> is equal to 3 (an array of 3 chars).</p>

<p>Oh, did you notice the <code class="language-plaintext highlighter-rouge">(char*)</code> in the second example ? Isn’t it a bit dirty ?</p>

<ul>
  <li>yes, it is</li>
  <li>in C, string literals are <strong>unfortunately</strong> of type “array of char” and <strong>not</strong> “array of const char”</li>
  <li>in reality, the type is <strong>really</strong> “array of const char”: if you dare to write inside a string literal, chances are you’ll segfault, because the string literal is inside a global <em>read-only</em> data segment</li>
</ul>

<h2 id="parenthesis-have-sometimes-different-meanings">Parenthesis Have Sometimes Different Meanings</h2>

<p>Let’s have a look at two code samples:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="n">j</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="n">foo</span><span class="p">(</span><span class="n">i</span><span class="o">++</span><span class="p">,</span> <span class="n">j</span><span class="o">+=</span><span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="o">++</span><span class="p">);</span>
</code></pre></div></div>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="n">j</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="n">bar</span><span class="p">((</span><span class="n">i</span><span class="o">++</span><span class="p">,</span> <span class="n">j</span><span class="o">+=</span><span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="o">++</span><span class="p">));</span>
</code></pre></div></div>

<p>The first sample calls a function named <code class="language-plaintext highlighter-rouge">foo</code> taking three integers as arguments. Is has an <em>undefined behavior</em>: side-effects of <code class="language-plaintext highlighter-rouge">i</code> and <code class="language-plaintext highlighter-rouge">j</code> will be executed at an undefined time, and might even be optimized. You basically don’t have a clue of what <code class="language-plaintext highlighter-rouge">foo</code> will receive as arguments.</p>

<p>The second sample calls a function named <code class="language-plaintext highlighter-rouge">bar</code> taking <strong>one</strong> integer argument. It has a perfectly defined behavior. This integer is the result of the <a href="https://en.wikipedia.org/wiki/Comma_operator"><code class="language-plaintext highlighter-rouge">comma operator</code></a> <code class="language-plaintext highlighter-rouge">i++, j+=i, j++</code>, which can be rewritten as the pseudo-C++-code:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">static</span> <span class="kt">int</span> <span class="nf">my_comma</span><span class="p">(</span><span class="kt">int</span> <span class="o">&amp;</span><span class="n">i</span><span class="p">,</span> <span class="kt">int</span> <span class="o">&amp;</span><span class="n">j</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">i</span><span class="o">++</span><span class="p">;</span>
  <span class="n">j</span><span class="o">+=</span><span class="n">i</span><span class="p">;</span>
  <span class="k">return</span> <span class="n">j</span><span class="o">++</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">...</span>

<span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="n">j</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="n">bar</span><span class="p">(</span><span class="n">my_comma</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">));</span>
</code></pre></div></div>

<p>The comma operator evaluates each expression starting from the leftmost member to the rightmost member, and yields the last member as value. All other values are discarded (each expression should yield <code class="language-plaintext highlighter-rouge">void</code>, actually). And between each member evaluation, a <a href="https://en.wikipedia.org/wiki/Sequence_point">sequence point</a> is committed: side-effects of each expression are committed for the next expression. This is <em>why this expression has a perfectly defined behavior</em>.</p>

<h2 id="logical-andor-can-replace-if">Logical And/Or Can Replace <code class="language-plaintext highlighter-rouge">If</code></h2>

<p>Two pretty nice feature of the “boolean and” (<code class="language-plaintext highlighter-rouge">&amp;&amp;</code>) and the “boolean or” (<code class="language-plaintext highlighter-rouge">||</code>) operators is that they:</p>

<ul>
  <li>commit side-effects between each operation: <code class="language-plaintext highlighter-rouge">if (foo[i++] != '\0' &amp;&amp; bar[i++] != '\0')</code> is perfectly defined, as the post-increment of i will be committed before the evaluation of the right expression (ie. a <a href="https://en.wikipedia.org/wiki/Sequence_point">sequence point</a> exists between left and right operator evaluation)</li>
  <li>these operators are <a href="https://en.wikipedia.org/wiki/Short-circuit_evaluation">short-circuit</a> operators, which means that there is a <strong>guarantee</strong> that the right member is <em>only</em> evaluated if the result of the left member evaluation could not provide the result of the expression ; which allows to write something like <code class="language-plaintext highlighter-rouge">if (foo != NULL &amp;&amp; *foo != '\0')</code> without fearing of a NULL pointer dereferencing</li>
</ul>

<h2 id="beware-of-char-buffer1024--">Beware of <code class="language-plaintext highlighter-rouge">char buffer[1024] = ""</code></h2>

<p>Not only is it a bit dirty to have large buffers on the stack (I must admit that some legacy code in httrack is filled with that, cough cough), but the C standard ensure that when initializing a structure or an array, <em>missing</em> elements shall be initialized to <em>zero</em>.</p>

<p>This basically means here that the first element of the array is <em>explicitly</em> initialized to zero (this is the terminating \0 of the empty string), and the 1023 other elements will be <em>implicitly</em> initialized to zero, too. The performance impact is the same as <code class="language-plaintext highlighter-rouge">memset(buffer, 0, sizeof(buffer))</code>.</p>

<p>Use at least an explicit <code class="language-plaintext highlighter-rouge">buffer[0] = '\0'</code> without any initializer when declaring the buffer.</p>

<h2 id="the-bogus-strchr-and-friends-definitions">The Bogus <code class="language-plaintext highlighter-rouge">strchr</code> (and Friends) Definitions</h2>

<p>Did you notice that ?</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">char</span> <span class="o">*</span><span class="nf">strchr</span><span class="p">(</span><span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">s</span><span class="p">,</span> <span class="kt">int</span> <span class="n">c</span><span class="p">);</span>
<span class="kt">char</span> <span class="o">*</span><span class="nf">strstr</span><span class="p">(</span><span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">haystack</span><span class="p">,</span> <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">needle</span><span class="p">);</span>
<span class="p">...</span>
</code></pre></div></div>

<p>All these function takes a <code class="language-plaintext highlighter-rouge">const char*</code> and return a <code class="language-plaintext highlighter-rouge">char*</code>. How can a function transform an array of constant chars to a non-constant one ?</p>

<p>Well, obviously this is impossible, and the only reason of this oddity was to offer a somewhat generic function that would work for both <code class="language-plaintext highlighter-rouge">char*</code> and <code class="language-plaintext highlighter-rouge">const char*</code> strings: accepting <code class="language-plaintext highlighter-rouge">const char*</code> as argument will <em>also</em> accept <code class="language-plaintext highlighter-rouge">char*</code> versions by silently casting it, and returning <code class="language-plaintext highlighter-rouge">char*</code> will <em>also</em> allow to store the result in a <code class="language-plaintext highlighter-rouge">const char*</code> for the same reason.</p>

<p>Unfortunately, this may lead to:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">char</span> <span class="o">*</span><span class="n">pos</span> <span class="o">=</span> <span class="n">strchr</span><span class="p">(</span><span class="n">my_const_string</span><span class="p">,</span> <span class="sc">' '</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">pos</span> <span class="o">!=</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
  <span class="o">*</span><span class="n">pos</span><span class="o">++</span> <span class="o">=</span> <span class="sc">'\0'</span><span class="p">;</span>  <span class="cm">/* oops, I forgot that the string was constant, and the compiler did not warn me! */</span>
  <span class="cm">/* kaboom here */</span>
<span class="p">}</span> 
</code></pre></div></div>

<p>A sane decision would be to split all these string handling functions into const and non-const ones. In C++, you can have specialized versions.</p>

<h2 id="never-divide-by-zero--or-divide-int_min-by--1-after-midnight">Never Divide By Zero .. Or Divide INT_MIN by -1 After Midnight</h2>

<p>Dividing an integer by zero leads to a <em>floating point exception</em> (Yes, this is not a floating point operation, but the error triggers one.).</p>

<p>But dividing the smallest signed integer by -1 <em>also</em> leads to the same <em>floating point exception</em>. This is a bit of a curiosity: when <em>dividing</em> by -1 the smallest integer (<code class="language-plaintext highlighter-rouge">-2147483648</code>), the result <em>should</em> be <code class="language-plaintext highlighter-rouge">2147483648</code>, but is <em>too large</em> (by one) to be represented in an integer (the largest integer is <code class="language-plaintext highlighter-rouge">2147483647</code>), and the divide operation then triggers the same exception that the one triggered by the <em>division by zero</em>.</p>

<p>However, <em>multiplying</em> the smallest signed integer by -1 <em>does not trigger anything</em> because… well, because the <em>multiply</em> operation <em>does not raises anything</em>, even when overflowing.</p>

<p>Therefore, the first printf below will gently print a result, but the second one won’t, and the program will abort:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/* Note: using volatile to prevent the div to be optimized out */</span>
<span class="k">volatile</span> <span class="kt">int</span> <span class="n">int_min</span> <span class="o">=</span> <span class="n">INT_MIN</span><span class="p">;</span>
<span class="k">volatile</span> <span class="kt">int</span> <span class="n">minus_1</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span><span class="p">;</span>
<span class="kt">int</span> <span class="nf">main</span><span class="p">(</span><span class="kt">int</span> <span class="n">argc</span><span class="p">,</span> <span class="kt">char</span> <span class="o">**</span><span class="n">argv</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">printf</span><span class="p">(</span><span class="s">"%d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">int_min</span> <span class="o">*</span> <span class="n">minus_1</span><span class="p">);</span>
  <span class="n">printf</span><span class="p">(</span><span class="s">"%d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">int_min</span> <span class="o">/</span> <span class="n">minus_1</span><span class="p">);</span>

  <span class="k">return</span> <span class="n">EXIT_SUCCESS</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>

<p>By the way, what’s the first result ? Well, <code class="language-plaintext highlighter-rouge">-2147483648</code>, because the 32-bit multiply operation of two 32-bit numbers provides a 64-bit result, which is <code class="language-plaintext highlighter-rouge">2147483648</code> (0x0000000080000000 in hexadecimal), and the lowest part is just truncated to the 32-bit <code class="language-plaintext highlighter-rouge">0x80000000</code> value, which is… <code class="language-plaintext highlighter-rouge">-2147483648</code> in <a href="https://en.wikipedia.org/wiki/Two%27s_complement">two’s complement</a> arithmetics. Basically the overflow was silently ignored. Got it ?</p>

<h2 id="macros-are-not-functions">Macros Are Not Functions</h2>

<p>Macros are often considered evil (and will even cause C++-fans to scream and have foam on their mouth), especially when hiding multiple argument evaluation:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#define min(a, b) ( (a) &lt; (b) ? (a) : (b) )
</span></code></pre></div></div>

<p>This function will not behave well when used with something like <code class="language-plaintext highlighter-rouge">min(++i, ++j)</code>: a second increment will be committed in either i or j…</p>

<p>Another difference between a function version and its macro version are the side effects, which are committed before and after calling a function, but <em>not</em> a macro.</p>

<h2 id="preprocessor-magic">Preprocessor Magic</h2>

<p>You know that <code class="language-plaintext highlighter-rouge">__FILE__</code> is a magic preprocessor variable holding the current source filename, and <code class="language-plaintext highlighter-rouge">__LINE__</code> the line number <em>at the time of the macro evaluation</em>. This is quite convenient to print debugging:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#define PRINT_CURRENT_FILE_AND_LINE printf("at %s:%d\n", __FILE__, __LINE__)
</span><span class="p">...</span>
<span class="kt">int</span> <span class="nf">main</span><span class="p">(...)</span> <span class="p">{</span>
<span class="p">...</span>
  <span class="n">PRINT_CURRENT_FILE_AND_LINE</span><span class="p">;</span>  <span class="cm">/* print some debug */</span>
<span class="p">...</span>
</code></pre></div></div>

<p>The <code class="language-plaintext highlighter-rouge">__FILE__</code> and <code class="language-plaintext highlighter-rouge">__LINE__</code> macros will be expanded to the location in the main() function, at expansion time.</p>

<p>You can use the expansion time behavior to convert the <code class="language-plaintext highlighter-rouge">__LINE__</code> numerical token to a string one, for example using a double macro expansion:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#define L_(TOKEN) #TOKEN
#define L(TOKEN) L_(TOKEN)
#define FILE_AND_LINE __FILE__ ":" L(__LINE__)
</span>
<span class="kt">int</span> <span class="nf">main</span><span class="p">(</span><span class="kt">int</span> <span class="n">argc</span><span class="p">,</span> <span class="kt">char</span> <span class="o">**</span><span class="n">argv</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">printf</span><span class="p">(</span><span class="s">"we are at: %s</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">FILE_AND_LINE</span><span class="p">);</span>
  <span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>

<p>A call to <code class="language-plaintext highlighter-rouge">FILE_AND_LINE</code> will form a string using the three substrings (in C, “foo” “bar” is equivalent to “foobar”). We need an intermediate <code class="language-plaintext highlighter-rouge">L_()</code> macro because otherwise <code class="language-plaintext highlighter-rouge">__LINE__</code> would not be expanded, and the resulting string would simply be … <code class="language-plaintext highlighter-rouge">"__LINE__"</code>.</p>

<h2 id="aliasing-rules">Aliasing Rules</h2>

<p>My headache is killing me, so I’m going to redirect you to the excellent article on <a href="http://cellperformance.beyond3d.com/articles/2006/06/understanding-strict-aliasing.html">cellperformance</a>. But aliasing rules are very, <em>very</em>, <strong>very</strong> fun, trust me!</p>

<h2 id="more-">More ?</h2>

<p>Do not hesitate to comment this article to complete it! There are other C-corner cases I probably didn’t mention…</p>

<p><strong>TL;DR</strong>: corner cases are not reserved to C++!</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[A Basic Glance At The Virtual Table]]></title>
    <link href="https://blog.httrack.com/blog/2014/05/09/a-basic-glance-at-the-virtual-table/"/>
    <updated>2014-05-09T15:30:00+02:00</updated>
    <id>https://blog.httrack.com/blog/2014/05/09/a-basic-glance-at-the-virtual-table</id>
    <content type="html"><![CDATA[<h2 id="virtual-functions-vtable-and-other-monsters">Virtual Functions, <code class="language-plaintext highlighter-rouge">vtable</code> And Other Monsters</h2>

<p>C++ inheritance, <a href="https://en.wikipedia.org/wiki/Virtual_function">virtual functions</a>, and <a href="https://en.wikipedia.org/wiki/Virtual_method_table">virtual tables</a> are sometimes sees as obscure beasts, something most people will not mess with.</p>

<p>Casting an object to another one or calling a virtual function are probably two of the most common operations you can think of in C++, so it is a shame that such <em>simple</em> operations are not fully understood by everyone.</p>

<p>I have been programming in <code class="language-plaintext highlighter-rouge">C</code> for many years, and one of the benefits of <code class="language-plaintext highlighter-rouge">C</code> is that there is no real magic: when you define a structure, you can <em>see</em> the layout in memory, you can <em>feel</em> the pointers being padded, and every other operations, such as casting a pointer to another one (something you should only do with great care, by the way) are just re-interpreting memory addresses from one type to another one or shifting addresses a bit. No magic. No surprises. Accessing the second member of a pointer array ? This is just basic pointer arithmetic (indirect move with shifting and index, something that even the venerable <a href="https://en.wikipedia.org/wiki/Motorola_68000"><code class="language-plaintext highlighter-rouge">68000</code></a> processor had if my memory is correct)</p>

<p>Some people go straight in <a href="https://en.wikipedia.org/wiki/C++"><code class="language-plaintext highlighter-rouge">C++</code></a> (or even in <a href="https://en.wikipedia.org/wiki/Java_%28programming_language%29"><code class="language-plaintext highlighter-rouge">Java</code></a>) when they start learning programming - which is in my point of view a bad thing - and tend to lose their focus on what’s really happening behind the curtains (ie. inside the memory)</p>

<p>But things are generally much simpler that you thought.</p>

<h2 id="back-to-basis">Back To Basis</h2>

<p>Let’s first have a look at a very simple object. A C++ one:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">class</span> <span class="n">A</span> <span class="p">{</span>
<span class="nl">private:</span>
  <span class="kt">int</span> <span class="n">id</span><span class="p">;</span>

<span class="nl">public:</span>
  <span class="n">A</span><span class="p">()</span><span class="o">:</span> <span class="n">id</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">printf</span><span class="p">(</span><span class="s">"yay, my address is %p, and my id is %d!</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">this</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">id</span><span class="p">);</span>
  <span class="p">}</span>
<span class="p">};</span>
</code></pre></div></div>

<p>This class contains only a constructor, and has one member (an integer). These two things are not of the same nature: variable members and functions are of different kind (yes, this is a really smart and insightful remark, I know).</p>

<p>To make things clearer, we can view this class as a C-structure, and its functions as global functions taking a (hidden) “this” member as first argument (the pointer to the structure itself). The instance members can be accessed implicitly, without the <code class="language-plaintext highlighter-rouge">this-&gt;</code> pointer indirection, by the way: C++ will add the missing bits.</p>

<p>The code above can be rewritten in <strong>pure C</strong> as follows:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// this is the actual "class A" instance data layout!</span>
<span class="k">struct</span> <span class="n">A</span> <span class="p">{</span>
  <span class="kt">int</span> <span class="n">id</span><span class="p">;</span>
<span class="p">};</span>

<span class="c1">// this is the class A equivalent constructor (a global function)</span>
<span class="kt">void</span> <span class="nf">A_A</span><span class="p">(</span><span class="k">struct</span> <span class="n">A</span><span class="o">*</span> <span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">id</span> <span class="o">=</span> <span class="mi">42</span><span class="p">;</span>
  <span class="n">printf</span><span class="p">(</span><span class="s">"yay, my address is %p, and my id is %d!</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">this</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">id</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>

<p>… and, rather that just declaring:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  <span class="n">A</span> <span class="n">a</span><span class="p">;</span>
</code></pre></div></div>

<p>… you’ll need to call the <em>constructor</em> manually:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  <span class="k">struct</span> <span class="n">A</span> <span class="n">a</span><span class="p">;</span>
  <span class="n">A_A</span><span class="p">(</span><span class="o">&amp;</span><span class="n">a</span><span class="p">);</span>  <span class="c1">// call constructor manually!</span>
</code></pre></div></div>

<p>Here it is! C++ in plain C! Of course a C++ compiler will <em>hide everything behind the curtains</em> when using regular classes, but the actual <em>compiled code</em> will be more or less the same.</p>

<p>If you add more members and/or more methods in this class, the principle is the same: extend the structure layout, and/or add more global functions that take an hidden <em>this</em> first argument.</p>

<p><strong>Quizz</strong>: what is the size of a <code class="language-plaintext highlighter-rouge">class A</code> object instance ? The answer is pretty simple: the class only contains one member (<code class="language-plaintext highlighter-rouge">id</code>), which is an integer (32-bit here) ; the <code class="language-plaintext highlighter-rouge">sizeof(A)</code> is therefore 4 (bytes). No magic.</p>

<p><strong>Quizz</strong>: what will be the size of a <code class="language-plaintext highlighter-rouge">class A</code> object instance if a regular member function is added ? The answer is pretty simple, too: its size won’t change, as it still contains only one member - only an additional global function will be added in the code.</p>

<h2 id="what-about-inheritance--this-seems-a-bit-more-complicated">What About Inheritance ? This Seems a Bit More Complicated!</h2>

<p>No, no, <em>not at all</em>.</p>

<p>Let’s take a simple (yet representative) example: our previous A class, another B class with a single “age” member, and at last a final class C with a member “mode”, but inheriting from A and B.</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">class</span> <span class="n">A</span> <span class="p">{</span>
<span class="nl">public:</span>
  <span class="kt">int</span> <span class="n">id</span><span class="p">;</span>

  <span class="n">A</span><span class="p">()</span><span class="o">:</span> <span class="n">id</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">printf</span><span class="p">(</span><span class="s">"A: yay, my address is %p, my size is %zu, and my id is %d!</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span>
           <span class="n">this</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">),</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">id</span><span class="p">);</span>
  <span class="p">}</span>
<span class="p">};</span>

<span class="n">class</span> <span class="n">B</span> <span class="p">{</span>
<span class="nl">public:</span>
  <span class="kt">int</span> <span class="n">age</span><span class="p">;</span>

  <span class="n">B</span><span class="p">()</span><span class="o">:</span> <span class="n">age</span><span class="p">(</span><span class="mi">7</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">printf</span><span class="p">(</span><span class="s">"B: yay, my address is %p, my size is %zu, and my age is %d!</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span>
           <span class="n">this</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">),</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">age</span><span class="p">);</span>
  <span class="p">}</span>
<span class="p">};</span>

<span class="n">class</span> <span class="n">C</span><span class="o">:</span> <span class="n">public</span> <span class="n">A</span><span class="p">,</span> <span class="n">public</span> <span class="n">B</span> <span class="p">{</span>
<span class="nl">public:</span>
  <span class="kt">int</span> <span class="n">mode</span><span class="p">;</span>

  <span class="n">C</span><span class="p">()</span><span class="o">:</span> <span class="n">mode</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">printf</span><span class="p">(</span><span class="s">"C: yay, my address is %p, my size is %zu, my id, age and mode are %d, %d, %d!</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span>
           <span class="n">this</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">),</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">id</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">age</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">mode</span><span class="p">);</span>
  <span class="p">}</span>
<span class="p">};</span>
</code></pre></div></div>

<p>Instantiating an object from C (<code class="language-plaintext highlighter-rouge">C c;</code>) will print something like:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A: yay, my address is 0x73620f9f5c20, my size is 4, and my id is 42!
B: yay, my address is 0x73620f9f5c24, my size is 4, and my age is 7!
C: yay, my address is 0x73620f9f5c20, my size is 12, my id, age and mode are 42, 7, -1!
</code></pre></div></div>

<p>The <code class="language-plaintext highlighter-rouge">class C</code> constructor called first the upper inherited classes constructors (<code class="language-plaintext highlighter-rouge">class A</code> constructor and <code class="language-plaintext highlighter-rouge">class B</code> constructor), and therefore prints their information first. Here again, <code class="language-plaintext highlighter-rouge">C++</code> hides everything behind curtains: the calls to base classes constructors are generated automatically, before your actual <code class="language-plaintext highlighter-rouge">class C</code> constructor code.</p>

<p>You can note that the address seen by the constructors of A and C are identical, <em>but not the one seen by B</em>. How can it be ?</p>

<p>Let’s rewrite everything in plain C:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">struct</span> <span class="n">A</span> <span class="p">{</span>
  <span class="kt">int</span> <span class="n">id</span><span class="p">;</span>
<span class="p">};</span>

<span class="c1">// class A constructor</span>
<span class="kt">void</span> <span class="nf">A_A</span><span class="p">(</span><span class="k">struct</span> <span class="n">A</span><span class="o">*</span> <span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">id</span> <span class="o">=</span> <span class="mi">42</span><span class="p">;</span>
  <span class="n">printf</span><span class="p">(</span><span class="s">"A: yay, my address is %p, my size is %zu, and my id is %d!</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span>
         <span class="n">this</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">),</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">id</span><span class="p">);</span>
<span class="p">}</span>

<span class="k">struct</span> <span class="n">B</span> <span class="p">{</span>
  <span class="kt">int</span> <span class="n">age</span><span class="p">;</span>
<span class="p">};</span>

<span class="c1">// class B constructor</span>
<span class="kt">void</span> <span class="nf">B_B</span><span class="p">(</span><span class="k">struct</span> <span class="n">B</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">age</span> <span class="o">=</span> <span class="mi">7</span><span class="p">;</span>
  <span class="n">printf</span><span class="p">(</span><span class="s">"B: yay, my address is %p, my size is %zu, and my age is %d!</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span>
         <span class="n">this</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">),</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">age</span><span class="p">);</span>
<span class="p">}</span>

<span class="k">struct</span> <span class="n">C</span> <span class="p">{</span>
  <span class="k">struct</span> <span class="n">A</span> <span class="n">a</span><span class="p">;</span>
  <span class="k">struct</span> <span class="n">B</span> <span class="n">b</span><span class="p">;</span>
  <span class="kt">int</span> <span class="n">mode</span><span class="p">;</span>
<span class="p">};</span>

<span class="kt">void</span> <span class="nf">C_C</span><span class="p">(</span><span class="k">struct</span> <span class="n">C</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="c1">// manually call base classes constructors first</span>
  <span class="n">A_A</span><span class="p">(</span><span class="o">&amp;</span><span class="n">this</span><span class="o">-&gt;</span><span class="n">a</span><span class="p">);</span>
  <span class="n">B_B</span><span class="p">(</span><span class="o">&amp;</span><span class="n">this</span><span class="o">-&gt;</span><span class="n">b</span><span class="p">);</span>
  <span class="c1">// then, user code</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">mode</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span><span class="p">;</span>
  <span class="n">printf</span><span class="p">(</span><span class="s">"C: yay, my address is %p, my size is %zu, my id, age and mode are %d, %d, %d!</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span>
         <span class="n">this</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">),</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">a</span><span class="p">.</span><span class="n">id</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">b</span><span class="p">.</span><span class="n">age</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">mode</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>

<p>The <code class="language-plaintext highlighter-rouge">struct C</code> layout is pretty straightforward: the first element is the <code class="language-plaintext highlighter-rouge">struct A</code> members - the reason why constructors from class A and class C did see the same address, the second member is the member(s) of class B, and finally comes the member(s) of C.</p>

<p>In both C++ and C variants,</p>

<ul>
  <li>the size of C is 12 bytes (the three integers)</li>
  <li>the location of A within C is at the beginning (the first four bytes)</li>
  <li>the location of B within C is after the area dedicated to A (the next four bytes)</li>
  <li>the location of members specific to C then follows</li>
</ul>

<p>Casting will provide the same behavior, by the way:</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">static_cast&lt;A*&gt;(&amp;c))</code> == <code class="language-plaintext highlighter-rouge">&amp;(&amp;c)-&gt;a</code> == 0x73620f9f5c20</li>
  <li><code class="language-plaintext highlighter-rouge">static_cast&lt;B*&gt;(&amp;c))</code> == <code class="language-plaintext highlighter-rouge">&amp;(&amp;c)-&gt;b</code> == 0x73620f9f5c24</li>
  <li><code class="language-plaintext highlighter-rouge">static_cast&lt;C*&gt;(&amp;c))</code> == <code class="language-plaintext highlighter-rouge">&amp;c</code> == 0x73620f9f5c20</li>
</ul>

<h2 id="okay-but-virtual-functions--can-you-do-that-in-plain-c-">Okay, But Virtual Functions ? Can You Do That In Plain C ?</h2>

<p>Well, let’s consider a simple example: simply add a virtual function <code class="language-plaintext highlighter-rouge">print</code> to our previous sample, aiming to print the object class name and parameters. The idea is that even if we <code class="language-plaintext highlighter-rouge">cast</code> the high-level <code class="language-plaintext highlighter-rouge">class C</code> instance into <code class="language-plaintext highlighter-rouge">A*</code> or <code class="language-plaintext highlighter-rouge">B*</code>, we’ll see the same behavior.</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">class</span> <span class="n">A</span> <span class="p">{</span>
<span class="nl">public:</span>
  <span class="kt">int</span> <span class="n">id</span><span class="p">;</span>

  <span class="n">A</span><span class="p">()</span><span class="o">:</span> <span class="n">id</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">printf</span><span class="p">(</span><span class="s">"A: yay, my address is %p, my size is %zu, and my id is %d!</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span>
           <span class="n">this</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">),</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">id</span><span class="p">);</span>
  <span class="p">}</span>

  <span class="k">virtual</span> <span class="kt">void</span> <span class="n">print</span><span class="p">()</span> <span class="p">{</span>
    <span class="n">printf</span><span class="p">(</span><span class="s">"I am A(%d)</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">id</span><span class="p">);</span>
  <span class="p">}</span>
<span class="p">};</span>

<span class="n">class</span> <span class="n">B</span> <span class="p">{</span>
<span class="nl">public:</span>
  <span class="kt">int</span> <span class="n">age</span><span class="p">;</span>

  <span class="n">B</span><span class="p">()</span><span class="o">:</span> <span class="n">age</span><span class="p">(</span><span class="mi">7</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">printf</span><span class="p">(</span><span class="s">"B: yay, my address is %p, my size is %zu, and my age is %d!</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span>
           <span class="n">this</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">),</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">age</span><span class="p">);</span>
  <span class="p">}</span>

  <span class="k">virtual</span> <span class="kt">void</span> <span class="n">print</span><span class="p">()</span> <span class="p">{</span>
    <span class="n">printf</span><span class="p">(</span><span class="s">"I am B(%d)</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">age</span><span class="p">);</span>
  <span class="p">}</span>
<span class="p">};</span>

<span class="n">class</span> <span class="n">C</span><span class="o">:</span> <span class="n">public</span> <span class="n">A</span><span class="p">,</span> <span class="n">public</span> <span class="n">B</span> <span class="p">{</span>
<span class="nl">public:</span>
  <span class="kt">int</span> <span class="n">mode</span><span class="p">;</span>

  <span class="n">C</span><span class="p">()</span><span class="o">:</span> <span class="n">mode</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">printf</span><span class="p">(</span><span class="s">"C: yay, my address is %p, my size is %zu, my id, age and mode are %d, %d, %d!</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span>
           <span class="n">this</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">),</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">id</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">age</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">mode</span><span class="p">);</span>
  <span class="p">}</span>

  <span class="k">virtual</span> <span class="kt">void</span> <span class="n">print</span><span class="p">()</span> <span class="p">{</span>
    <span class="n">printf</span><span class="p">(</span><span class="s">"I am C(%d, %d, %d)</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">id</span><span class="p">,</span> <span class="n">age</span><span class="p">,</span> <span class="n">mode</span><span class="p">);</span>
  <span class="p">}</span>
<span class="p">};</span>
</code></pre></div></div>

<p>If we create an instance of <code class="language-plaintext highlighter-rouge">C</code> called <code class="language-plaintext highlighter-rouge">c</code> (<code class="language-plaintext highlighter-rouge">C c;</code>), the following calls will produce exactly the same output:</p>

<ul>
  <li>c.print();</li>
  <li>static_cast&lt;A*&gt;(&amp;c)-&gt;print();</li>
  <li>static_cast&lt;B*&gt;(&amp;c)-&gt;print();</li>
  <li>static_cast&lt;C*&gt;(&amp;c)-&gt;print();</li>
</ul>

<p>Where is the magic ? Let’s first have a look at the output, when creating this object:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A: yay, my address is 0x726fa25b7c00, my size is 16, and my id is 42!
B: yay, my address is 0x726fa25b7c10, my size is 16, and my age is 7!
C: yay, my address is 0x726fa25b7c00, my size is 32, my id, age and mode are 42, 7, -1!
</code></pre></div></div>

<p>We can see that <code class="language-plaintext highlighter-rouge">A</code> and <code class="language-plaintext highlighter-rouge">B</code> are not anymore 4 bytes, but 16. And C is 32 bytes! Somehow, the <code class="language-plaintext highlighter-rouge">C++</code> compiler did add some bytes in our objects without telling us!</p>

<p>Let’s have a look at each object instance actual contents within each classes, by dumping data as pointers in constructors, to figure ou <em>what has been added</em> inside our beloved <del>structures</del> classes:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// simple dump function</span>
<span class="k">static</span> <span class="kt">void</span> <span class="nf">print_object</span><span class="p">(</span><span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">name</span><span class="p">,</span> <span class="kt">void</span> <span class="o">*</span><span class="n">this_</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">size</span><span class="p">)</span> <span class="p">{</span>
  <span class="kt">void</span> <span class="o">**</span><span class="n">ugly</span> <span class="o">=</span> <span class="n">reinterpret_cast</span><span class="o">&lt;</span><span class="kt">void</span><span class="o">**&gt;</span><span class="p">(</span><span class="n">this_</span><span class="p">);</span>
  <span class="kt">size_t</span> <span class="n">i</span><span class="p">;</span>
  <span class="n">printf</span><span class="p">(</span><span class="s">"created %s at address %p of size %zu</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">this_</span><span class="p">,</span> <span class="n">size</span><span class="p">);</span>
  <span class="k">for</span><span class="p">(</span><span class="n">i</span> <span class="o">=</span> <span class="mi">0</span> <span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">size</span> <span class="o">/</span> <span class="k">sizeof</span><span class="p">(</span><span class="kt">void</span><span class="o">*</span><span class="p">)</span> <span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">printf</span><span class="p">(</span><span class="s">"  pointer[%zu] == %p</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">i</span><span class="p">,</span> <span class="n">ugly</span><span class="p">[</span><span class="n">i</span><span class="p">]);</span>
  <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p>And using <code class="language-plaintext highlighter-rouge">print_object(__FUNCTION__, this, sizeof(*this));</code> inside each constructor.</p>

<p>Remember that the constructors are called as follows when creating an instance of C:</p>

<ul>
  <li>base class A constructor is called magically (<code class="language-plaintext highlighter-rouge">this</code> points to the beginning of the allocated C structure)</li>
  <li>base class B constructor is called magically (<code class="language-plaintext highlighter-rouge">this</code> starts at 16 bytes from the beginning of the allocated C structure)</li>
  <li>top level C constructor user code is executed (<code class="language-plaintext highlighter-rouge">this</code> points to the beginning of the allocated C structure)</li>
</ul>

<p>Here’s the result:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>created A at address 0x7082f962ccb0 of size 16
  pointer[0] == 0x400f20
  pointer[1] == 0x2a
created B at address 0x7082f962ccc0 of size 16
  pointer[0] == 0x400f00
  pointer[1] == 0x7
created C at address 0x7082f962ccb0 of size 32
  pointer[0] == 0x400ed0
  pointer[1] == 0x2a
  pointer[2] == 0x400ee8
  pointer[3] == 0xffffffff00000007
</code></pre></div></div>

<p>We can <strong>see</strong> that:</p>

<ul>
  <li>A contains <em>an unknown additional pointer</em> (<code class="language-plaintext highlighter-rouge">0x400f20</code>), and the id variable (<code class="language-plaintext highlighter-rouge">0x2a</code> == <code class="language-plaintext highlighter-rouge">42</code>) padded to 8 bytes</li>
  <li>B contains <em>another unknown additional pointer</em> (<code class="language-plaintext highlighter-rouge">0x400f00</code>), and the age variable (<code class="language-plaintext highlighter-rouge">7</code>) padded to 8 bytes</li>
  <li>C embeds A and B, and the mode variable (-1): these are seen as <code class="language-plaintext highlighter-rouge">0x2a</code>, and <code class="language-plaintext highlighter-rouge">0xffffffff00000007</code> (<code class="language-plaintext highlighter-rouge">00000007</code> for the age variable, and <code class="language-plaintext highlighter-rouge">ffffffff</code> for the mode variable: <code class="language-plaintext highlighter-rouge">-1</code> is <code class="language-plaintext highlighter-rouge">0xffffffff</code> in <a href="https://en.wikipedia.org/wiki/Two%27s_complement">two complement</a>’s world) - and the two additional pointers</li>
</ul>

<p>So it <em>seems</em> that the structure size change reason is that an hidden pointer member (8 bytes in the 64-bit world) has been added to the class instance - and the compiler probably padded the structure to 16 bytes, because you want to have 8-byte alignment when dealing with arrays: if you have an array of two objects, the second object hidden pointer has to be padded to 8 bytes, enforcing the structure to be a multiple of 8 bytes. The 4 bytes used for padding are unused, but useful to maintain the alignment.</p>

<p>Hey, wait! The two pointers are different, when printed inside the C constructor! Yes, good catch: the additional A pointer which was <code class="language-plaintext highlighter-rouge">0x400f20</code> inside the <code class="language-plaintext highlighter-rouge">class A</code> constructor has been changed into <code class="language-plaintext highlighter-rouge">0x400ed0</code> inside the <code class="language-plaintext highlighter-rouge">class C</code> constructor, and the additional B pointer which was <code class="language-plaintext highlighter-rouge">0x400f00</code> inside the <code class="language-plaintext highlighter-rouge">class B</code> constructor has been changed into <code class="language-plaintext highlighter-rouge">0x400ee8</code> inside the <code class="language-plaintext highlighter-rouge">class C</code> constructor.</p>

<p>Oh, and a last note: C is <em>only</em> 32 bytes.</p>

<ul>
  <li>there is no additional hidden pointer for C!</li>
  <li>the compiler apparently used the padding area of <code class="language-plaintext highlighter-rouge">class B</code> for the <code class="language-plaintext highlighter-rouge">mode</code> variable (smart, isn’t it ?)</li>
</ul>

<p>Okay, so <strong>two magical pointers</strong> have been added because we defined a virtual function. What does it mean ?</p>

<p>Let’s go back to basic: <em>how would you implement this class hierarchy in pure C ?</em></p>

<p>Here’s what you may do:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">struct</span> <span class="n">A</span> <span class="p">{</span>
  <span class="kt">int</span> <span class="n">id</span><span class="p">;</span>
<span class="p">};</span>

<span class="c1">// class A constructor</span>
<span class="kt">void</span> <span class="nf">A_A</span><span class="p">(</span><span class="k">struct</span> <span class="n">A</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">id</span> <span class="o">=</span> <span class="mi">42</span><span class="p">;</span>
  <span class="n">print_object</span><span class="p">(</span><span class="n">__FUNCTION__</span><span class="p">,</span> <span class="n">this</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">));</span>
<span class="p">}</span>

<span class="kt">void</span> <span class="nf">A_print</span><span class="p">(</span><span class="k">struct</span> <span class="n">A</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">printf</span><span class="p">(</span><span class="s">"I am A(%d)</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">id</span><span class="p">);</span>
<span class="p">}</span>

<span class="k">struct</span> <span class="n">B</span> <span class="p">{</span>
  <span class="kt">int</span> <span class="n">age</span><span class="p">;</span>
<span class="p">};</span>

<span class="c1">// class B constructor</span>
<span class="kt">void</span> <span class="nf">B_B</span><span class="p">(</span><span class="k">struct</span> <span class="n">B</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">age</span> <span class="o">=</span> <span class="mi">7</span><span class="p">;</span>
  <span class="n">print_object</span><span class="p">(</span><span class="n">__FUNCTION__</span><span class="p">,</span> <span class="n">this</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">));</span>
<span class="p">}</span>

<span class="kt">void</span> <span class="nf">B_print</span><span class="p">(</span><span class="k">struct</span> <span class="n">B</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">printf</span><span class="p">(</span><span class="s">"I am B(%d)</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">age</span><span class="p">);</span>
<span class="p">}</span>

<span class="k">struct</span> <span class="n">C</span> <span class="p">{</span>
  <span class="k">struct</span> <span class="n">A</span> <span class="n">a</span><span class="p">;</span>
  <span class="k">struct</span> <span class="n">B</span> <span class="n">b</span><span class="p">;</span>
  <span class="kt">int</span> <span class="n">mode</span><span class="p">;</span>
<span class="p">};</span>

<span class="kt">void</span> <span class="nf">C_C</span><span class="p">(</span><span class="k">struct</span> <span class="n">C</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="c1">// manually call base classes constructors first</span>
  <span class="n">A_A</span><span class="p">(</span><span class="o">&amp;</span><span class="n">this</span><span class="o">-&gt;</span><span class="n">a</span><span class="p">);</span>
  <span class="n">B_B</span><span class="p">(</span><span class="o">&amp;</span><span class="n">this</span><span class="o">-&gt;</span><span class="n">b</span><span class="p">);</span>
  <span class="c1">// then, user code</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">mode</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span><span class="p">;</span>
  <span class="n">print_object</span><span class="p">(</span><span class="n">__FUNCTION__</span><span class="p">,</span> <span class="n">this</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">));</span>
<span class="p">}</span>

<span class="kt">void</span> <span class="nf">C_print</span><span class="p">(</span><span class="k">struct</span> <span class="n">C</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">printf</span><span class="p">(</span><span class="s">"I am C(%d, %d, %d)</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">a</span><span class="p">.</span><span class="n">id</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">b</span><span class="p">.</span><span class="n">age</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">mode</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Okay, looks more or less what we had previously. But what about the <em>virtual</em> function ? We have to be sure that we call the right function, even if we only have a A* or a B* pointer (and we don’t <em>know</em> that this is actually a C object class behind).</p>

<p>A solution can be to add a <strong>function pointer</strong> <em>for every virtual function</em> in each classes, and initialize inside the constructor!</p>

<p>Something like this:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">struct</span> <span class="n">A</span> <span class="p">{</span>
  <span class="kt">void</span> <span class="p">(</span><span class="o">*</span><span class="n">print</span><span class="p">)(</span><span class="k">struct</span> <span class="n">A</span> <span class="o">*</span><span class="n">this</span><span class="p">);</span>
  <span class="kt">int</span> <span class="n">id</span><span class="p">;</span>
<span class="p">};</span>

<span class="c1">// class A constructor</span>
<span class="kt">void</span> <span class="nf">A_A</span><span class="p">(</span><span class="k">struct</span> <span class="n">A</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">print</span> <span class="o">=</span> <span class="n">A_print</span><span class="p">;</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">id</span> <span class="o">=</span> <span class="mi">42</span><span class="p">;</span>
  <span class="n">print_object</span><span class="p">(</span><span class="n">__FUNCTION__</span><span class="p">,</span> <span class="n">this</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">));</span>
<span class="p">}</span>

<span class="k">struct</span> <span class="n">B</span> <span class="p">{</span>
  <span class="kt">void</span> <span class="p">(</span><span class="o">*</span><span class="n">print</span><span class="p">)(</span><span class="k">struct</span> <span class="n">B</span> <span class="o">*</span><span class="n">this</span><span class="p">);</span>
  <span class="kt">int</span> <span class="n">age</span><span class="p">;</span>
<span class="p">};</span>

<span class="c1">// class B constructor</span>
<span class="kt">void</span> <span class="nf">B_B</span><span class="p">(</span><span class="k">struct</span> <span class="n">B</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">print</span> <span class="o">=</span> <span class="n">B_print</span><span class="p">;</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">age</span> <span class="o">=</span> <span class="mi">7</span><span class="p">;</span>
  <span class="n">print_object</span><span class="p">(</span><span class="n">__FUNCTION__</span><span class="p">,</span> <span class="n">this</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">));</span>
<span class="p">}</span>

<span class="kt">void</span> <span class="nf">B_print</span><span class="p">(</span><span class="k">struct</span> <span class="n">B</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">printf</span><span class="p">(</span><span class="s">"I am B(%d)</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">age</span><span class="p">);</span>
<span class="p">}</span>

<span class="k">struct</span> <span class="n">C</span> <span class="p">{</span>
  <span class="k">struct</span> <span class="n">A</span> <span class="n">a</span><span class="p">;</span>
  <span class="k">struct</span> <span class="n">B</span> <span class="n">b</span><span class="p">;</span>
  <span class="kt">void</span> <span class="p">(</span><span class="o">*</span><span class="n">print</span><span class="p">)(</span><span class="k">struct</span> <span class="n">C</span> <span class="o">*</span><span class="n">this</span><span class="p">);</span>
  <span class="kt">int</span> <span class="n">mode</span><span class="p">;</span>
<span class="p">};</span>

<span class="kt">void</span> <span class="nf">C_C</span><span class="p">(</span><span class="k">struct</span> <span class="n">C</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="c1">// manually call base classes constructors first</span>
  <span class="n">A_A</span><span class="p">(</span><span class="o">&amp;</span><span class="n">this</span><span class="o">-&gt;</span><span class="n">a</span><span class="p">);</span>
  <span class="n">B_B</span><span class="p">(</span><span class="o">&amp;</span><span class="n">this</span><span class="o">-&gt;</span><span class="n">b</span><span class="p">);</span>
  <span class="c1">// then, user code</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">print</span> <span class="o">=</span> <span class="n">C_print</span><span class="p">;</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">a</span><span class="p">.</span><span class="n">print</span> <span class="o">=</span> <span class="n">C_print</span><span class="p">;</span>  <span class="c1">// we patch our own A-&gt;print function !!</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">b</span><span class="p">.</span><span class="n">print</span> <span class="o">=</span> <span class="n">C_print</span><span class="p">;</span>  <span class="c1">// and we need to do it for B-&gt;print too!!</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">mode</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span><span class="p">;</span>
  <span class="n">print_object</span><span class="p">(</span><span class="n">__FUNCTION__</span><span class="p">,</span> <span class="n">this</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">));</span>
<span class="p">}</span>

<span class="kt">void</span> <span class="nf">C_print</span><span class="p">(</span><span class="k">struct</span> <span class="n">C</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">printf</span><span class="p">(</span><span class="s">"I am C(%d, %d, %d)</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">a</span><span class="p">.</span><span class="n">id</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">b</span><span class="p">.</span><span class="n">age</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">mode</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>

<p>What did we do here ?</p>

<ul>
  <li>A and B constructors initialize their versions of <code class="language-plaintext highlighter-rouge">print</code></li>
  <li>C constructor initialize its own version of <code class="language-plaintext highlighter-rouge">print</code> after A and B constructors, and <strong>overrides the one inside A and B</strong></li>
</ul>

<p>Neat, isn’t it ?</p>

<p>Now, whatever pointer you have, these three lines will produce identical behavior:</p>

<ul>
  <li>a-&gt;print(a);</li>
  <li>b-&gt;print(b);</li>
  <li>c-&gt;print(c);</li>
</ul>

<p>Hey, by the way, <em>we do not need an additional pointer for C</em>, because we already have the correct version in two locations: <code class="language-plaintext highlighter-rouge">a.print</code> and <code class="language-plaintext highlighter-rouge">b.print</code>. Let’s remove this useless pointer in C (and the useless initialization), and use:</p>

<ul>
  <li>a-&gt;print(a);</li>
  <li>b-&gt;print(b);</li>
  <li>c-&gt;a.print(c);</li>
</ul>

<p><strong>There is a little problem</strong>, however, when using a C instance cast in A* or B*. When you call <code class="language-plaintext highlighter-rouge">a-&gt;print(a)</code>, the <code class="language-plaintext highlighter-rouge">a</code> pointer is of type A*. And therefore the assignment <code class="language-plaintext highlighter-rouge">this-&gt;a.print = C_print</code> will produce a warning. Similarly, when you call b-&gt;print(b), the <code class="language-plaintext highlighter-rouge">this</code> pointer will have a type of B*, and not only it is of the wrong type, but it has a different address, as c-&gt;b is located <em>sixteen bytes</em> after the beginning of C.</p>

<p>We need <em>specialized</em> versions with correct casts!</p>

<ul>
  <li>casting from A* to C* is straightforward: <em>we know</em> that the address is the same, and the type can be safely cast</li>
  <li>casting from B* to C* requires to adjust the address to find the beginning of C, and a cast (B is in the <em>middle</em> of C, remember)</li>
</ul>

<p>Here we go:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">void</span> <span class="nf">C_print</span><span class="p">(</span><span class="k">struct</span> <span class="n">C</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">printf</span><span class="p">(</span><span class="s">"I am C(%d, %d, %d)</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">a</span><span class="p">.</span><span class="n">id</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">b</span><span class="p">.</span><span class="n">age</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">mode</span><span class="p">);</span>
<span class="p">}</span>

<span class="kt">void</span> <span class="nf">CA_print</span><span class="p">(</span><span class="k">struct</span> <span class="n">A</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">C_print</span><span class="p">((</span><span class="k">struct</span> <span class="n">C</span><span class="o">*</span><span class="p">)</span> <span class="n">this</span><span class="p">);</span>
<span class="p">}</span>

<span class="kt">void</span> <span class="nf">CB_print</span><span class="p">(</span><span class="k">struct</span> <span class="n">B</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="c1">// we know that this points after the struct A layout</span>
  <span class="c1">// so let's fix the pointer by ourselves!</span>
  <span class="kt">size_t</span> <span class="n">offset</span> <span class="o">=</span> <span class="p">(</span><span class="kt">size_t</span><span class="p">)</span> <span class="o">&amp;</span><span class="p">((</span><span class="k">struct</span> <span class="n">C</span><span class="o">*</span><span class="p">)</span> <span class="nb">NULL</span><span class="p">)</span><span class="o">-&gt;</span><span class="n">b</span><span class="p">;</span>
  <span class="n">C_print</span><span class="p">((</span><span class="k">struct</span> <span class="n">C</span><span class="o">*</span><span class="p">)</span> <span class="p">(</span> <span class="p">(</span> <span class="p">(</span><span class="kt">char</span><span class="o">*</span><span class="p">)</span> <span class="n">this</span> <span class="p">)</span> <span class="o">-</span> <span class="n">offset</span> <span class="p">)</span> <span class="p">);</span>
<span class="p">}</span>

<span class="kt">void</span> <span class="nf">C_C</span><span class="p">(</span><span class="k">struct</span> <span class="n">C</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="c1">// manually call base classes constructors first</span>
  <span class="n">A_A</span><span class="p">(</span><span class="o">&amp;</span><span class="n">this</span><span class="o">-&gt;</span><span class="n">a</span><span class="p">);</span>
  <span class="n">B_B</span><span class="p">(</span><span class="o">&amp;</span><span class="n">this</span><span class="o">-&gt;</span><span class="n">b</span><span class="p">);</span>
  <span class="c1">// then, override functions</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">a</span><span class="p">.</span><span class="n">print</span> <span class="o">=</span> <span class="n">CA_print</span><span class="p">;</span>  <span class="c1">// we patch our own function !!</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">b</span><span class="p">.</span><span class="n">print</span> <span class="o">=</span> <span class="n">CB_print</span><span class="p">;</span>  <span class="c1">// and we need to do it for B too!!</span>
  <span class="c1">// this-&gt;print = C_print;  // useless</span>
  <span class="c1">// then, user code</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">mode</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span><span class="p">;</span>
  <span class="n">print_object</span><span class="p">(</span><span class="n">__FUNCTION__</span><span class="p">,</span> <span class="n">this</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">));</span>
<span class="p">}</span>
</code></pre></div></div>

<p>With these adjustments, things are smooth:</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">c.a.print(&amp;c.a)</code> ==&gt; I am C(42, 7, -1)</li>
  <li><code class="language-plaintext highlighter-rouge">c.b.print(&amp;c.b)</code> ==&gt; I am C(42, 7, -1)</li>
  <li><code class="language-plaintext highlighter-rouge">c.print(&amp;c)</code> ==&gt; I am C(42, 7, -1)</li>
</ul>

<p>Perfect, isn’t it ?</p>

<p><strong>Not yet</strong>: what if we add more <em>virtual functions</em> in our structures ? This will increase the object size, and require more pointer initialization in constructors! What a waste of space and time: we are assigning <em>the same pointers</em> each time. Could we <em>prepare</em> the set of pointers for each classes in a static, global structure, and just initialize a <em>pointer to these static data</em> inside the constructors ? Neat, isn’t it ?</p>

<p>Here’s the modified version - let’s call the global structure tables containing specialized function pointers <code class="language-plaintext highlighter-rouge">vtbl</code>, as in <em>virtual table</em>, because this is actually what they are: table for virtual functions!</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// type for global virtual function table structure A</span>
<span class="c1">// this structure enumerates all virtual functions of A</span>
<span class="k">struct</span> <span class="n">A</span><span class="p">;</span>
<span class="k">struct</span> <span class="n">table_A</span> <span class="p">{</span>
  <span class="kt">void</span> <span class="p">(</span><span class="o">*</span><span class="n">print</span><span class="p">)(</span><span class="k">struct</span> <span class="n">A</span> <span class="o">*</span><span class="n">this</span><span class="p">);</span>
<span class="p">};</span>

<span class="k">struct</span> <span class="n">A</span> <span class="p">{</span>
  <span class="k">const</span> <span class="k">struct</span> <span class="n">table_A</span> <span class="o">*</span><span class="n">vtbl</span><span class="p">;</span>
  <span class="kt">int</span> <span class="n">id</span><span class="p">;</span>
<span class="p">};</span>

<span class="kt">void</span> <span class="nf">A_print</span><span class="p">(</span><span class="k">struct</span> <span class="n">A</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">printf</span><span class="p">(</span><span class="s">"I am A(%d)</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">id</span><span class="p">);</span>
<span class="p">}</span>

<span class="c1">// the global, static table data for A</span>
<span class="c1">// this structure enumerates all virtual functions of B</span>
<span class="k">static</span> <span class="k">const</span> <span class="k">struct</span> <span class="n">table_A</span> <span class="n">table_A_for_A</span> <span class="o">=</span> <span class="p">{</span> <span class="n">A_print</span> <span class="p">};</span>

<span class="kt">void</span> <span class="nf">A_A</span><span class="p">(</span><span class="k">struct</span> <span class="n">A</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">vtbl</span> <span class="o">=</span> <span class="o">&amp;</span><span class="n">table_A_for_A</span><span class="p">;</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">id</span> <span class="o">=</span> <span class="mi">42</span><span class="p">;</span>
  <span class="n">print_object</span><span class="p">(</span><span class="n">__FUNCTION__</span><span class="p">,</span> <span class="n">this</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">));</span>
<span class="p">}</span>

<span class="c1">// type for global virtual function table structure B</span>
<span class="k">struct</span> <span class="n">B</span><span class="p">;</span>
<span class="k">struct</span> <span class="n">table_B</span> <span class="p">{</span>
  <span class="kt">void</span> <span class="p">(</span><span class="o">*</span><span class="n">print</span><span class="p">)(</span><span class="k">struct</span> <span class="n">B</span> <span class="o">*</span><span class="n">this</span><span class="p">);</span>
<span class="p">};</span>

<span class="k">struct</span> <span class="n">B</span> <span class="p">{</span>
  <span class="k">const</span> <span class="k">struct</span> <span class="n">table_B</span> <span class="o">*</span><span class="n">vtbl</span><span class="p">;</span>
  <span class="kt">int</span> <span class="n">age</span><span class="p">;</span>
<span class="p">};</span>

<span class="kt">void</span> <span class="nf">B_print</span><span class="p">(</span><span class="k">struct</span> <span class="n">B</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">printf</span><span class="p">(</span><span class="s">"I am B(%d)</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">age</span><span class="p">);</span>
<span class="p">}</span>

<span class="c1">// the global, static table data for B</span>
<span class="k">static</span> <span class="k">const</span> <span class="k">struct</span> <span class="n">table_B</span> <span class="n">table_B_for_B</span> <span class="o">=</span> <span class="p">{</span> <span class="n">B_print</span> <span class="p">};</span>

<span class="kt">void</span> <span class="nf">B_B</span><span class="p">(</span><span class="k">struct</span> <span class="n">B</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">vtbl</span> <span class="o">=</span> <span class="o">&amp;</span><span class="n">table_B_for_B</span><span class="p">;</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">age</span> <span class="o">=</span> <span class="mi">7</span><span class="p">;</span>
  <span class="n">print_object</span><span class="p">(</span><span class="n">__FUNCTION__</span><span class="p">,</span> <span class="n">this</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">));</span>
<span class="p">}</span>

<span class="k">struct</span> <span class="n">C</span> <span class="p">{</span>
  <span class="k">struct</span> <span class="n">A</span> <span class="n">a</span><span class="p">;</span>
  <span class="k">struct</span> <span class="n">B</span> <span class="n">b</span><span class="p">;</span>
  <span class="kt">int</span> <span class="n">mode</span><span class="p">;</span>
<span class="p">};</span>

<span class="kt">void</span> <span class="nf">C_print</span><span class="p">(</span><span class="k">struct</span> <span class="n">C</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">printf</span><span class="p">(</span><span class="s">"I am C(%d, %d, %d)</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">a</span><span class="p">.</span><span class="n">id</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">b</span><span class="p">.</span><span class="n">age</span><span class="p">,</span> <span class="n">this</span><span class="o">-&gt;</span><span class="n">mode</span><span class="p">);</span>
<span class="p">}</span>

<span class="kt">void</span> <span class="nf">CA_print</span><span class="p">(</span><span class="k">struct</span> <span class="n">A</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">C_print</span><span class="p">((</span><span class="k">struct</span> <span class="n">C</span><span class="o">*</span><span class="p">)</span> <span class="n">this</span><span class="p">);</span>
<span class="p">}</span>

<span class="kt">void</span> <span class="nf">CB_print</span><span class="p">(</span><span class="k">struct</span> <span class="n">B</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="c1">// we know that this points after the struct A layout</span>
  <span class="c1">// so let's fix the pointer by ourselves!</span>
  <span class="kt">size_t</span> <span class="n">offset</span> <span class="o">=</span> <span class="p">(</span><span class="kt">size_t</span><span class="p">)</span> <span class="o">&amp;</span><span class="p">((</span><span class="k">struct</span> <span class="n">C</span><span class="o">*</span><span class="p">)</span> <span class="nb">NULL</span><span class="p">)</span><span class="o">-&gt;</span><span class="n">b</span><span class="p">;</span>
  <span class="n">C_print</span><span class="p">((</span><span class="k">struct</span> <span class="n">C</span><span class="o">*</span><span class="p">)</span> <span class="p">(</span> <span class="p">(</span> <span class="p">(</span><span class="kt">char</span><span class="o">*</span><span class="p">)</span> <span class="n">this</span> <span class="p">)</span> <span class="o">-</span> <span class="n">offset</span> <span class="p">)</span> <span class="p">);</span>
<span class="p">}</span>

<span class="c1">// the global, static tables data for A and B structures, for C</span>
<span class="k">static</span> <span class="k">const</span> <span class="k">struct</span> <span class="n">table_A</span> <span class="n">table_A_for_C</span> <span class="o">=</span> <span class="p">{</span> <span class="n">CA_print</span> <span class="p">};</span>
<span class="k">static</span> <span class="k">const</span> <span class="k">struct</span> <span class="n">table_B</span> <span class="n">table_B_for_C</span> <span class="o">=</span> <span class="p">{</span> <span class="n">CB_print</span> <span class="p">};</span>

<span class="kt">void</span> <span class="nf">C_C</span><span class="p">(</span><span class="k">struct</span> <span class="n">C</span> <span class="o">*</span><span class="n">this</span><span class="p">)</span> <span class="p">{</span>
  <span class="c1">// manually call base classes constructors first</span>
  <span class="n">A_A</span><span class="p">(</span><span class="o">&amp;</span><span class="n">this</span><span class="o">-&gt;</span><span class="n">a</span><span class="p">);</span>
  <span class="n">B_B</span><span class="p">(</span><span class="o">&amp;</span><span class="n">this</span><span class="o">-&gt;</span><span class="n">b</span><span class="p">);</span>
  <span class="c1">// Override virtual function pointers that have just been initialized by</span>
  <span class="c1">// A and B constructors with our own version. This allows to override all</span>
  <span class="c1">// possible virtual functions in base classes!</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">a</span><span class="p">.</span><span class="n">vtbl</span> <span class="o">=</span> <span class="o">&amp;</span><span class="n">table_A_for_C</span><span class="p">;</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">b</span><span class="p">.</span><span class="n">vtbl</span> <span class="o">=</span> <span class="o">&amp;</span><span class="n">table_B_for_C</span><span class="p">;</span>
  <span class="n">this</span><span class="o">-&gt;</span><span class="n">mode</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span><span class="p">;</span>
  <span class="n">print_object</span><span class="p">(</span><span class="n">__FUNCTION__</span><span class="p">,</span> <span class="n">this</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="o">*</span><span class="n">this</span><span class="p">));</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Neat, isn’t it ?</p>

<p>Well, I’m telling you a secret: <em>this is more or less what the C++ compiler does</em> (yes, behind the curtains) when implementing virtual functions. The virtual function tables are called <em>virtual tables</em>, and are static structures inside the compiled code. And inside each corresponding class constructor, additional code is <em>inserted</em> just between base class initialization calls, and user code, to set all virtual table pointers corresponding to the class type.</p>

<p>Some additional remarks on this design:</p>

<ul>
  <li>
    <p>if you call a virtual function <em>inside</em> a base class constructor, you will <strong>not</strong> call the top-level function version, because the virtual table pointer has not <em>yet</em> been initialized (it will be initialized <em>after</em> the end of the base class constructor)</p>
  </li>
  <li>
    <p>you need <em>one</em> static structure containing all necessary stuff for a given class</p>
  </li>
  <li>
    <p>you can put additional information on these virtual table structures, which can be useful!</p>

    <ul>
      <li>the class name</li>
      <li>type information functions (what is this object’s real class ?), for example by implementing a hidden <code class="language-plaintext highlighter-rouge">getClassType()</code> virtual function</li>
      <li>dynamic cast support</li>
    </ul>
  </li>
</ul>

<p>… and this is exactly what most compiler do, actually: <a href="https://en.wikipedia.org/wiki/Run-time_type_information">Runtime Type Information</a> data is generally included as soon as a virtual table needs to be created for a given class (ie. you can use dynamic_cast&lt;&gt;())</p>

<p>Now you know what’s going on when you define a virtual function: rather than calling the function directly, an indirection through a virtual table is done.</p>

<p>This has some drawbacks:</p>

<ul>
  <li>
    <p>a little function call impact on performances, because you need to read the actual function pointer in a static table</p>
  </li>
  <li>
    <p>a potentially <em>huge</em> impact on performances because <em>you can not inline anymore function calls</em></p>
  </li>
</ul>

<h2 id="a-deeper-look">A Deeper Look</h2>

<p>Let’s dig a little bit more, using <a href="https://en.wikipedia.org/wiki/GNU_Compiler_Collection">GCC</a> version of the virtual tables.</p>

<p>If you run your program under <a href="https://en.wikipedia.org/wiki/GNU_Debugger">GDB</a> and examine an instance of <code class="language-plaintext highlighter-rouge">class C</code>, you’ll see:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(gdb) p c
$1 = {&lt;A&gt; = {_vptr.A = 0x400e70 &lt;vtable for C+16&gt;, id = 42}, &lt;B&gt; = {_vptr.B = 0x400e88 &lt;vtable for C+40&gt;, age = 7}, mode = -1}
</code></pre></div></div>

<p>Note the <code class="language-plaintext highlighter-rouge">_vptr.A = 0x400e70</code> and <code class="language-plaintext highlighter-rouge">_vptr.B = 0x400e88</code> : these are the virtual tables.</p>

<p>What is behind these pointers ? Recent GCC releases allows you to dig a bit into an object’s virtual table:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(gdb) info vtbl c
vtable for 'C' @ 0x400e70 (subobject @ 0x775bfbf98600):
[0]: 0x400b00 &lt;C::print()&gt;

vtable for 'B' @ 0x400e88 (subobject @ 0x775bfbf98610):
[0]: 0x400b34 &lt;non-virtual thunk to C::print()&gt;
</code></pre></div></div>

<p>Otherwise, you can easily guess that each vtables starts with the same kind of function array we created for our C version:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(gdb) p ((void**)0x400e70)[0]
$5 = (void *) 0x400b00 &lt;C::print()&gt;

(gdb) p ((void**)0x400e88)[0]
$9 = (void *) 0x400b34 &lt;non-virtual thunk to C::print()&gt;
</code></pre></div></div>

<h2 id="easter-egg">Easter Egg</h2>

<p>Now you understand a bit more the concept or vtable, let’s see a dirty-but-fun hack (valid for GCC, possibly for Visual C++ too): how to <em>override</em> a specific object’s virtual function table, and rewire virtual functions at runtime</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Our patched version of C::printf (or A::printf)</span>
<span class="k">static</span> <span class="kt">void</span> <span class="nf">our_function</span><span class="p">(</span><span class="n">C</span> <span class="o">*</span><span class="n">thiz</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">printf</span><span class="p">(</span><span class="s">"I am now a new C(%d, %d, %d)!</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">thiz</span><span class="o">-&gt;</span><span class="n">id</span><span class="p">,</span> <span class="n">thiz</span><span class="o">-&gt;</span><span class="n">age</span><span class="p">,</span> <span class="n">thiz</span><span class="o">-&gt;</span><span class="n">mode</span><span class="p">);</span>
<span class="p">}</span>

<span class="c1">// Our patched version of B::printf</span>
<span class="k">static</span> <span class="kt">void</span> <span class="nf">our_function_from_B</span><span class="p">(</span><span class="n">B</span> <span class="o">*</span><span class="n">thiz</span><span class="p">)</span> <span class="p">{</span>
  <span class="c1">// This is the same boring arithmetic than before, except that</span>
  <span class="c1">// we use a fake C instance located at a random (but non-NULL) address</span>
  <span class="c1">// to compute the offset. Note that the actual instance address will</span>
  <span class="c1">// never be dereferenced, so do not fear illegal accesses!.</span>
  <span class="n">C</span> <span class="o">*</span><span class="n">dummy</span> <span class="o">=</span> <span class="n">reinterpret_cast</span><span class="o">&lt;</span><span class="n">C</span><span class="o">*&gt;</span><span class="p">(</span><span class="mh">0x42</span><span class="p">);</span>
  <span class="n">B</span> <span class="o">*</span><span class="n">dummy_B</span> <span class="o">=</span> <span class="n">static_cast</span><span class="o">&lt;</span><span class="n">B</span><span class="o">*&gt;</span><span class="p">(</span><span class="n">dummy</span><span class="p">);</span>
  <span class="kt">char</span> <span class="o">*</span><span class="n">dummy_ptr</span> <span class="o">=</span> <span class="n">reinterpret_cast</span><span class="o">&lt;</span><span class="kt">char</span><span class="o">*&gt;</span><span class="p">(</span><span class="n">dummy</span><span class="p">);</span>
  <span class="kt">char</span> <span class="o">*</span><span class="n">dummy_B_ptr</span> <span class="o">=</span> <span class="n">reinterpret_cast</span><span class="o">&lt;</span><span class="kt">char</span><span class="o">*&gt;</span><span class="p">(</span><span class="n">dummy_B</span><span class="p">);</span>
  <span class="kt">size_t</span> <span class="n">offset</span> <span class="o">=</span> <span class="n">dummy_B_ptr</span> <span class="o">-</span> <span class="n">dummy_ptr</span><span class="p">;</span>
  <span class="kt">char</span> <span class="o">*</span><span class="n">ptr</span> <span class="o">=</span> <span class="n">reinterpret_cast</span><span class="o">&lt;</span><span class="kt">char</span><span class="o">*&gt;</span><span class="p">(</span><span class="n">thiz</span><span class="p">);</span>
  <span class="n">C</span> <span class="o">*</span><span class="n">thiz_C</span> <span class="o">=</span> <span class="n">reinterpret_cast</span><span class="o">&lt;</span><span class="n">C</span><span class="o">*&gt;</span><span class="p">(</span><span class="n">ptr</span> <span class="o">-</span> <span class="n">offset</span><span class="p">);</span>
  <span class="n">our_function</span><span class="p">(</span><span class="n">thiz_C</span><span class="p">);</span>
<span class="p">}</span>

<span class="kt">int</span> <span class="nf">main</span><span class="p">(</span><span class="kt">int</span> <span class="n">argc</span><span class="p">,</span> <span class="kt">char</span> <span class="o">**</span><span class="n">argv</span><span class="p">)</span> <span class="p">{</span>
  <span class="c1">// create an instance of C</span>
  <span class="n">C</span> <span class="n">c</span><span class="p">;</span>

  <span class="c1">// Hack the c instance and override its print() function!</span>
  <span class="kt">void</span> <span class="o">*</span><span class="n">our_vtbl</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span> <span class="n">reinterpret_cast</span><span class="o">&lt;</span><span class="kt">void</span><span class="o">*&gt;</span><span class="p">(</span><span class="n">our_function</span><span class="p">)</span> <span class="p">};</span>
  <span class="kt">void</span> <span class="o">*</span><span class="n">our_vtbl2</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span> <span class="n">reinterpret_cast</span><span class="o">&lt;</span><span class="kt">void</span><span class="o">*&gt;</span><span class="p">(</span><span class="n">our_function_from_B</span><span class="p">)</span> <span class="p">};</span>
  <span class="c1">// patch for A* and C*</span>
  <span class="o">*</span><span class="n">reinterpret_cast</span><span class="o">&lt;</span><span class="kt">void</span><span class="o">**&gt;</span><span class="p">(</span><span class="n">static_cast</span><span class="o">&lt;</span><span class="n">A</span><span class="o">*&gt;</span><span class="p">(</span><span class="o">&amp;</span><span class="n">c</span><span class="p">))</span> <span class="o">=</span> <span class="n">our_vtbl</span><span class="p">;</span>
  <span class="c1">// patch for B*</span>
  <span class="o">*</span><span class="n">reinterpret_cast</span><span class="o">&lt;</span><span class="kt">void</span><span class="o">**&gt;</span><span class="p">(</span><span class="n">static_cast</span><span class="o">&lt;</span><span class="n">B</span><span class="o">*&gt;</span><span class="p">(</span><span class="o">&amp;</span><span class="n">c</span><span class="p">))</span> <span class="o">=</span> <span class="n">our_vtbl2</span><span class="p">;</span>

  <span class="c1">// get the pointers (note: GCC will optimize direct calls to c.printf)</span>
  <span class="n">A</span> <span class="o">*</span><span class="n">pa</span> <span class="o">=</span> <span class="o">&amp;</span><span class="n">c</span><span class="p">;</span>
  <span class="n">B</span> <span class="o">*</span><span class="n">pb</span> <span class="o">=</span> <span class="o">&amp;</span><span class="n">c</span><span class="p">;</span>
  <span class="n">C</span> <span class="o">*</span><span class="n">pc</span> <span class="o">=</span> <span class="o">&amp;</span><span class="n">c</span><span class="p">;</span>

  <span class="c1">// and call printf() ... what will be printed ?</span>
  <span class="n">pa</span><span class="o">-&gt;</span><span class="n">print</span><span class="p">();</span>
  <span class="n">pb</span><span class="o">-&gt;</span><span class="n">print</span><span class="p">();</span>
  <span class="n">pc</span><span class="o">-&gt;</span><span class="n">print</span><span class="p">();</span>

  <span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>

<p>And yes, your object was hacked:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>I am now a new C(42, 7, -1)!
I am now a new C(42, 7, -1)!
I am now a new C(42, 7, -1)!
</code></pre></div></div>

<p>Neat, isn’t it ? (yes, yes, not really portable though)</p>

<h2 id="references">References</h2>

<ul>
  <li>
    <p>You may have a look at the <a href="http://refspecs.linuxbase.org/cxxabi-1.83.html#vtable">Itanium C++ ABI, Virtual Table Layout</a> description (GCC <a href="http://gcc.gnu.org/onlinedocs/gcc/Compatibility.html">is using</a> the Itanium ABI for vtables, or at least something very close)</p>
  </li>
  <li>
    <p>For Windows coders, the <a href="http://www.openrce.org/articles/full_view/23">Reversing Microsoft Visual C++ Part II</a> page will be a goldmine, too</p>
  </li>
</ul>

<p><strong>TL;DR</strong>: remember that C++ used to be <em>enhanced C</em> a long time ago, and most basic features can be easily implemented <strong>and understood</strong> through plain C code!</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[A Story Of realloc (And Laziness)]]></title>
    <link href="https://blog.httrack.com/blog/2014/04/05/a-story-of-realloc-and-laziness/"/>
    <updated>2014-04-05T09:46:00+02:00</updated>
    <id>https://blog.httrack.com/blog/2014/04/05/a-story-of-realloc-and-laziness</id>
    <content type="html"><![CDATA[<h2 id="a-simple-macro">A Simple Macro</h2>

<p>It began with a simple macro: (<em>approximative code</em>)</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#define ADD_BYTE(C) do {            \
  if (offset == capa) {             \
    if (capa &lt; 16) {                \
      capa = 16;                    \
    } else {                        \
      capa &lt;&lt;= 1;                   \
    }                               \
    buffer = realloc(buffer, capa); \
    assert(buffer != NULL);         \
  }                                 \
  buffer[offset++] = (C);           \
} while(0)
</span></code></pre></div></div>

<p>For the <a href="https://en.wikipedia.org/wiki/C_%28programming_language%29"><code class="language-plaintext highlighter-rouge">C</code></a>-illiterates, this simple macro just adds the byte ‘C’ to the dynamically allocated buffer <code class="language-plaintext highlighter-rouge">buffer</code>, whose capacity (in bytes) is <code class="language-plaintext highlighter-rouge">capa</code>. The <em>next position to be written</em> is identified by <code class="language-plaintext highlighter-rouge">offset</code>. The capacity is doubled each time the buffer is filled (starting with a minimum of 16 bytes).</p>

<p>We’re adding bytes in a dynamic buffer - this is one of the most common thing you can do in a given program (used in strings, in arrays, etc.)</p>

<p>The problem is to understand if the reallocation strategy is good enough. If you take a look at the <strong>complexity</strong>, assuming <code class="language-plaintext highlighter-rouge">realloc</code> is <a href="https://en.wikipedia.org/wiki/Big_O_notation"><em>O(N)</em></a>, you quickly conclude that adding a single byte is on <em>average</em> O(1), which is good.</p>

<p>But what is the worst-case complexity, when reallocating a buffer ?</p>

<blockquote>
  <p>Guy: Are you sure this is the good strategy ? It looks like you have a big performance issue if you extend the size of a large array - say, one gigabyte for example. Imagine the impact, especially if the buffer has to be moved from the swap.</p>
</blockquote>

<blockquote>
  <p>Me: Hummm, I never really thought about this, but I think it’s fine. I’m pretty sure the system can handle that gently, actually.</p>
</blockquote>

<blockquote>
  <p>Guy: I think a linked structure might be a better idea - maybe with the same exponential strategy, for example</p>
</blockquote>

<blockquote>
  <p>Me: I’m pretty sure this is a waste of time</p>
</blockquote>

<blockquote>
  <p>Guy: Okay, show me</p>
</blockquote>

<h2 id="sheesh">Sheesh</h2>

<p>Okay, now I need to back my arguments. As all programmers, I am lazy. I should say: <em>as all programmers should be</em>, I am lazy. Laziness is an incentive to be smarter and more efficient. You are <em>too lazy</em> to execute repetitive or boring tasks, and you need something <em>smarter</em>, something <em>faster</em>. This is sometimes called <em>laziness</em>. I prefer to call that <strong>efficiency</strong>.</p>

<p>Having a linked list of blocks to store a buffer would be a bit cumbersome. I didn’t say <em>impossible</em> - <a href="http://fr.wikipedia.org/wiki/Impossible_n%27est_pas_fran%C3%A7ais"><em>Impossible n’est pas français</em></a> as we say sometimes (well, it did not end well for Napoléon, but nevermind) - but cumbersome. Copying a sub-array or storing it on disk would require some work, we probably would need an index of all arrays base pointers, get the low offset 2-based logarithm to get the initial block, and do some other boring stuff. Sheesh, I’m already tired.</p>

<p>I have the feeling that being lazy is the right thing to do here. Because for small memory blocks, we basically do not care, and for large blocks, the system’s virtual memory will gently take care of the issue.</p>

<p>Let’s have a look.</p>

<h2 id="what-is-realloc-by-the-way-">What is <code class="language-plaintext highlighter-rouge">realloc()</code>, By The Way ?</h2>

<p>It is a regular function, <a href="http://pubs.opengroup.org/onlinepubs/009695299/functions/realloc.html">standardized by POSIX</a>, located in the <a href="https://en.wikipedia.org/wiki/C_POSIX_library">C library</a>. On linux, you can find it inside the <code class="language-plaintext highlighter-rouge">libc.so</code> library, with its siblings <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/malloc.html"><code class="language-plaintext highlighter-rouge">malloc</code></a> and <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/free.html"><code class="language-plaintext highlighter-rouge">free</code></a>:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nm <span class="nt">-D</span> /lib/x86_64-linux-gnu/libc.so.6 | <span class="nb">grep</span> <span class="nt">-E</span> <span class="s2">"T (malloc|realloc|free)$"</span>
000000000007e450 T free
000000000007e030 T malloc
000000000007e4e0 T realloc
</code></pre></div></div>

<p>For the curious, “<code class="language-plaintext highlighter-rouge">T</code>” stands for “text symbol” (the uppercase means that the symbol is public and visible) ; the <a href="https://en.wikipedia.org/wiki/Code_segment"><em>text segment</em></a> of a program being the <em>code</em>, the <a href="https://en.wikipedia.org/wiki/Data_segment"><em>data segment</em></a> being initialized data (variables), and the <a href="https://en.wikipedia.org/wiki/.bss"><em>bss segment</em></a> being uninitialized (well, actually initialized to <em>zero</em>) data (variables).</p>

<p>Allocating memory, reallocating memory and freeing memory is done through an allocator (thank you Captain Obvious!). There are plenty of them (one of the most common being the <a href="https://en.wikipedia.org/wiki/Buddy_memory_allocation">buddy allocator</a>).</p>

<p>We can implement our own trivial allocator using the venerable <a href="http://pubs.opengroup.org/onlinepubs/7908799/xsh/brk.html"><code class="language-plaintext highlighter-rouge">sbrk</code></a> call, which basically expand the end of the <em>data segment</em> to have more space:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#include</span> <span class="cpf">&lt;sys/types.h&gt;</span><span class="cp">
#include</span> <span class="cpf">&lt;unistd.h&gt;</span><span class="cp">
#include</span> <span class="cpf">&lt;string.h&gt;</span><span class="cp">
</span>
<span class="kt">void</span> <span class="o">*</span><span class="nf">malloc</span><span class="p">(</span><span class="kt">size_t</span> <span class="n">size</span><span class="p">)</span> <span class="p">{</span>
  <span class="k">return</span> <span class="n">sbrk</span><span class="p">(</span><span class="n">size</span><span class="p">);</span>
<span class="p">}</span>

<span class="kt">void</span> <span class="nf">free</span><span class="p">(</span><span class="kt">void</span> <span class="o">*</span><span class="n">ptr</span><span class="p">)</span> <span class="p">{</span>
  <span class="cm">/* who cares ? */</span>
<span class="p">}</span>

<span class="cm">/* Note: meant to grow only (yes, I am lazy) */</span>
<span class="kt">void</span> <span class="o">*</span><span class="nf">realloc</span><span class="p">(</span><span class="kt">void</span> <span class="o">*</span><span class="n">ptr</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">size</span><span class="p">)</span> <span class="p">{</span>
  <span class="kt">void</span> <span class="o">*</span><span class="n">nptr</span> <span class="o">=</span> <span class="n">malloc</span><span class="p">(</span><span class="n">size</span><span class="p">);</span>
  <span class="k">if</span> <span class="p">(</span><span class="n">nptr</span> <span class="o">==</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
  <span class="p">}</span>
  <span class="c1">// 'old_size' is supposed to be known :)</span>
  <span class="c1">// (for example put in a header before the actual block)</span>
  <span class="n">memcpy</span><span class="p">(</span><span class="n">nptr</span><span class="p">,</span> <span class="n">ptr</span><span class="p">,</span> <span class="n">old_size</span><span class="p">);</span>
  <span class="n">free</span><span class="p">(</span><span class="n">ptr</span><span class="p">);</span>
  <span class="k">return</span> <span class="n">nptr</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>

<p><em>[ Note: as someone noted on hacker news, the old_size address needs to be known here ; for example by using a header before the returned address. And no, we shouldn’t free the original block upon realloc failure :) ]</em></p>

<p>A real allocator would be a <em>bit</em> more clever, of course, with complex data structures aimed to limit the number of actual <code class="language-plaintext highlighter-rouge">memcpy</code> calls.</p>

<p>We need to dig a bit inside the <a href="http://www.gnu.org/software/libc/libc.html">Glibc</a> used on Linux to figure out what’s happening for large blocks.</p>

<h2 id="digging-in-the-glibc">Digging In The Glibc</h2>

<p>Just <a href="https://www.gnu.org/software/libc/download.html">download</a> a recent glibc release, and have a look at the source tree.</p>

<p>There is an <em>interesting</em> directory out there: the <code class="language-plaintext highlighter-rouge">malloc</code> directory. Just pick the <a href="http://fossies.org/linux/misc/glibc-2.19.tar.gz:a/glibc-2.19/malloc/malloc.c#l_52"><code class="language-plaintext highlighter-rouge">malloc.c</code></a> file and open it in an editor.</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/*
  The main properties of the algorithms are:
  * For large (&gt;= 512 bytes) requests, it is a pure best-fit allocator,
    with ties normally decided via FIFO (i.e. least recently used).
  * For small (&lt;= 64 bytes by default) requests, it is a caching
    allocator, that maintains pools of quickly recycled chunks.
  * In between, and for combinations of large and small requests, it does
    the best it can trying to meet both goals at once.
  * For very large requests (&gt;= 128KB by default), it relies on system
    memory mapping facilities, if supported.

  For a longer but slightly out of date high-level description, see
     http://gee.cs.oswego.edu/dl/html/malloc.html
*/</span>
</code></pre></div></div>

<p>The interesting part (for us) is: <em>For very large requests (&gt;= 128KB by default), it relies on system memory mapping facilities, if supported.</em>.</p>

<p>The <em>128KB by default</em> can actually be controlled by the <a href="http://man7.org/linux/man-pages/man3/mallopt.3.html"><code class="language-plaintext highlighter-rouge">mallopt</code></a> function:</p>

<blockquote>
  <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  M_MMAP_THRESHOLD
         For allocations greater than or equal to the limit specified
         (in bytes) by M_MMAP_THRESHOLD that can't be satisfied from
         the free list, the memory-allocation functions employ mmap(2)
         instead of increasing the program break using sbrk(2).
</code></pre></div>  </div>
</blockquote>

<p>Okay, so as I said previously, the system’s virtual memory will handle everything when dealing with <em>large blocks</em>.</p>

<p>Basically, it means that:</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">malloc</code> will be trunked to anonymous <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/mmap.html"><code class="language-plaintext highlighter-rouge">mmap</code></a></li>
  <li><code class="language-plaintext highlighter-rouge">free</code> will be trunked to <a href="http://pubs.opengroup.org/onlinepubs/009696799/functions/munmap.html"><code class="language-plaintext highlighter-rouge">munmap</code></a></li>
  <li><code class="language-plaintext highlighter-rouge">realloc</code> will be trunked to <a href="http://man7.org/linux/man-pages/man2/mremap.2.html"><code class="language-plaintext highlighter-rouge">mremap</code></a> (Darn, not yet in POSIX ? Guys ?)</li>
</ul>

<p>Okay, so we need to dig a little deeper inside mremap.</p>

<p>A <a href="http://man7.org/linux/man-pages/man2/mremap.2.html"><code class="language-plaintext highlighter-rouge">man mremap</code></a> will tell us that:</p>

<blockquote>
  <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  mremap()  uses  the  Linux page table scheme.  mremap() changes the mapping between virtual addresses and memory pages.  This can be
  used to implement a very efficient realloc(3).
</code></pre></div>  </div>
</blockquote>

<p>Humm, <em>very efficient realloc</em>. How efficient ?</p>

<p>First, <code class="language-plaintext highlighter-rouge">mmap</code>, <code class="language-plaintext highlighter-rouge">munmap</code>, and <code class="language-plaintext highlighter-rouge">mremap</code> do have definitions in the glibc. Well, pretty small entry points, actually:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nm <span class="nt">-D</span> /lib/x86_64-linux-gnu/libc.so.6 | <span class="nb">grep</span> <span class="nt">-E</span> <span class="s2">"(mmap|munmap|mremap)$"</span>
00000000000e4350 W mmap
00000000000e9080 W mremap
00000000000e4380 W munmap
</code></pre></div></div>

<p>Note that the default entry points are <em>weak symbols</em> in this case, which basically means that they can be overridden by <em>someone else</em>, during dynamic link ; eg. in this case:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ldd /tmp/sample
        linux-vdso.so.1 <span class="o">(</span>0x00007584a8aaa000<span class="o">)</span>
        libc.so.6 <span class="o">=&gt;</span> /lib/x86_64-linux-gnu/libc.so.6 <span class="o">(</span>0x00007584a86e6000<span class="o">)</span>
        /lib64/ld-linux-x86-64.so.2 <span class="o">(</span>0x00007584a8aac000
</code></pre></div></div>

<p>… the symbols <em>may</em> be overridden by <code class="language-plaintext highlighter-rouge">linux-vdso.so.1</code>, which is a <a href="http://man7.org/linux/man-pages/man7/vdso.7.html">magical library</a> mapped in all programs on Linux to speedup some calls involving <a href="https://en.wikipedia.org/wiki/System_call">system calls</a>.</p>

<p>Anyway, our symbols in the glibc are only trunks to kernel system calls (aka “syscalls”) whether they are in the glibc or inside the vdso ; see the <a href="http://fossies.org/linux/misc/glibc-2.19.tar.gz:a/glibc-2.19/sysdeps/unix/sysv/linux/mmap64.c"><code class="language-plaintext highlighter-rouge">sysdeps/unix/sysv/linux/mmap64.c</code></a> default implementations, for example:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">void</span> <span class="o">*</span>
<span class="nf">__mmap64</span> <span class="p">(</span><span class="kt">void</span> <span class="o">*</span><span class="n">addr</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">len</span><span class="p">,</span> <span class="kt">int</span> <span class="n">prot</span><span class="p">,</span> <span class="kt">int</span> <span class="n">flags</span><span class="p">,</span> <span class="kt">int</span> <span class="n">fd</span><span class="p">,</span> <span class="n">off64_t</span> <span class="n">offset</span><span class="p">)</span>
<span class="p">{</span>
  <span class="c1">// removed arguments check</span>
  <span class="kt">void</span> <span class="o">*</span><span class="n">result</span><span class="p">;</span>
  <span class="n">result</span> <span class="o">=</span> <span class="p">(</span><span class="kt">void</span> <span class="o">*</span><span class="p">)</span>
    <span class="n">INLINE_SYSCALL</span> <span class="p">(</span><span class="n">mmap2</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="n">addr</span><span class="p">,</span>
                    <span class="n">len</span><span class="p">,</span> <span class="n">prot</span><span class="p">,</span> <span class="n">flags</span><span class="p">,</span> <span class="n">fd</span><span class="p">,</span>
                    <span class="p">(</span><span class="kt">off_t</span><span class="p">)</span> <span class="p">(</span><span class="n">offset</span> <span class="o">&gt;&gt;</span> <span class="n">page_shift</span><span class="p">));</span>
  <span class="k">return</span> <span class="n">result</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Okay, so our initial question is not anymore glibc-related, but involves the Linux kernel.</p>

<h2 id="digging-in-the-kernel">Digging In The Kernel</h2>

<p>Now <a href="https://www.kernel.org/">download</a> a recent kernel version and let’s have a quick look at what <code class="language-plaintext highlighter-rouge">mremap</code> involves.</p>

<p>If you take a look at the <a href="http://www.tldp.org/LDP/tlk/sources/sources.html">The Linux Kernel Howto</a>, you see that there is an interesting directory for us:</p>

<blockquote>
  <p>mm
   This directory contains all of the memory management code. The architecture specific memory management code lives down in arch/*/mm/, for example arch/i386/mm/fault.c.</p>
</blockquote>

<p>Ah, nice, just what we needed to check!</p>

<p>The <a href="http://lxr.free-electrons.com/source/mm/mremap.c#L461"><code class="language-plaintext highlighter-rouge">mm/mremap.c</code></a> file seems to be of some interest. And inside the file, you’ll discover the syscall entry point of the mremap function. It’s just here:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/*
 * Expand (or shrink) an existing mapping, potentially moving it at the
 * same time (controlled by the MREMAP_MAYMOVE flag and available VM space)
 *
 * MREMAP_FIXED option added 5-Dec-1999 by Benjamin LaHaise
 * This option implies MREMAP_MAYMOVE.
 */</span>
<span class="n">SYSCALL_DEFINE5</span><span class="p">(</span><span class="n">mremap</span><span class="p">,</span> <span class="kt">unsigned</span> <span class="kt">long</span><span class="p">,</span> <span class="n">addr</span><span class="p">,</span> <span class="kt">unsigned</span> <span class="kt">long</span><span class="p">,</span> <span class="n">old_len</span><span class="p">,</span>
                <span class="kt">unsigned</span> <span class="kt">long</span><span class="p">,</span> <span class="n">new_len</span><span class="p">,</span> <span class="kt">unsigned</span> <span class="kt">long</span><span class="p">,</span> <span class="n">flags</span><span class="p">,</span>
                <span class="kt">unsigned</span> <span class="kt">long</span><span class="p">,</span> <span class="n">new_addr</span><span class="p">)</span>
<span class="p">{</span>
</code></pre></div></div>

<p>This is basically where you <em>enter</em> in the kernel when invoking the corresponding syscall in your user code (through the glibc or through the corresponding vdso thunk).</p>

<p>If you read this function’s code, you first see various arguments checks, and trivial cases being handled (especially <em>shrinking</em> a mapped memory block, which is just freeing the pages on the tail).</p>

<p>The kernel then attempt to expand the mapped area by growing it (as would do an allocator in a C library with <code class="language-plaintext highlighter-rouge">sbrk</code>), and successfully return if expanding the area was possible.</p>

<p>All these trivial cases are <em>O(1)</em> (even if entering in the kernel has a cost - a cost lower now that interrupts aren’t involved anymore, but this is still costly).</p>

<p>What about the <em>worst case</em> ?</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code>        <span class="cm">/*
         * We weren't able to just expand or shrink the area,
         * we need to create a new one and move it..
         */</span>
        <span class="n">ret</span> <span class="o">=</span> <span class="o">-</span><span class="n">ENOMEM</span><span class="p">;</span>
        <span class="k">if</span> <span class="p">(</span><span class="n">flags</span> <span class="o">&amp;</span> <span class="n">MREMAP_MAYMOVE</span><span class="p">)</span> <span class="p">{</span>
                <span class="kt">unsigned</span> <span class="kt">long</span> <span class="n">map_flags</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
                <span class="k">if</span> <span class="p">(</span><span class="n">vma</span><span class="o">-&gt;</span><span class="n">vm_flags</span> <span class="o">&amp;</span> <span class="n">VM_MAYSHARE</span><span class="p">)</span>
                        <span class="n">map_flags</span> <span class="o">|=</span> <span class="n">MAP_SHARED</span><span class="p">;</span>

                <span class="n">new_addr</span> <span class="o">=</span> <span class="n">get_unmapped_area</span><span class="p">(</span><span class="n">vma</span><span class="o">-&gt;</span><span class="n">vm_file</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">new_len</span><span class="p">,</span>
                                        <span class="n">vma</span><span class="o">-&gt;</span><span class="n">vm_pgoff</span> <span class="o">+</span>
                                        <span class="p">((</span><span class="n">addr</span> <span class="o">-</span> <span class="n">vma</span><span class="o">-&gt;</span><span class="n">vm_start</span><span class="p">)</span> <span class="o">&gt;&gt;</span> <span class="n">PAGE_SHIFT</span><span class="p">),</span>
                                        <span class="n">map_flags</span><span class="p">);</span>
                <span class="k">if</span> <span class="p">(</span><span class="n">new_addr</span> <span class="o">&amp;</span> <span class="o">~</span><span class="n">PAGE_MASK</span><span class="p">)</span> <span class="p">{</span>
                        <span class="n">ret</span> <span class="o">=</span> <span class="n">new_addr</span><span class="p">;</span>
                        <span class="k">goto</span> <span class="n">out</span><span class="p">;</span>
                <span class="p">}</span>

                <span class="n">map_flags</span> <span class="o">=</span> <span class="n">vma</span><span class="o">-&gt;</span><span class="n">vm_flags</span><span class="p">;</span>
                <span class="n">ret</span> <span class="o">=</span> <span class="n">move_vma</span><span class="p">(</span><span class="n">vma</span><span class="p">,</span> <span class="n">addr</span><span class="p">,</span> <span class="n">old_len</span><span class="p">,</span> <span class="n">new_len</span><span class="p">,</span> <span class="n">new_addr</span><span class="p">);</span>
                <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="p">(</span><span class="n">ret</span> <span class="o">&amp;</span> <span class="o">~</span><span class="n">PAGE_MASK</span><span class="p">))</span> <span class="p">{</span>
                        <span class="n">track_exec_limit</span><span class="p">(</span><span class="n">current</span><span class="o">-&gt;</span><span class="n">mm</span><span class="p">,</span> <span class="n">addr</span><span class="p">,</span> <span class="n">addr</span> <span class="o">+</span> <span class="n">old_len</span><span class="p">,</span> <span class="mi">0UL</span><span class="p">);</span>
                        <span class="n">track_exec_limit</span><span class="p">(</span><span class="n">current</span><span class="o">-&gt;</span><span class="n">mm</span><span class="p">,</span> <span class="n">new_addr</span><span class="p">,</span> <span class="n">new_addr</span> <span class="o">+</span> <span class="n">new_len</span><span class="p">,</span> <span class="n">map_flags</span><span class="p">);</span>
                <span class="p">}</span>
        <span class="p">}</span>
</code></pre></div></div>

<p>At first glance, the kernel is doing what we did in our trivial allocator:</p>

<ul>
  <li>allocate a new area</li>
  <li>copy the previous area to the new one, and free the previous one (ie. move)</li>
</ul>

<p>But let’s have a <em>closer</em> look.</p>

<p>The <a href="http://lxr.free-electrons.com/source/mm/mremap.c#L229"><code class="language-plaintext highlighter-rouge">move_vma</code></a> call follows:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">static</span> <span class="kt">unsigned</span> <span class="kt">long</span> <span class="nf">move_vma</span><span class="p">(</span><span class="k">struct</span> <span class="n">vm_area_struct</span> <span class="o">*</span><span class="n">vma</span><span class="p">,</span>
                <span class="kt">unsigned</span> <span class="kt">long</span> <span class="n">old_addr</span><span class="p">,</span> <span class="kt">unsigned</span> <span class="kt">long</span> <span class="n">old_len</span><span class="p">,</span>
                <span class="kt">unsigned</span> <span class="kt">long</span> <span class="n">new_len</span><span class="p">,</span> <span class="kt">unsigned</span> <span class="kt">long</span> <span class="n">new_addr</span><span class="p">)</span>
<span class="p">{</span>
<span class="p">...</span>
        <span class="n">new_pgoff</span> <span class="o">=</span> <span class="n">vma</span><span class="o">-&gt;</span><span class="n">vm_pgoff</span> <span class="o">+</span> <span class="p">((</span><span class="n">old_addr</span> <span class="o">-</span> <span class="n">vma</span><span class="o">-&gt;</span><span class="n">vm_start</span><span class="p">)</span> <span class="o">&gt;&gt;</span> <span class="n">PAGE_SHIFT</span><span class="p">);</span>
        <span class="n">new_vma</span> <span class="o">=</span> <span class="n">copy_vma</span><span class="p">(</span><span class="o">&amp;</span><span class="n">vma</span><span class="p">,</span> <span class="n">new_addr</span><span class="p">,</span> <span class="n">new_len</span><span class="p">,</span> <span class="n">new_pgoff</span><span class="p">,</span>
                           <span class="o">&amp;</span><span class="n">need_rmap_locks</span><span class="p">);</span>
        <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="n">new_vma</span><span class="p">)</span>
                <span class="k">return</span> <span class="o">-</span><span class="n">ENOMEM</span><span class="p">;</span>

        <span class="n">moved_len</span> <span class="o">=</span> <span class="n">move_page_tables</span><span class="p">(</span><span class="n">vma</span><span class="p">,</span> <span class="n">old_addr</span><span class="p">,</span> <span class="n">new_vma</span><span class="p">,</span> <span class="n">new_addr</span><span class="p">,</span> <span class="n">old_len</span><span class="p">,</span>
                                     <span class="n">need_rmap_locks</span><span class="p">);</span>
</code></pre></div></div>

<p>There are two interesting calls here:</p>

<ul>
  <li>the <code class="language-plaintext highlighter-rouge">copy_vma</code> call</li>
  <li>the <code class="language-plaintext highlighter-rouge">move_page_tables</code> call</li>
</ul>

<p>The <code class="language-plaintext highlighter-rouge">copy_vma</code> function is inside the sibling <a href="http://lxr.free-electrons.com/source/mm/mmap.c#L2781"><code class="language-plaintext highlighter-rouge">mm/mmap.c</code></a> file, and basically moves the internal kernel structure related to the mmap’ed block, which is a <code class="language-plaintext highlighter-rouge">vm_area_struct</code> structure.</p>

<p>You can find its definition in <a href="http://lxr.free-electrons.com/source/include/linux/mm_types.h#L240"><code class="language-plaintext highlighter-rouge">include/linux/mm_types.h</code></a> ; this <em>small</em> structure contains all information on the region: the start and end address, the mmap’ed file on disk if any, etc.</p>

<p>So moving this one will be cheap, <em>O(1)</em>.</p>

<p>What about <a href="http://lxr.free-electrons.com/source/mm/mremap.c#L164"><code class="language-plaintext highlighter-rouge">move_page_tables</code></a> ?</p>

<p>The interesting part seems to lie within this loop:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">...</span>
        <span class="k">for</span> <span class="p">(;</span> <span class="n">old_addr</span> <span class="o">&lt;</span> <span class="n">old_end</span><span class="p">;</span> <span class="n">old_addr</span> <span class="o">+=</span> <span class="n">extent</span><span class="p">,</span> <span class="n">new_addr</span> <span class="o">+=</span> <span class="n">extent</span><span class="p">)</span> <span class="p">{</span>
<span class="p">...</span>
                <span class="n">move_ptes</span><span class="p">(</span><span class="n">vma</span><span class="p">,</span> <span class="n">old_pmd</span><span class="p">,</span> <span class="n">old_addr</span><span class="p">,</span> <span class="n">old_addr</span> <span class="o">+</span> <span class="n">extent</span><span class="p">,</span>
                          <span class="n">new_vma</span><span class="p">,</span> <span class="n">new_pmd</span><span class="p">,</span> <span class="n">new_addr</span><span class="p">,</span> <span class="n">need_rmap_locks</span><span class="p">);</span>
<span class="p">...</span>
</code></pre></div></div>

<p>The code is <em>a bit</em> complicated (and lacks a bit some comments, too), but is mostly basic arithmetics and accounting.</p>

<p>The <a href="http://lxr.free-electrons.com/source/mm/mremap.c#L90"><code class="language-plaintext highlighter-rouge">move_ptes</code></a> function contains itself the deepest inner loop:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">...</span>
        <span class="k">for</span> <span class="p">(;</span> <span class="n">old_addr</span> <span class="o">&lt;</span> <span class="n">old_end</span><span class="p">;</span> <span class="n">old_pte</span><span class="o">++</span><span class="p">,</span> <span class="n">old_addr</span> <span class="o">+=</span> <span class="n">PAGE_SIZE</span><span class="p">,</span>
                                   <span class="n">new_pte</span><span class="o">++</span><span class="p">,</span> <span class="n">new_addr</span> <span class="o">+=</span> <span class="n">PAGE_SIZE</span><span class="p">)</span> <span class="p">{</span>
                <span class="k">if</span> <span class="p">(</span><span class="n">pte_none</span><span class="p">(</span><span class="o">*</span><span class="n">old_pte</span><span class="p">))</span>
                        <span class="k">continue</span><span class="p">;</span>
                <span class="n">pte</span> <span class="o">=</span> <span class="n">ptep_get_and_clear</span><span class="p">(</span><span class="n">mm</span><span class="p">,</span> <span class="n">old_addr</span><span class="p">,</span> <span class="n">old_pte</span><span class="p">);</span>
                <span class="n">pte</span> <span class="o">=</span> <span class="n">move_pte</span><span class="p">(</span><span class="n">pte</span><span class="p">,</span> <span class="n">new_vma</span><span class="o">-&gt;</span><span class="n">vm_page_prot</span><span class="p">,</span> <span class="n">old_addr</span><span class="p">,</span> <span class="n">new_addr</span><span class="p">);</span>

<span class="p">...</span>
                <span class="n">set_pte_at</span><span class="p">(</span><span class="n">mm</span><span class="p">,</span> <span class="n">new_addr</span><span class="p">,</span> <span class="n">new_pte</span><span class="p">,</span> <span class="n">pte</span><span class="p">);</span>
        <span class="p">}</span>
<span class="p">...</span>
</code></pre></div></div>

<p>What the hell are we doing in this loop ? We basically are moving the <strong>PTE</strong>, the <a href="https://www.kernel.org/doc/gorman/html/understand/understand006.html#htoc16">Page Table Entries</a> of the mmap’ed region somewhere else. These are basically long integers assigned to each pages.</p>

<p>We’re pretty done here: the kernel never actually <em>touched</em> a single bit of memory related to our allocated block, and just swapped few bits to relocate the entire region elsewhere.</p>

<p>We may argue that we are <em>still</em> <em>O(N)</em> technically, but</p>

<ul>
  <li>instead of copying entire pages of 4KB (or 2MB for huge pages), we are swapping integers inside the kernel structure</li>
  <li>we are touching hot structures in the kernel, neither <em>cold</em> memory, nor bytes swapped <em>on disk</em></li>
</ul>

<p>Therefore, we are <em>O(N)</em>, but with a huge divisor.</p>

<p>Oh, by the way, did you know that <em>O(N)</em> was in many cases misleading ?</p>

<p>In this case, N can be <strong>at most</strong> 2^48 (the maximum virtual space size). Most machines only deal with few gigabytes of memory, at most 64GB for most architectures, that is, 2^36 bytes. A huge page is 2^21 bytes, leaving 2^15 operations at most for a <code class="language-plaintext highlighter-rouge">mremap</code> (yes, that’s <em>only</em> 32768 operations).</p>

<p>Well, the worst case scenario is <em>cheap</em> in all cases, and this is what I suspected from the beginning.</p>

<p><em>Further read: do not hesitate to have a look at Gorman’s book <a href="https://www.kernel.org/doc/gorman/pdf/understand.pdf">Understanding the Linux Virtual Memory Manager</a>.</em></p>

<p><strong>TL;DR</strong>: do not worry, be lazy, and trust <code class="language-plaintext highlighter-rouge">realloc</code>.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Security is Painful]]></title>
    <link href="https://blog.httrack.com/blog/2014/03/30/security-is-painful/"/>
    <updated>2014-03-30T11:56:00+02:00</updated>
    <id>https://blog.httrack.com/blog/2014/03/30/security-is-painful</id>
    <content type="html"><![CDATA[<h2 id="where-do-you-need-security-">Where Do You Need Security ?</h2>

<p>Well, pretty everywhere. As an open-source programmer maintaining a website, I need:</p>

<ol>
  <li>A <a href="https://en.wikipedia.org/wiki/GNU_Privacy_Guard">GPG</a> key to sign sources for <a href="http://www.debian.org/">Debian</a></li>
  <li>An appropriate certificates to <a href="https://en.wikipedia.org/wiki/Code_signing">sign my code</a> on Windows</li>
  <li>An appropriate certificate for the <a href="https://en.wikipedia.org/wiki/HTTP_Secure">https</a> website versions, so that my users can use a secured version, involving multiple sub-domains, with dedicated certificates</li>
  <li>An appropriate certificate for the <a href="https://en.wikipedia.org/wiki/STARTTLS">smtp with starttls</a> server, allowing to have a bit more secure channel when sending and receiving emails (oh hay NSA!)</li>
  <li>The DNS <a href="https://en.wikipedia.org/wiki/Domain_Name_System_Security_Extensions">DNSSEC</a> features, securing a bit my DNS naming system</li>
</ol>

<p>You’re not obliged to secure everything, of course, but most of the stuff above is pretty standard nowadays, and it’s hard not to fulfill all these requirements.</p>

<h2 id="a-gpg-key-to-sign-sources-on-linux">A GPG key to sign sources on Linux</h2>

<p><a href="https://en.wikipedia.org/wiki/Debian">Debian</a> involves thousands of volunteers across the world collaborating to build a complete operating system, and one key aspect is <strong>trust</strong> among developers. It means that each developer needs to pass certain validation steps to be <em>accepted</em>, and this includes demonstrating your technical skills and your ability to fit the organization <a href="http://www.debian.org/social_contract">spirit</a>. Another requirement is to <strong>identify</strong> who is responsible for packaging a program (in my case, httrack), as Debian needs to clearly know who uploaded what.</p>

<p>This seems a bit odd to some people: in a virtual world where anonymity is often considered a requirement, and a right - especially in the open-source community - Debian asks people to identify themselves, and identify the work they do.</p>

<p>But the reason behind is trust and accountability: Debian (and all derivative distributions based on its code base) is used by people who would have a hard time using untrusted <em>binaries</em>. What if a <em>backdoor</em> is injected somewhere, for example ? How can we be sure that uploaded packages haven’t been compromised somehow ? Trust is not an option when privacy and security of users is involved (especially in countries where what you may write on twitter can lead you in jail)</p>

<p>What does it mean for a developer like me ?</p>

<ul>
  <li>I had to create a GPG key through <code class="language-plaintext highlighter-rouge">gpg --gen-key</code> (well, I already <a href="http://pgp.mit.edu/pks/lookup?op=vindex&amp;search=0xDCF8D65B03F35988">had one</a>)</li>
  <li>I asked three existing Debian developers to meet them <em>in real life</em> (and gladly paid them a beer! :p) so that they can verify my identity (ie. at least one <em>identity card</em>), and <a href="http://www.debian.org/events/keysigning">sign</a> my GPG key</li>
  <li>I need to sign produced sources each time I am doing a release, before uploading them to Debian - uploads won’t be accepted otherwise</li>
</ul>

<p>Oh, by the way, I’ll need to update my key - Debian will soon require to have <a href="https://lists.debian.org/debian-devel-announce/2014/03/msg00003.html">more secure</a> keys by discarding the old 1024 “Diffie” keys. This will involve <a href="https://www.apache.org/dev/key-transition.html">key transition</a> steps, including meeting folks at Debian again to re-sign my key.</p>

<h2 id="appropriate-certificates-to-sign-the-windows-installer-version">Appropriate certificates to sign the Windows installer version</h2>

<p>If you do not want your users to have the creepy “<em>The publisher could not be verified, are you sure you want to run this software ?</em>” alert message when attempting to install your program, you need to sign at least the installer.</p>

<p>Microsoft provides a handful utility called <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/aa387764%28v=vs.85%29.aspx"><code class="language-plaintext highlighter-rouge">signtool.exe</code></a> to sign executables to be distributed:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s2">"C:</span><span class="se">\P</span><span class="s2">rogram Files</span><span class="se">\M</span><span class="s2">icrosoft SDKs</span><span class="se">\W</span><span class="s2">indows</span><span class="se">\v</span><span class="s2">6.0A</span><span class="se">\B</span><span class="s2">in</span><span class="se">\s</span><span class="s2">igntool.exe"</span> sign /f <span class="s2">"certificate.pfx"</span> /t http://timestamp.verisign.com/scripts/timstamp.dll myprogram.exe
</code></pre></div></div>

<p>The only problem is that we need an <em>official</em> certificate to do that. Want to have a free <a href="https://en.wikipedia.org/wiki/Public-key_infrastructure">PKI</a> like what we have on Debian ? NOPE guys, you need those expensive little pieces of numbers - basically a simple certificate will cost you a thousand dollars. Per year. Yep. Per year.</p>

<p>Fortunately for me, <a href="https://www.certum.eu/">Certum</a> is nice enough to provide <a href="https://www.certum.eu/certum/cert,offer_en_open_source_cs.xml">free certificates</a> to open-source developers. Thank you, guys!</p>

<p>We only have to create a signing request certificate:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openssl req <span class="nt">-new</span> <span class="nt">-newkey</span> rsa:2048 <span class="nt">-keyout</span> signing.pem <span class="nt">-out</span> signing.pem
</code></pre></div></div>

<p>And copy/paste the certificate request part to the provider.</p>

<p>We then need to convert the produced PEM certificate into a useable PFX one for <code class="language-plaintext highlighter-rouge">signtool.exe</code>:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openssl pkcs12 <span class="nt">-export</span> <span class="nt">-out</span> certificate.pfx <span class="nt">-inkey</span> signing.pem <span class="nt">-in</span> certificate.pem
</code></pre></div></div>

<h2 id="appropriate-certificates-for-https">Appropriate certificates for https</h2>

<p>If you have an online store, or are selling something, having a secured version of your website is just an absolute requirement.</p>

<p>But, increasingly, https is becoming the de-facto standard (especially after the PRISM scandal, and <em>especially</em> considering that <em>nothing will change</em> on the government’s side) pretty everywhere. The upcoming HTTP/2.0 protocol may even make https a <a href="http://www.theregister.co.uk/2013/11/14/http_20_encryption_proposal_sparks_hot_debate/">requirement</a> for all sites.</p>

<p>Having a secured website is unfortunately conditioned to the <a href="http://www.theregister.co.uk/2011/04/11/state_of_ssl_analysis/">broken and expensive</a> HTTPS public key infrastructure.</p>

<p>Fortunately enough, the guys at <a href="http://www.startssl.com/">StartSSL</a> also offer free SSL certificates for everyone. These are “low” level certificates, but this is perfectly fine for me (no, I don’t want the <em>golden-platinum certificate</em>). Thank you too, guys!</p>

<p>Things are a bit complicated when actually setting up Apache.</p>

<p>First, each virtual host needs its certificate:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SSLCertificateFile /etc/apache2/ssl/certs/my-server.pem
</code></pre></div></div>

<p>We need to get the StartSSL certificate chain locally:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wget http://www.startssl.com/certs/sub.class1.server.ca.pem
wget http://www.startssl.com/certs/ca.pem -O startssl-ca.pem
wget --no-check-certificate https://www.startssl.com/certs/ca-bundle.pem -O startssl-ca-bundle.pem
</code></pre></div></div>

<p>Then, spend few hours trying to figure out how to mitigate the various known attacks against SSL. Darn. Sheesh. Please give me some feedback if you thing the setup below can be improved :)</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Enable SSL
SSLEngine on

# Disable compression (CRIME attack)
# "Available in httpd 2.2.24 and later"
# SSLCompression off

# Enable only secure protocols: TLSv1, but not SSLv2/SSLv3
SSLProtocol all -SSLv2 -SSLv3

# Perfect forward secrecy + BEAST mitigation
# See &lt;https://community.qualys.com/blogs/securitylabs/2013/08/05/configuring-apache-nginx-and-openssl-for-forward-secrecy&gt;
SSLCipherSuite EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!ADH:!SSLv2:+HIGH:-MEDIUM

# Pay attention to the order ciphers are specified
SSLHonorCipherOrder On
SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown

# Certificate global settings
SSLVerifyClient none
SSLCACertificateFile /etc/apache2/ssl/certs/startssl-ca.pem
SSLCertificateChainFile /etc/apache2/ssl/certs/sub.class1.server.ca.pem
</code></pre></div></div>

<p>You can check whether your setup is fine, using openssl as commandline tool:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openssl s_client <span class="nt">-CApath</span> /etc/ssl/cert <span class="nt">-showcerts</span> <span class="nt">-connect</span> www.example.com:443
</code></pre></div></div>

<p>And you can also <a href="https://www.ssllabs.com/ssltest/">use online tools</a> to control used ciphers, options… if you manage to have 100% to all tests, congratulations!</p>

<h2 id="smtp-with-certificates">SMTP with certificates</h2>

<p>Yes, you also need to pay a certificate for that. The SSL PKI is definitely broken and mostly a <em>scam</em>. Haha.</p>

<p>Fortunately enough, we may use a free certificate for that, too :)</p>

<p>Note that securing the SMTP transactions is <strong>not</strong> considered <em>sufficient</em> when exchanging sensitive information - use <a href="https://en.wikipedia.org/wiki/GNU_Privacy_Guard">GPG</a>, guys!</p>

<p>For sendmail, I ended up with something like this:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>define(`CERT_DIR', `/etc/mail/certs')
define(`confCACERT_PATH', `CERT_DIR')
define(`confCACERT', `CERT_DIR/CAcert.pem')
define(`confSERVER_CERT', `CERT_DIR/MYcert.pem')
define(`confSERVER_KEY', `CERT_DIR/MYkey.pem')
define(`confCLIENT_CERT', `CERT_DIR/MYcert.pem')
define(`confCLIENT_KEY', `CERT_DIR/MYkey.pem')
TRUST_AUTH_MECH(`LOGIN PLAIN CRAM-MD5 DIGEST-MD5')

dnl Do not choke on missing/invalid client cert
define(`confTLS_SRV_OPTIONS', `V')

LOCAL_CONFIG
O CipherList=ALL
</code></pre></div></div>

<p>It took me a while to understand that the missing <code class="language-plaintext highlighter-rouge">define('confTLS_SRV_OPTIONS', 'V')</code> line was the culprit for rejecting some mails…</p>

<p>One last check to ensure your server is correctly setup:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openssl s_client <span class="nt">-CApath</span> /etc/ssl/cert <span class="nt">-showcerts</span> <span class="nt">-connect</span> smtp.example.com:25 <span class="nt">-starttls</span> smtp
</code></pre></div></div>

<h2 id="securing-the-dns-with-dnssec">Securing the DNS with DNSSEC</h2>

<p>At last, securing a DNS server becomes increasingly important, too, especially when governments attempt to <a href="http://www.bortzmeyer.org/dns-routing-hijack-turkey.html">hijack</a> it. <a href="https://en.wikipedia.org/wiki/Domain_Name_System_Security_Extensions">DNSSEC</a> is not yet <em>mainstream</em> unfortunately, and operating systems can not enforce its use <em>yet</em> (which means that evil dictatorships can still intercept public DNS and corrupt the replies).</p>

<p>DNSSEC seems a bit cumbersome to implement at first glance, but at least the system rely on existing DNS infrastructure, and will not cost you additional fees. You’ll find numerous useful FAQ and blogs explaining how to deploy this thing.</p>

<p>You first need to adjust a bit your own DNS server settings to handle DNSSEC. With BIND 9, this can be done through built-in features:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dnssec-enable yes;
dnssec-validation auto;
dnssec-lookaside auto;
managed-keys-directory "/var/cache/bind";
</code></pre></div></div>

<p>Then, you need a <a href="https://en.wikipedia.org/wiki/Domain_Name_System_Security_Extensions#Key_management">Key Signink Key (aka. “KSK”)</a> for each domain you own. The key, to be renewed every two years or so, is meant to sign other keys. The idea behind is that this key is to be declared to the upstream registrar, through your own domain administration system, and the registrar will sign it, to certify that it is <em>legit</em>. This allows to build a trust chain up to the root DNSSEC authority, whose certificate is known by DNS resolvers. With this key, you’ll be able to create a shorter-lifespan key, the <em>Zone Signing Keys (aka. “ZSK”)</em>, aimed to sign the various entries in your DNS zone. This shorter-lifespan key will be put in your DNS zone among other entries, as a DNSKEY entry (the record contains the public key side).</p>

<p>Here comes the KSK:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dnssec-keygen <span class="nt">-r</span> /dev/urandom <span class="nt">-a</span> RSASHA256 <span class="nt">-b</span> 4096 <span class="nt">-n</span> ZONE <span class="nt">-f</span> KSK httrack.com
</code></pre></div></div>

<p>And, at last, you need to create regularly (every two or three months) a new ZSK:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dnssec-keygen <span class="nt">-K</span> dnssec/zsk <span class="nt">-r</span> /dev/urandom <span class="nt">-a</span> RSASHA256 <span class="nt">-b</span> 2048 <span class="nt">-n</span> ZONE httrack.com
</code></pre></div></div>

<p>And do not forget to sign your zone each time you need to modify it, using the ZSK key:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dnssec-signzone <span class="nt">-K</span> dnssec/zsk <span class="nt">-e</span> +3024000 httrack.com
service bind9 reload
</code></pre></div></div>

<p>The <em>.signed</em> versions of the DNS zones will have to be used in place of the previous unsigned ones (<code class="language-plaintext highlighter-rouge">named.conf</code>), of course.</p>

<p>The whole setup is not totally straightforward, and will probably cause headache to newcomers. Using <a href="http://dnsviz.net/">online tools</a> to verify the DNS signed records or to check the <a href="http://dnssec-debugger.verisignlabs.com/">DNSSEC setup</a> will be extremely helpful.</p>

<p><strong>TL;DR</strong>: Security is sometimes painful, but this must not stop you. Your users are trusting you!</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[What Books Really Helped You ?]]></title>
    <link href="https://blog.httrack.com/blog/2014/03/23/what-books-really-helped-you/"/>
    <updated>2014-03-23T09:15:00+01:00</updated>
    <id>https://blog.httrack.com/blog/2014/03/23/what-books-really-helped-you</id>
    <content type="html"><![CDATA[<h2 id="helped-you-">Helped You ?</h2>

<p>There are countless books in computer science (and related domains), but it is sometimes hard to pick the <em>right</em> one, the one which provides an <em>easy access</em> to a domain, but at the same time <em>helps you</em> to embrace it.</p>

<p>Algorithms is a typical case, but not the only one: a good professor will not teach you more <em>advanced</em> algorithms, or more <em>efficient</em> ones. A good professor will make <em>complicated</em> things <em>less complicated</em>. An excellent one will turn them into <em>simple</em> things. A bad one will not only turn things <em>even more complicated</em>, but will also hinder any further reflexion, because he’ll convince you that this is not something <em>for you</em> (ie. you are probably too <em>limited</em> for that).</p>

<p>Books are more or less like professors: there are countless of badly written ones, but at the end very few <em>excellent</em> ones.</p>

<p>The motivation behind reading books can vary:</p>

<ul>
  <li>beginners wanting to teach themselves</li>
  <li>computer science students who want a good complement</li>
  <li>experienced programmers who want to refresh a little bit their memory</li>
  <li>experienced programmers who are willing to be interviewed for a new job (Google et al. are famous for asking algorithm-related questions)</li>
</ul>

<p>Let’s review what I enjoyed reading in the past few years.</p>

<h2 id="my-personal-list">My Personal List</h2>

<p>Yes, this list is personal, non-exhaustive, with probably some questionable choices. Some people may want to complete it, or will disagree with several ones, but hey, I think it’s a good start anyway.</p>

<ul>
  <li>
    <p><a href="https://en.wikipedia.org/wiki/The_C_Programming_Language">The C programming language</a> by <a href="https://en.wikipedia.org/wiki/Brian_Kernighan">Brian Kernighan</a> and <a href="https://en.wikipedia.org/wiki/Dennis_Ritchie">Dennis Ritchie</a>. Probably the most controversial one, but this book was the first C-language-related book I read, at a time Internet was not <em>mainstream</em>, and it was a great pleasure to switch from a gfa-basic/assembly developer to a C developer in only few weeks. I would probably not recommend the book nowadays, but I am really grateful for the (possibly imperfect) first C “professor” I had, and this book is one of the main reasons why I am writing C today.</p>
  </li>
  <li>
    <p><a href="https://www.schneier.com/book-applied.html">Applied Cryptography</a> by <a href="https://en.wikipedia.org/wiki/Bruce_Schneier">Bruce Schneier</a> was also my first cryptography-related book and even if I’m still (very) far from being an expert in this field, it really helped me a lot understanding many of the concepts behind it. I would really recommend this book to anyone interested by cryptography (and I would say that <em>everybody should be interested by cryptography</em>, and not only programmers)</p>
  </li>
  <li>
    <p><a href="http://algs4.cs.princeton.edu/home/">Algorithms</a> by <a href="https://en.wikipedia.org/wiki/Robert_Sedgewick_%28computer_scientist%29">Robert Sedgewick</a> and <a href="http://www.cs.princeton.edu/~wayne/research/">Kevin Wayne</a> is without hesitation the best algorithm book I have ever read, and I would gladly recommend it to anyone - and by <em>anyone</em> I really mean <strong>anyone</strong>: beginners, students, confirmed developers… I was first a bit <em>disappointed</em> by the rather basic level of the first chapter (quickly reviewing fundamental concepts, including <em>for</em> loops, etc.), but this is a key point of this book: the progression is uniform, and before you even realize it, more complex things appear. I have been studying algorithms during my college years, and I must admit that some of them totally flew over my head ; including <a href="https://en.wikipedia.org/wiki/Dijkstra%27s_algorithm">Dijkstra’s algorithm</a>, something I painfully implemented the first time when I was still a student (and I did not really understand the underlying concepts at that time, I must admit - this was bruteforce coding for me). In <em>Algorithms</em>, Dijkstra is not something which appears from the void, but is within a logical difficulty progression - and this book taught me how <em>simple</em> Dijkstra’s algorithm was. Yes, really simple: a direct graph search through a priority queue, with edge relaxation - after reading few chapters, I guarantee that this is a really simple thing do to. More generally, very intimidating algorithms many developers are <em>afraid</em> to check out are explained in such way that not only it makes your fears disappear, but gives you the trust to explore further domains (did you know that <em>red-black-trees</em> were not as crazy as they seemed ?). Ah, and a last note: algorithms in this books are written in Java, but are easily adaptable to other languages (in my case, C and C++)</p>
  </li>
  <li>
    <p><a href="http://www.hackersdelight.org/">Hacker’s Delight</a> by Henry S. Warren is a delicious book for bit-fiddling fans and more generally hackers (and by hackers I do not mean <em>script kiddies trying to deface the NSA website</em>, but <a href="http://www.catb.org/esr/faqs/hacker-howto.html">hackers</a>). Have you ever wondered how to count the number of leading ones in an integer without spending your time in a loop ? Or how to divide an integer number <em>without</em> actually using the division operation (slow) but only the multiply one ? This book is not only a collection of good recipes, it also explains the maths behind them. I strongly recommend this book to all advanced programmers, but also to anyone wanting to dive inside the numbers behind our computers.</p>
  </li>
  <li>
    <p><a href="http://www.cs.bell-labs.com/cm/cs/pearls/">Programming Pearls</a> by <a href="https://en.wikipedia.org/wiki/Jon_Bentley">Jon Bentley</a> is a really enjoyable book on algorithms - and <em>beyond</em> algorithms - with personal and very insightful <em>war stories</em>, entertaining remarks, and more. I really recommend this little gem, which will teach you how simple and powerful a markov chain can be, for example. Oh, and Jon Bentley is also the author of the excellent <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.11.8470&amp;rep=rep1&amp;type=pdf">long common strings</a> data compression (which can be viewed as a “low frequency” compression algorithm, compared to classical “high frequency” algorithms such as zlib, which basically means that it can be piped to a secondary compression, with a combined ratio)</p>
  </li>
  <li>
    <p><a href="http://livre.g6.asso.fr/index.php?title=Main_Page">IPv6 Théorie et Pratique</a> by <a href="http://g6.asso.fr">G6 association</a> is a French (yes, sorry guys) book which taught me everything on <a href="https://en.wikipedia.org/wiki/IPv6">IPv6</a> at a time IPv6 was <em>really</em> not mainstream (well, it is still not mainstream today, cough cough cough…). It is very unfortunate O’Reilly France disappeared, but at least this book is still online, and I would recommend it too.</p>
  </li>
</ul>

<h2 id="what-about-videos-">What About Videos ?</h2>

<p>Oh yes, some videos can also be extremely helpful.</p>

<ul>
  <li>
    <p>For a very good and non-experts oriented introduction to cryptography, I would strongly recommend the <a href="https://www.youtube.com/playlist?list=PLB4D701646DAF0817">Gambling with Secrets</a> series, and the <a href="https://www.youtube.com/playlist?list=PLbg3ZX2pWlgLoapF5VvM_8h5OR-XW9pbr">RSA Encryption</a> one. These small entertaining videos are really great (even for non-developers)</p>
  </li>
  <li>
    <p>An excellent <a href="https://www.coursera.org/course/algs4partI">algorithm course</a> by Robert Sedgewick and Kevin Wayne which is a good complement to the <a href="http://algs4.cs.princeton.edu/home/">Algorithms</a> book. Robert Sedgewick is an excellent professor, and the courses videos are intended for anyone (including beginners). Do not hesitate to enroll to this course, one more time: this is really <em>worth it</em>.</p>
  </li>
</ul>

<p><strong>TL;DR</strong>: No, really, sometimes reading is the shortest path!</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Embrace Unicode (And Do Not Worry)]]></title>
    <link href="https://blog.httrack.com/blog/2014/03/16/embrace-unicode-and-do-not-worry/"/>
    <updated>2014-03-16T10:00:00+01:00</updated>
    <id>https://blog.httrack.com/blog/2014/03/16/embrace-unicode-and-do-not-worry</id>
    <content type="html"><![CDATA[<h2 id="the-fear-of-unicodeutf-8">The Fear of Unicode/UTF-8</h2>

<p>As a programmer - especially as a C/C++ programmer - you might be reluctant to embrace the Unicode beast. You vaguely know that <code class="language-plaintext highlighter-rouge">wchar.h</code> contains various functions to handle the dirty things, but it does not make things clearer. In Java, programmers are using <code class="language-plaintext highlighter-rouge">String</code> without troubles, until they want to convert them from/to arrays of bytes (<em>hey, how Java is supposed to handle supplemental characters ?</em>)</p>

<p>Unicode is very simple in its essence, even if you can make things unreasonably complex (have you ever <a href="http://www.unicode.org/reports/tr15/">decomposed</a> an Unicode character ?): this is basically the extension of the ASCII concept (assigning a number to a character) to fit more characters.</p>

<h2 id="the-simplicity-behind-the-complexity">The Simplicity Behind The Complexity</h2>

<p>Yes, Unicode is rather simple if you keep things simple.</p>

<p>Let’s begin with some concepts:</p>

<ul>
  <li>
    <p><a href="http://www.unicode.org">Unicode</a>: a consortium aimed to standardize all possible characters in use on Earth (even the most improbable languages - but this <a href="http://www.unicode.org/alloc/nonapprovals.html">does not include</a> <a href="https://en.wikipedia.org/wiki/Klingon">Klingon</a>), such as latin characters, Chinese, Arabic, etc., and assign them a <strong>number</strong>. Numbers from 0 to 127 are the standard 7-bit ASCII, and code points above are non-ascii characters, obviously (the first range assigned above being latin letters <em>with</em> accents)</p>
  </li>
  <li>
    <p><a href="https://en.wikipedia.org/wiki/UTF-8">UTF-8</a>: a way to encode an Unicode character code into a sequence of bytes ; 7-bit ASCII is encoded “as is”, using 7-bit ASCII (ie. the code point for the letter “A”, which is the letter 65 in the Unicode world, is encoded into the ASCII byte “A”, which is <em>also</em> 65 in the ASCII world), and the rest is using a sequence of bytes in the range (in hex) <code class="language-plaintext highlighter-rouge">80</code>..<code class="language-plaintext highlighter-rouge">FD</code> (<code class="language-plaintext highlighter-rouge">FE</code> and <code class="language-plaintext highlighter-rouge">FF</code> are never used, which is pretty cool, because this greatly helps making the difference between an UTF-8 stream and other encodings)</p>
  </li>
  <li>
    <p>UCS-2: encoding of Unicode points as 16-bit wide characters - basically, an array of <code class="language-plaintext highlighter-rouge">uint16_t</code> - note that you may only encode Unicode points from 0 to 65535 (something which is called the <a href="https://en.wikipedia.org/wiki/Plane_%28Unicode%29#Basic_Multilingual_Plane">“<em>Basic Multilingual Plane</em>”</a>, or <em>BMP</em>). This is generally used internally only - most people won’t store or send raw arrays of <code class="language-plaintext highlighter-rouge">uint16_t</code>. You may have two flavors: UCS-2BE (big-endian flavor), or UCS-2LE (little-endian flavor), and a common trick to detect the endianness is to put a <a href="https://en.wikipedia.org/wiki/Byte_order_mark"><em>zero-width no-break space</em></a> (sometimes called <em>byte order mark</em>) as first invisible character in a text stream, whose code point is <code class="language-plaintext highlighter-rouge">FEFF</code>. The “reverse” code point <code class="language-plaintext highlighter-rouge">FFFE</code> is <em>invalid</em> in Unicode, and should never appear anywhere. Oh, did I tell you that <code class="language-plaintext highlighter-rouge">FE</code> and <code class="language-plaintext highlighter-rouge">FF</code> never appear in an UTF-8 stream previously ? If you use the <em>byte order mark</em>, you’ll never confuse an UCS stream with an UTF-8 one.</p>
  </li>
  <li>
    <p>UCS-4: encoding of Unicode points as 32-bit wide characters - basically, an array of <code class="language-plaintext highlighter-rouge">uint32_t</code>. You also have two flavors: UCS-4BE (big-endian flavor), or UCS-4LE (little-endian flavor), and the byte-order-mark trick is also used in many places.</p>
  </li>
  <li>
    <p><a href="https://en.wikipedia.org/wiki/UTF-16">UTF-16</a>: encoding of Unicode points as 16-bit wide characters (like UCS-2), except that Unicode points beyond the <a href="https://en.wikipedia.org/wiki/Plane_%28Unicode%29#Basic_Multilingual_Plane"><em>Basic Multilingual Plane</em></a> are encoded as two 16-bit characters in the <em>surrogate</em> range. This is a bit wicked: we are encoding a character using two characters, and each of them are encoded as 16-bit characters. You can treat an UTF-16 stream as a regular UCS-2 stream - in this case, code points beyond the <em>Basic Multilingual Plane</em> will be seen as a pair of unknown-class characters (<em>surrogates</em>).</p>
  </li>
  <li>
    <p><a href="https://en.wikipedia.org/wiki/UTF-32">UTF-32</a>: this is actually the synonym for UCS-4 (and is in my opinion an confusing naming: this is <em>not</em> a variable length encoding). Oh, and you are <em>not</em> supposed to have <em>surrogates</em> (used for UTF-16, see above) inside an UCS-4/UTF-32 stream.</p>
  </li>
</ul>

<h2 id="more-precisely-what-about-utf-8-">More Precisely, What About UTF-8 ?</h2>

<p>Let’s have a look as UTF-8, which is generally the preferred encoding for storing and sending data (most RFC are using UTF-8 as default encoding nowadays).</p>

<p>The good news is that 7-bit ASCII is a subset of UTF-8: nothing will change on this side.</p>

<p>For the rest, UTF-8 is a very simple and smart encoding (<a href="https://en.wikipedia.org/wiki/Ken_Thompson">Ken Thompson</a> is behind this fine piece of work), and decoding it is not really hard.</p>

<p>If a character outside the 7-bit range is used, UTF-8 will encode it as a byte sequence. For example, the “<a href="http://unicode.org/cldr/utility/character.jsp?a=00e9">é</a>” letter (latin letter e with acute accent), which is in the Unicode world the point <code class="language-plaintext highlighter-rouge">E9</code> (in hexadecimal - that is, 233 in decimal), will be encoded as bytes <code class="language-plaintext highlighter-rouge">C3</code> and <code class="language-plaintext highlighter-rouge">A9</code> (<code class="language-plaintext highlighter-rouge">C3 A9</code>) (which is displayed as “Ã©” - something you’ll learn to recognize and which generally is a sign of UTF-8 not being recognized by a client). The “<a href="http://www.unicode.org/cgi-bin/GetUnihanData.pl?codepoint=59eb">姫</a>” Chinese character (“princess”), which is in the Unicode world the point <code class="language-plaintext highlighter-rouge">59EB</code> (23019 in decimal), will be encoded as <code class="language-plaintext highlighter-rouge">E5 A7 AB</code> (displayed as “å§«” if your display client or browser is having troubles recognizing it).</p>

<p>To understand UTF-8, you just need to know one thing: for a given byte, visualize its 8 bits, and count the <strong>number of leading “1”</strong> (ie. the number of “1” before the first “0”). This is the only information you need to know whether this byte is a single 7-bit ASCII character, the beginning of an UTF-8 sequence, or an intermediate byte within an UTF-8 sequence. Oh, and if the byte is the beginning of a sequence, you also know how long it is: its length is <em>precisely</em> the number of leading “1”. Isn’t that magic ?</p>

<p>Yep, this is it.</p>

<p>Basically,</p>

<ul>
  <li>no leading “1”: 7-bit ASCII (<code class="language-plaintext highlighter-rouge">0</code>..<code class="language-plaintext highlighter-rouge">7F</code>)</li>
  <li>one leading “1”: in the middle of a sequence</li>
  <li>two or more leading “1”: a sequence whose length is the number of leading “1”</li>
</ul>

<p>This is an extremely good property of UTF-8: you can easily cut an UTF-8 string/skip characters/count characters without bothering about actual code points. Only bother about leading “1”’s.</p>

<p>Oh, by the way: did you know that the <em>number of leading “1”</em> could be computed without using a loop ? Take a look at the <a href="http://www.hackersdelight.org/">Hacker’s Delight</a> recipes (do not hesitate to buy the book, it is worth it, and <em>recipes</em> are presented with their <em>mathematical</em> explanation - there is no pure magic out there)</p>

<p>Here’s the trick: (this is the number of leading <em>zeros</em>, but you probably have an idea on how to use this function to compute the number of leading <em>ones</em>)</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/* Hacker's delight number of leading zeros. */</span>
<span class="k">static</span> <span class="kt">unsigned</span> <span class="kt">int</span> <span class="nf">nlz8</span><span class="p">(</span><span class="kt">unsigned</span> <span class="kt">char</span> <span class="n">x</span><span class="p">)</span> <span class="p">{</span>
  <span class="kt">unsigned</span> <span class="kt">int</span> <span class="n">b</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>

  <span class="k">if</span> <span class="p">(</span><span class="n">x</span> <span class="o">&amp;</span> <span class="mh">0xf0</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">x</span> <span class="o">&gt;&gt;=</span> <span class="mi">4</span><span class="p">;</span>
  <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
    <span class="n">b</span> <span class="o">+=</span> <span class="mi">4</span><span class="p">;</span>
  <span class="p">}</span>

  <span class="k">if</span> <span class="p">(</span><span class="n">x</span> <span class="o">&amp;</span> <span class="mh">0x0c</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">x</span> <span class="o">&gt;&gt;=</span> <span class="mi">2</span><span class="p">;</span>
  <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
    <span class="n">b</span> <span class="o">+=</span> <span class="mi">2</span><span class="p">;</span>
  <span class="p">}</span>

  <span class="k">if</span> <span class="p">(</span><span class="o">!</span> <span class="p">(</span><span class="n">x</span> <span class="o">&amp;</span> <span class="mh">0x02</span><span class="p">)</span> <span class="p">)</span> <span class="p">{</span>
    <span class="n">b</span><span class="o">++</span><span class="p">;</span>
  <span class="p">}</span>

  <span class="k">return</span> <span class="n">b</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>

<p>We know how to recognize UTF-8 sequences, and we even know how to get their length. How to encode Unicode points ?</p>

<p>Encoding is extremely simple. It could not be simpler, actually: visualize your Unicode point as a stream of bits (starting from the leading “1”). You’ll encode them in the remaining space, in the first leading character, and in the remaining sequence. The remaining space within sequence bytes is the space after the leading “1” and the “0” delimiter.</p>

<p>For example, if the leading byte defines a 4-byte sequence, then it will start with four “1”, and a “0” delimiter. Three bits remains available for encoding (the first three bits of the Unicode point number, starting from the leading “1”, will be encoded there). Continuing characters (in sequence) have a leading “1” followed by the “0” delimiter: six bits are available for encoding.</p>

<p>Let’s go back to the “姫” Chinese character. This character is <a href="http://www.unicode.org/cgi-bin/GetUnihanData.pl?codepoint=59eb"><code class="language-plaintext highlighter-rouge">59EB</code></a> in the Unicode charts ; here’s how it is encoded as <code class="language-plaintext highlighter-rouge">E5 A7 AB</code> (and sorry for my lame Chinese writing skills!):</p>

<p><img src="/images/misc/utf8_encoding_of_59eb.png" alt="Encoding 59EB" /></p>

<h2 id="an-exercise-for-you">An Exercise For You!</h2>

<p>As an exercise, you can now easily rewrite the <a href="http://pubs.opengroup.org/onlinepubs/000095399/functions/getwchar.html"><code class="language-plaintext highlighter-rouge">getwchar()</code></a> function in an UTF-8 environment (locale).</p>

<p>Here’s my try (do not cheat! look at this solution <em>after</em> you tried!):</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#include</span> <span class="cpf">&lt;stdio.h&gt;</span><span class="cp">
#include</span> <span class="cpf">&lt;stdlib.h&gt;</span><span class="cp">
</span>
<span class="cp">#define UTF8_ERROR ( (int) (-2) )
</span>
<span class="cm">/* Hacker's delight number of leading zeros. */</span>
<span class="k">static</span> <span class="kt">unsigned</span> <span class="kt">int</span> <span class="nf">nlz8</span><span class="p">(</span><span class="kt">unsigned</span> <span class="kt">char</span> <span class="n">x</span><span class="p">)</span> <span class="p">{</span>
  <span class="kt">unsigned</span> <span class="kt">int</span> <span class="n">b</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>

  <span class="k">if</span> <span class="p">(</span><span class="n">x</span> <span class="o">&amp;</span> <span class="mh">0xf0</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">x</span> <span class="o">&gt;&gt;=</span> <span class="mi">4</span><span class="p">;</span>
  <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
    <span class="n">b</span> <span class="o">+=</span> <span class="mi">4</span><span class="p">;</span>
  <span class="p">}</span>

  <span class="k">if</span> <span class="p">(</span><span class="n">x</span> <span class="o">&amp;</span> <span class="mh">0x0c</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">x</span> <span class="o">&gt;&gt;=</span> <span class="mi">2</span><span class="p">;</span>
  <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
    <span class="n">b</span> <span class="o">+=</span> <span class="mi">2</span><span class="p">;</span>
  <span class="p">}</span>

  <span class="k">if</span> <span class="p">(</span><span class="o">!</span> <span class="p">(</span><span class="n">x</span> <span class="o">&amp;</span> <span class="mh">0x02</span><span class="p">)</span> <span class="p">)</span> <span class="p">{</span>
    <span class="n">b</span><span class="o">++</span><span class="p">;</span>
  <span class="p">}</span>

  <span class="k">return</span> <span class="n">b</span><span class="p">;</span>
<span class="p">}</span>

<span class="cm">/* Length of an UTF-8 sequence. */</span>
<span class="k">static</span> <span class="kt">size_t</span> <span class="nf">utf8_length</span><span class="p">(</span><span class="k">const</span> <span class="kt">char</span> <span class="n">lead</span><span class="p">)</span> <span class="p">{</span>
  <span class="k">const</span> <span class="kt">unsigned</span> <span class="kt">char</span> <span class="n">f</span> <span class="o">=</span> <span class="p">(</span><span class="kt">unsigned</span> <span class="kt">char</span><span class="p">)</span> <span class="n">lead</span><span class="p">;</span>
  <span class="k">return</span> <span class="n">nlz8</span><span class="p">(</span><span class="o">~</span><span class="n">f</span><span class="p">);</span>
<span class="p">}</span>

<span class="cm">/* Equivalent to getwchar() on an UTF-8 locale. */</span>
<span class="k">static</span> <span class="kt">int</span> <span class="nf">utf8_getchar</span><span class="p">()</span> <span class="p">{</span>
  <span class="k">const</span> <span class="kt">int</span> <span class="n">c</span> <span class="o">=</span> <span class="n">getchar</span><span class="p">();</span>
  <span class="k">const</span> <span class="kt">size_t</span> <span class="n">len</span> <span class="o">=</span> <span class="n">utf8_length</span><span class="p">(</span><span class="n">c</span><span class="p">);</span>
  <span class="k">if</span> <span class="p">(</span><span class="n">c</span> <span class="o">&lt;</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>  <span class="cm">/* EOF */</span>
    <span class="k">return</span> <span class="n">EOF</span><span class="p">;</span>
  <span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="n">len</span> <span class="o">==</span> <span class="mi">1</span><span class="p">)</span> <span class="p">{</span>  <span class="cm">/* ASCII */</span>
    <span class="k">return</span> <span class="n">UTF8_ERROR</span><span class="p">;</span>
  <span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="n">len</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>  <span class="cm">/* Error (in-sequence) */</span>
    <span class="k">return</span> <span class="n">c</span><span class="p">;</span>
  <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>  <span class="cm">/* UTF-8 */</span>
    <span class="kt">unsigned</span> <span class="kt">int</span> <span class="n">uc</span> <span class="o">=</span> <span class="n">c</span> <span class="o">&amp;</span> <span class="p">(</span> <span class="p">(</span><span class="mi">1</span> <span class="o">&lt;&lt;</span> <span class="p">(</span><span class="mi">7</span> <span class="o">-</span> <span class="n">len</span><span class="p">))</span> <span class="o">-</span> <span class="mi">1</span> <span class="p">);</span>
    <span class="kt">size_t</span> <span class="n">i</span><span class="p">;</span>
    <span class="k">for</span><span class="p">(</span><span class="n">i</span> <span class="o">=</span> <span class="mi">0</span> <span class="p">;</span> <span class="n">i</span> <span class="o">+</span> <span class="mi">1</span> <span class="o">&lt;</span> <span class="n">len</span> <span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
      <span class="k">const</span> <span class="kt">int</span> <span class="n">c</span> <span class="o">=</span> <span class="n">getchar</span><span class="p">();</span>
      <span class="cm">/* not EOF, and second bit shall always be cleared */</span>
      <span class="k">if</span> <span class="p">(</span><span class="n">c</span> <span class="o">!=</span> <span class="o">-</span><span class="mi">1</span> <span class="o">&amp;&amp;</span> <span class="p">(</span> <span class="n">c</span> <span class="o">&gt;&gt;</span> <span class="mi">6</span> <span class="p">)</span> <span class="o">==</span> <span class="mh">0x2</span><span class="p">)</span> <span class="p">{</span>
       <span class="n">uc</span> <span class="o">&lt;&lt;=</span> <span class="mi">6</span><span class="p">;</span>
       <span class="n">uc</span> <span class="o">|=</span> <span class="p">(</span><span class="n">c</span> <span class="o">&amp;</span> <span class="mh">0x3f</span><span class="p">);</span>
      <span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="n">c</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
        <span class="k">return</span> <span class="n">EOF</span><span class="p">;</span>
      <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
        <span class="k">return</span> <span class="n">UTF8_ERROR</span><span class="p">;</span>
      <span class="p">}</span>
    <span class="p">}</span>
    <span class="k">return</span> <span class="p">(</span><span class="kt">int</span><span class="p">)</span> <span class="n">uc</span><span class="p">;</span>
  <span class="p">}</span>
<span class="p">}</span>

<span class="kt">int</span> <span class="nf">main</span><span class="p">(</span><span class="kt">int</span> <span class="n">argc</span><span class="p">,</span> <span class="kt">char</span><span class="o">*</span> <span class="n">argv</span><span class="p">[])</span> <span class="p">{</span>
  <span class="k">for</span><span class="p">(;;)</span> <span class="p">{</span>
    <span class="k">const</span> <span class="kt">int</span> <span class="n">c</span> <span class="o">=</span> <span class="n">utf8_getchar</span><span class="p">();</span>
    <span class="k">if</span> <span class="p">(</span><span class="n">c</span> <span class="o">==</span> <span class="n">EOF</span><span class="p">)</span> <span class="p">{</span>
      <span class="k">break</span><span class="p">;</span>
    <span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="n">c</span> <span class="o">==</span> <span class="n">UTF8_ERROR</span><span class="p">)</span> <span class="p">{</span>
      <span class="n">fprintf</span><span class="p">(</span><span class="n">stderr</span><span class="p">,</span> <span class="s">"* UTF-8 error</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
    <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
      <span class="n">printf</span><span class="p">(</span><span class="s">"unicode character 0x%04x</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">c</span><span class="p">);</span>
    <span class="p">}</span>
  <span class="p">}</span>
  <span class="k">return</span> <span class="n">EXIT_SUCCESS</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Note that the above code can be improved a little bit, by unrolling the <code class="language-plaintext highlighter-rouge">for()</code> loop, and using few <code class="language-plaintext highlighter-rouge">switch</code>.</p>

<p><strong>TL;DR</strong>: Do not worry, and embrace UTF-8 and the Unicode world.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[What Are Your GCC Flags ?]]></title>
    <link href="https://blog.httrack.com/blog/2014/03/09/what-are-your-gcc-flags/"/>
    <updated>2014-03-09T17:32:00+01:00</updated>
    <id>https://blog.httrack.com/blog/2014/03/09/what-are-your-gcc-flags</id>
    <content type="html"><![CDATA[<h2 id="say-what-">Say What ?</h2>

<p>Your GCC build flags. Yes, this is actually an interesting question! I have been building code at the various places where I work for many years, on different architectures, and tweaking the build flags has always been an important task.</p>

<p>I mean, you probably are too experimented to <em>just</em> use something like:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcc <span class="nt">-c</span> fubar.c <span class="nt">-o</span> fubar.o
</code></pre></div></div>

<p>And you probably use <code class="language-plaintext highlighter-rouge">-W -Wall</code>, or additional flags, to tune the compiler behavior.</p>

<p>The idea behind is not only to carefully optimize produced bytecode, but also to improve its quality, and its security. Compiler warnings are critical to spot many programming errors or lethal typos that would otherwise consume days of debugging (and a big pile of money!).</p>

<p>Many beginners are still using nowadays the default <code class="language-plaintext highlighter-rouge">gcc</code> command without further tuning, and this always makes me cringe. Warning messages are not <em>annoying</em>, they are <em>useful</em>. And the minutes you spend fixing them (Note that I did not say <em>hide</em>, but <strong>fix</strong>) will spare you days or even months of nightmares.</p>

<p>Here are the flags we are using where I work:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcc <span class="nt">-pipe</span> <span class="nt">-m64</span> <span class="nt">-ansi</span> <span class="nt">-fPIC</span> <span class="nt">-g</span> <span class="nt">-O3</span> <span class="nt">-fno-exceptions</span> <span class="nt">-fstack-protector</span> <span class="nt">-Wl</span>,-z,relro <span class="nt">-Wl</span>,-z,now <span class="nt">-fvisibility</span><span class="o">=</span>hidden <span class="nt">-W</span> <span class="nt">-Wall</span> <span class="nt">-Wno-unused-parameter</span> <span class="nt">-Wno-unused-function</span> <span class="nt">-Wno-unused-label</span> <span class="nt">-Wpointer-arith</span> <span class="nt">-Wformat</span> <span class="nt">-Wreturn-type</span> <span class="nt">-Wsign-compare</span> <span class="nt">-Wmultichar</span> <span class="nt">-Wformat-nonliteral</span> <span class="nt">-Winit-self</span> <span class="nt">-Wuninitialized</span> <span class="nt">-Wno-deprecated</span> <span class="nt">-Wformat-security</span> <span class="nt">-Werror</span> <span class="nt">-c</span> source.c <span class="nt">-o</span> dest.o
</code></pre></div></div>

<p>Within these fancy flags, I was among the craziest tuner. Here’s the summary of what I shamelessly added:</p>

<ul>
  <li>
    <p><code class="language-plaintext highlighter-rouge">-pipe</code>
I prefer NOT to use temporary files when possible, and use pipes, for example when compiling preprocessed code into intermediate assembly source (.S). This has no real impact on performances (temporary files are generally put on some kind of tmpfs filesystem), but this is a bit cleaner.</p>
  </li>
  <li>
    <p><code class="language-plaintext highlighter-rouge">-fno-exceptions</code>
I don’t like C++ exceptions. And we don’t use them where I work. So let’s remove the overhead generated by unwinding code (ie. code and data aimed to allow the runtime to “rollback” function calls in case an exception is thrown, eg. object destructors that need to be called, etc.). Bonus: it may also produce a bit faster code. What if an exception is thrown (eg. by the runtime or the STL) by the way ? Well, <code class="language-plaintext highlighter-rouge">abort()</code> will be called instead, which is <em>perfectly fine</em> as thrown exceptions are either programming errors (out_of_range, bad_cast etc.) or critical conditions (<code class="language-plaintext highlighter-rouge">new</code> throwing bad_alloc)</p>
  </li>
  <li>
    <p><code class="language-plaintext highlighter-rouge">-fstack-protector -Wl,-z,relro -Wl,-z,now -Wformat-security</code>
These flags are actually ripped from the <a href="https://wiki.debian.org/Hardening">Debian Hardening Wiki</a>, and are aimed to detect stack smashing issues, have a read-only global offset table (preventing any attacks involving writing through the GOT), and various format strings suspicious usages.</p>
  </li>
  <li>
    <p><code class="language-plaintext highlighter-rouge">-fvisibility=hidden</code>
Ah yes, this one is cool. It changes the default extern symbol export mode in GCC to “hidden”. Basically, it means that “extern” symbols are visible by all units within a library, but not outside this library. If you ever wrote code on Windows environments, you probably remember those: <code class="language-plaintext highlighter-rouge">_declspec(dllexport)</code> and <code class="language-plaintext highlighter-rouge">_declspec(dllimport)</code>. Without them, symbols inside your DLL would not be exported (or imported). On POSIX systems, you export all <code class="language-plaintext highlighter-rouge">extern</code> symbols, and they become visible in your .so library. But this default mode has several drawbacks</p>
    <ul>
      <li>You are exporting internal functions you do not want people to mess with</li>
      <li>You may have internal symbols conflicting with other libraries</li>
      <li>You may have troubles having your code building on Windows “out of the box”</li>
    </ul>
  </li>
</ul>

<p>The last argument was actually the strongest one: the build would be broken several times per week because <em>someone forgot to properly export the symbols and the code was building fine on Linux but not on Windows</em>. For all these reasons, using <code class="language-plaintext highlighter-rouge">-fvisibility=hidden</code> (and properly exporting symbols using <code class="language-plaintext highlighter-rouge">__attribute__ ((visibility("default")))</code>) was a true relief!</p>

<ul>
  <li>
    <p><code class="language-plaintext highlighter-rouge">-Wpointer-arith</code>
We do not want to know the size of <code class="language-plaintext highlighter-rouge">void</code>, and we won’t use it!</p>
  </li>
  <li>
    <p><code class="language-plaintext highlighter-rouge">-Wformat-nonliteral</code>
Banishing “non literal format strings” was actually a sane decision. Especially when the format string source was some kind of user generated input - we’re not in 1990 anymore, and security issues involving string format can not be ignored anymore.</p>
  </li>
  <li>
    <p><code class="language-plaintext highlighter-rouge">-Winit-self</code>
This is actually a joke in the C standard.</p>
  </li>
</ul>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="n">i</span><span class="p">;</span>
</code></pre></div></div>

<p>Yes, this code is perfectly valid, and won’t even produce a single warning by default. And, of course, the behavior is unspecified – the variable <code class="language-plaintext highlighter-rouge">i</code> is left “initialized” with a garbage value. I have been bitten by this idiotic default behavior too often before turning on this specific warning. Shame on me.</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">-Werror</code>
Haha. This one was <strong>painful</strong> to propagate through all of our code. It basically breaks the build once you have a warning somewhere. Yes, the code won’t build unless it is <em>totally</em> warning-free. Yes, people were a bit upset at me at the beginning :), and they were even more upset when committing code that would break on a new compiler version (but not on an older one). This was painful. This was hard. But I did not give up! And at the end, the overall impact was <strong>tremendously</strong> positive.
    <ul>
      <li>Warnings emitted during build are useless unless someone check them regularly (and <code class="language-plaintext highlighter-rouge">grep</code>‘ping thousands of lines of logs is not a cool thing to do, and <em>nobody</em> is going to do that)</li>
      <li>As more warnings are emitted, and <em>ignored</em>, new warnings tend to be ignored too. As more warnings are emitted, people tend to give up at fixing them (<em>Hey! I’m not the janitor!</em>).</li>
      <li>Warnings are sometimes annoying, but most of the time they are <em>warnings</em> that <strong>you are doing something wrong</strong> (possibly <em>really wrong</em>)</li>
      <li>It is far better to spend five minutes fixing a warning than spending few months on a vicious bug (yes strict aliasing rules, I am talking about you)</li>
      <li>You can always disable specific warnings (<code class="language-plaintext highlighter-rouge">-Wno-*</code> flags) if needed!</li>
    </ul>
  </li>
</ul>

<p>Oh, and on the linker side, I did a bit of tweaking too:</p>

<ul>
  <li>
    <p><code class="language-plaintext highlighter-rouge">-Wl,-O1</code>
Did you know that you also have an optimization flag for the linker ? Now you know!</p>
  </li>
  <li>
    <p><code class="language-plaintext highlighter-rouge">-Wl,--discard-all</code>
Discard all local symbols, for the sake of library size.</p>
  </li>
  <li>
    <p><code class="language-plaintext highlighter-rouge">-Wl,--no-undefined</code>
Yes, I don’t like missing (unresolved) symbols at link time, even if this is actually a feature. It makes the Windows build easier too, by preventing behavioral differences between Linux/Windows.</p>
  </li>
  <li>
    <p><code class="language-plaintext highlighter-rouge">-Wl,--build-id=sha1</code>
This adds a fancy “build identifier” in all produced modules, which has a deterministic value (ie. two builds with the same sources and build flags will produce the same identifier), allowing to control whether or not you managed to rebuild from scratch and produce the same binaries.</p>
  </li>
  <li>
    <p><code class="language-plaintext highlighter-rouge">-rdynamic</code> (note: this actually passes the <code class="language-plaintext highlighter-rouge">-export-dynamic</code> flag to <code class="language-plaintext highlighter-rouge">ld</code>)
This linker option allows you to have more exported symbols within the dynamic symbol table, which is actually a nice thing when attempting to have readable backtraces!</p>
  </li>
</ul>

<p>That’s all, folks.</p>

<p><strong>TL;DR</strong>: Take the time to read carefully <em>and</em> understand the <code class="language-plaintext highlighter-rouge">man gcc</code> and <code class="language-plaintext highlighter-rouge">man ld</code> pages. It is worth it.</p>

<p><em>Thanks to Andrew Hochhaus for changing -fvisibility=internal to -fvisibility=hidden.</em>
<em>Thanks for the insightful remarks to all <a href="https://news.ycombinator.com/item?id=7371806">hacker news</a> contributors, including additional flags and their advantages.</em></p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Fancy Standalone Visual C++ Compiler]]></title>
    <link href="https://blog.httrack.com/blog/2014/02/21/fancy-standalone-visual-c-plus-plus-compiler/"/>
    <updated>2014-02-21T18:41:00+01:00</updated>
    <id>https://blog.httrack.com/blog/2014/02/21/fancy-standalone-visual-c-plus-plus-compiler</id>
    <content type="html"><![CDATA[<h2 id="a-fancy-standalone-compiler-">A Fancy Standalone Compiler ?</h2>

<p>Yes. Many people like me are developing on multiple platforms, including Windows, but do not necessarily work <em>on Windows</em> for their primary platform. Besides, you may log-in on a random Windows machine with a very minimalistic setup (say, a new virtual machine, for example), and do not want to have a full compiler installed. You may also use multiple versions of a compiler (playing with Visual C++ 2008, 2010, 2012, 2013..), and switching between them may become a real burden.</p>

<p>For these many reasons, having a standalone compiler, say, ready-to-use on a network drive, is often a very handy solution.</p>

<h2 id="can-we-do-that-">Can We Do That ?</h2>

<p>Technically, using a full Visual C++ install deported on an external location, without actually installing anything, is not really supported (I would say rather untested, though)</p>

<p>Fortunately enough, Visual C++ is nice and well-educated enough to accept being used in a standalone fashion ; at least for the commandline tools, such as cl.exe or link.exe. Yay!</p>

<h2 id="the-recipe">The Recipe</h2>

<p>The recipe is almost the same for all Visual C++ releases (tested from 2005 to 2013): blindly copying important directories in a safe place, adjusting some DLL’s. I will describe the rough necessary steps for 2013 and 2010 - you’ll need to adapt a bit to fit your needs, sometimes find a bit in subfolders to catch some DLL’s, but this should not be too cumbersome.</p>

<ul>
  <li>
    <p>Start on a fresh machine. Preferably on a virtual machine, up-to-date (service packs). The machine can be trashed after creating the remote compiler, of course. Having a fresh install is quite handy to spot what has been installed, and to avoid being polluted by external DLL’s (such as external redistributables)</p>
  </li>
  <li>
    <p>Install a regular Visual C++ Express version. Yes, I’m using the <em>Express</em> flavor, as it is simpler to use, and has probably less licensing issues than the “paid” release. You will miss some components, though, such as ATL and MFC (but who use them nowadays ? Err, I may have still some MFC code lying around by the way …)</p>
    <ul>
      <li>For 2013: fetch the <a href="http://www.microsoft.com/en-us/download/details.aspx?id=40787"><em>Microsoft Visual Studio Express 2013 for Windows Desktop ISO (English)</em></a> binaries and mount the ISO (or uncompress everything with some tool).</li>
      <li>For 2010: fetch <em>Visual C++ 2010 Express</em>, and <em>Microsoft Windows SDK for Windows 7 and .NET Framework 4</em></li>
    </ul>
  </li>
</ul>

<p><br /></p>
<ul>
  <li>Create a standalone directory with compiler binaries (to be placed later on a network drive, for example, or on an USB key):
    <ul>
      <li><strong>vssrc</strong> is generally something like <em>C:\Program Files (x86)\Microsoft Visual Studio 12.0</em> or <em>C:\Program Files (x86)\Microsoft Visual Studio 10.0</em></li>
      <li><strong>mssdk</strong> is generally something like <em>C:\Program Files\Microsoft SDKs\Windows</em></li>
      <li><strong>programfiles</strong> is generally something like <em>C:\Program Files</em></li>
    </ul>
  </li>
</ul>

<p>Do not forget to install every possible updates, patches, service packs…</p>

<h3 id="visual-c-2013">Visual C++ 2013</h3>

<table>
  <thead>
    <tr>
      <th>Source</th>
      <th>Destination</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>vssrc</strong>\VC</td>
      <td>VC</td>
    </tr>
    <tr>
      <td><strong>vssrc</strong>\Common7\IDE 10.0\Common7\IDE\{mspdb*.dll}</td>
      <td>VC\bin\</td>
    </tr>
    <tr>
      <td><strong>vssrc</strong>\VC\redist\x86\Microsoft.VC120.CRT\*.dll</td>
      <td>VC\bin\</td>
    </tr>
    <tr>
      <td><strong>vssrc</strong>\VC\redist\x86\Microsoft.VC120.CRT\*.dll</td>
      <td>VC\bin\amd64</td>
    </tr>
    <tr>
      <td><strong>vssrc</strong>\VC\bin\{mspdb*.*}</td>
      <td>VC\bin\amd64</td>
    </tr>
    <tr>
      <td><strong>vssrc</strong>\VC\bin\msobj120.dll</td>
      <td>VC\bin\amd64</td>
    </tr>
    <tr>
      <td><strong>programfiles</strong>\Microsoft SDKs\Windows\v7.1A\Bin</td>
      <td>bin</td>
    </tr>
    <tr>
      <td><strong>programfiles</strong>\Microsoft SDKs\Windows\v7.1A\Include</td>
      <td>include</td>
    </tr>
    <tr>
      <td><strong>programfiles</strong>\Microsoft SDKs\Windows\v7.1A\Lib</td>
      <td>lib</td>
    </tr>
    <tr>
      <td><strong>programfiles</strong>\MSBuild\12.0\Bin</td>
      <td>NET\Framework</td>
    </tr>
    <tr>
      <td><strong>programfiles</strong>\MSBuild\12.0\Bin\amd64</td>
      <td>NET\Framework64</td>
    </tr>
  </tbody>
</table>

<p><br />
<br /></p>
<h3 id="visual-c-2010">Visual C++ 2010</h3>

<table>
  <thead>
    <tr>
      <th>Source</th>
      <th>Destination</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>vssrc</strong>\VC</td>
      <td>VC</td>
    </tr>
    <tr>
      <td><strong>vssrc</strong>\Common7\IDE\msobj100.dll</td>
      <td>VC\bin\</td>
    </tr>
    <tr>
      <td><strong>vssrc</strong>\Common7\IDE\mspdb100.dll</td>
      <td>VC\bin\</td>
    </tr>
    <tr>
      <td><strong>vssrc</strong>\Common7\IDE\mspdbcore.dll</td>
      <td>VC\bin\</td>
    </tr>
    <tr>
      <td><strong>vssrc</strong>\Common7\IDE\mspdbsrv.exe</td>
      <td>VC\bin\</td>
    </tr>
    <tr>
      <td><strong>mssdk</strong>\v7.1\Redist\VC..\msvcr100.dll</td>
      <td>VC\bin\</td>
    </tr>
    <tr>
      <td><strong>mssdk</strong>\v7.1\Redist\VC..\x64\msvcr100.dll</td>
      <td>VC\bin\amd64</td>
    </tr>
    <tr>
      <td><strong>programfiles</strong>\Microsoft SDKs\Windows\v7.1\Bin</td>
      <td>bin</td>
    </tr>
    <tr>
      <td><strong>programfiles</strong>\Microsoft SDKs\Windows\v7.1\Include</td>
      <td>include</td>
    </tr>
    <tr>
      <td><strong>programfiles</strong>\Microsoft SDKs\Windows\v7.1\Lib</td>
      <td>lib</td>
    </tr>
    <tr>
      <td>C:\Windows\Microsoft.NET\Framework</td>
      <td>NET\Framework</td>
    </tr>
    <tr>
      <td>C:\Windows\Microsoft.NET\Framework64</td>
      <td>NET\Framework64</td>
    </tr>
  </tbody>
</table>

<p><br />
In all cases, you may also want to have <a href="http://technet.microsoft.com/en-us/sysinternals/bb896653">Process Explorer</a> and <a href="http://www.dependencywalker.com/">Dependency Walker</a> within the same place, as they are really useful tools for a developer.</p>

<p>Watch out: 64-bit binaries may be placed in different subfolders:</p>

<ul>
  <li>x86_x64</li>
  <li>x64</li>
  <li>amd64</li>
  <li>x86_amd64</li>
</ul>

<p>Yes, Microsoft had some troubles using a single naming scheme for 64-bit :)</p>

<p>After that, you only need to set environment variables to have the 64-bit or 32-bit directories in your PATH in some kind of cmd script ; eg:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>@SET <span class="nv">VSINSTALLDIR</span><span class="o">=</span>X:<span class="se">\d</span>ata<span class="se">\m</span>y-standalone-compiler
@SET <span class="nv">VCINSTALLDIR</span><span class="o">=</span>%VSINSTALLDIR%<span class="se">\V</span>C
@SET <span class="nv">WindowsSdkDir</span><span class="o">=</span>%VSINSTALLDIR%<span class="se">\</span>

@set <span class="nv">PATH</span><span class="o">=</span>%VCINSTALLDIR%<span class="se">\B</span>IN<span class="se">\x</span>86_amd64<span class="p">;</span>%VCINSTALLDIR%<span class="se">\B</span>IN<span class="p">;</span>%VSINSTALLDIR%<span class="se">\b</span><span class="k">in</span><span class="se">\x</span>64<span class="p">;</span>%PATH%
@set <span class="nv">LIB</span><span class="o">=</span>%VCINSTALLDIR%<span class="se">\L</span>IB<span class="se">\a</span>md64<span class="p">;</span>%VSINSTALLDIR%<span class="se">\L</span>IB<span class="se">\x</span>64
@set <span class="nv">INCLUDE</span><span class="o">=</span>%VCINSTALLDIR%<span class="se">\I</span>NCLUDE<span class="p">;</span>%VSINSTALLDIR%<span class="se">\I</span>NCLUDE
@set <span class="nv">LIBPATH</span><span class="o">=</span>%VCINSTALLDIR%<span class="se">\L</span>IB<span class="se">\a</span>md64<span class="p">;</span>%VSINSTALLDIR%<span class="se">\L</span>IB<span class="se">\x</span>64
</code></pre></div></div>

<p>You may deploy Visual C++ redistributables (32-bit or 64-bit) depending on your needs.</p>

<p>After that, you’re done - enjoying the simplicity of having a ready-to-use build setup anywhere.</p>

<p>A last note: I’m not fan of .NET, but apparently some tools will cause you some headache if you are using this solution. You have been warned! :)</p>

<p><strong>TL;DR</strong>: Who needs to install a compiler on every machine ?</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[I Do Not Want Your Search Bar]]></title>
    <link href="https://blog.httrack.com/blog/2013/11/24/i-do-not-want-your-search-bar/"/>
    <updated>2013-11-24T10:44:00+01:00</updated>
    <id>https://blog.httrack.com/blog/2013/11/24/i-do-not-want-your-search-bar</id>
    <content type="html"><![CDATA[<blockquote>
  <p>I am writing to follow up on an email regarding a possible sponsorship on HTTrack.com.
…</p>
</blockquote>

<blockquote>
  <p>I would like to offer you a partnership agreement to monetize your application HTTrack
…</p>
</blockquote>

<blockquote>
  <p>We are a software monetization platform and I’m contacting you to discuss possible partnership
…</p>
</blockquote>

<p><em>Sheesh</em>.</p>

<p>The first time I got one of these email, I really didn’t know what it was all about. At first glance, I thought they wanted to offer some kind of ad-platform for <a href="http://www.httrack.com/">httrack.com</a>, or maybe some kind of sponsored links.</p>

<p>I <del>generally</del> always decline these offers, as I do not plan to put more ads on the main site (the only ad, placed on the download page, is to support bandwidth fees, as I’m hosting the binaries directly). Oh, and also because many of these ad providers are really terrible - do you really want to have ads for counterfeit clothes, fake <em>enhancement pills</em>, or pyramid-scheme business ? Me neither.</p>

<p>After several emails, I decided to ask for a bit more information on these “<em>partnerships</em>” and “<em>opportunities</em>” to understand what they really wanted.</p>

<p>It appears that what they were interesting in was to put a “<em>download bar</em>” (or a “<em>software bundle</em>”, or a “<em>download optimizer</em>”) in the httrack installer executable, and share benefits with me (well, at least this is what they said).</p>

<p>The only purpose of the <em>search bar</em> was to “<em>enhance browser experience</em>” and things like that (yes, really).</p>

<h3 id="oh-great-a-search-toolbar">Oh great, a search toolbar!</h3>

<p>.. said <strong>nobody</strong>. <strong>Ever</strong>.</p>

<p>Do you have a search bar installed on your PC ? I mean, you have a browser, and you decide to install an <em>additional</em> bar to… search for things. Like with the regular bar.</p>

<p>No, I suppose. Me, neither. <strong>Nobody does</strong>. Even your grandma.</p>

<p>So, what’s the point ?</p>

<ul>
  <li>each time the <em>search bar</em> is installed by your bundled application, you get some revenue</li>
  <li>each time the user is using the search bar, the search is redirected to some kind of second-zone search site loaded with ads (and crap)</li>
  <li>ads do generate revenue, fueling the system</li>
</ul>

<p>Great, isn’t it ?</p>

<h3 id="hell-no-this-is-crapware">Hell no, this is crapware!</h3>

<p>Adware, spyware, crapware. Yes.
And this was my first reaction, and then I <em>politely</em> declined.</p>

<blockquote>
  <p>But you can opt-out your users during install!</p>
</blockquote>

<p>Oh great. So each time a user install the program and <em>does not carefully look at the checkbox enabled by default</em>, he’s going to install the bar. This is the kind of thing which <a href="http://www.computerworld.com/s/article/9236273/Oracle_will_continue_to_bundle_crapware_with_Java">infuriates me</a>.</p>

<blockquote>
  <p>But you can opt-in your users during install, too!</p>
</blockquote>

<p>I was still skeptical, but why not, after all ? People installing the program by clicking “next-next” will not be harmed. And after all, this is just a search bar, isn’t it ?</p>

<h3 id="pact-with-the-devil">Pact With The Devil</h3>

<p>Well, no, this is wrong. So wrong.</p>

<p>First, if you take look at the reputation of all these companies, you’ll be horrified. Endless list of users trying to remove the infamous <em>search bar</em>, trying to figure out why they caught this <em>virus</em>, why they have pornographic pop-ups everywhere, etc.</p>

<p>Second, nobody will install this component <em>opt-in</em>.</p>

<p>More precisely:</p>

<ul>
  <li>very few people will install the <em>opt-in</em> component, generating nearly no revenue at all</li>
  <li>people who trust you will do - and will be betrayed, and will hate you, and it will harm your reputation</li>
</ul>

<p>What’s the point in allowing <em>opt-in</em> for these companies, by the way, is it does not generate any real revenue ?</p>

<p>The bleak truth is that in this case they are not providing you a marginal revenue. They are using your reputation to cover their bad one.</p>

<p>And this is the last reason why you should never, ever, make a deal with these people: as a developer, you have a reputation. You built something great (with few bugs, possibly) and you have a large user base trusting you. Do not betray them.</p>

<h3 id="oh-and-one-more-thing">Oh, And One More Thing</h3>

<p>Unfortunately, some über-jerks found an even better solution: bundle your application <strong>without telling you</strong>, and put it on their smelly download site.</p>

<p>The consequences are extremely <a href="https://www.google.com/#q=%22httrack%22+%22installcore%22">annoying</a></p>

<ul>
  <li>users complaining that httrack contains <a href="http://www.herdprotect.com/httrack.exe-0de695173053b3dae684cda9fa4b814b8a5dfee8.aspx">trojans</a></li>
  <li>users who are reluctant in trusting httrack because of the bad PR</li>
  <li>more generally, bad PR for “free” softwares (are they even safe ?)</li>
</ul>

<p>That is why I recently decided to start signing the Windows installer version, with an official trusted SSL key (registered to <em>Xavier Roche</em>). I will probably do the same for the program DLL’s soon. By the way, on Debian platforms, I have been signing the sources from the beginning, and this is a very good practice.</p>

<p><strong>TL;DR</strong>: Search bars are crap, and you should always download programs on <strong>trusted</strong> sites.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Everything You Always Wanted To Know About fsync()]]></title>
    <link href="https://blog.httrack.com/blog/2013/11/15/everything-you-always-wanted-to-know-about-fsync/"/>
    <updated>2013-11-15T19:37:00+01:00</updated>
    <id>https://blog.httrack.com/blog/2013/11/15/everything-you-always-wanted-to-know-about-fsync</id>
    <content type="html"><![CDATA[<p>And then the developer wondered:</p>
<blockquote>
  <p>is my file properly sync’ed on disk ?</p>
</blockquote>

<p>You probably know <em>more or less</em> how databases (or things that look like one) store their data on disk in a <em>permanent</em> and <em>safe</em> way. Or at least, you know the basic principles. Or not ?</p>

<h3 id="being-on-acid">Being on <strong>AC</strong>(I)<strong>D</strong></h3>

<p>There are a bunch of concepts that first must be understood: what is <strong>atomicity</strong>, <strong>consistency</strong>, and <strong>durability</strong> ? These concepts apply on databases (see <a href="https://en.wikipedia.org/wiki/ACID">ACID</a>), but also on the underlying filesystem.</p>

<ul>
  <li>
    <p><strong>Atomicity</strong>: a write operation is fully executed at once, and is not <em>interleaved</em> with another one (if, for example, someone else is writing to the same location)</p>

    <p>Atomicity is typically guaranteed in operations involving filename handling ; for example, for <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/rename.html">rename</a>, “<em>specification requires that the action of the function be atomic</em>” - that is, when renaming a file from the old name to the new one, at no circumstances should you ever see the two files at the same time.</p>
  </li>
  <li>
    <p><strong>Consistency</strong>: integrity of data must be maintained when executing an operation, even in a crash event - for example, a power outage in the middle of a <code class="language-plaintext highlighter-rouge">rename()</code> operation shall not leave the filesystem in a “weird” state, with the filename being unreachable because its metadata has been corrupted. (ie. either the operation is lost, or the operation is committed.)</p>

    <p>Consistency is guaranteed on the filesystem level ; but you also need to have the same guarantee if you build a database on disk, for example by serializing/locking certain operations on a working area, and committing the transaction by changing some kind of generation number.</p>
  </li>
  <li>
    <p><strong>Durability</strong>: the write operation is durable, that is, unplugging the power cord (or a kernel panic, a crash…) shall not lose any data (hitting the hard disk with a hammer is however not covered!)</p>

    <p>This is an important one - at a given point, you must ensure that the data is actually written on disk <em>physically</em>, preventing any loss of data in case of a sudden power outage, for example. This is absolutely critical when dealing with a client/server architecture: the client may have its connection or transaction aborted at any time without troubles (ie. the transaction will be retried later), but once the server acknowledges it, no event should ever cause it to be lost (think of responsibility in a commercial transaction, or a digital signature, for example). For this reason, having the data committed in the internal system or hard disk cache is NOT durable for obvious reasons (unless there is a guarantee that no such power outage could happen - if a battery is used on a RAID array, for example).</p>

    <p>On POSIX systems, durability is achieved through sync operations (<a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/fsync.html"><code class="language-plaintext highlighter-rouge">fsync()</code></a>, <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/fdatasync.html"><code class="language-plaintext highlighter-rouge">fdatasync()</code></a>, <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/aio_fsync.html"><code class="language-plaintext highlighter-rouge">aio_fsync()</code></a>): “<em>The fsync() function is intended to force a physical write of data from the buffer cache, and to assure that after a system crash or other failure that all data up to the time of the fsync() call is recorded on the disk.</em>”. [Note: The difference between fsync() and fdatasync() is that the later does not necessarily update the meta-data associated with a file - such as the “last modified” date - but only the file data.]</p>
  </li>
</ul>

<p>Now that these concepts are a bit clearer, let’s go back to our filesystem!</p>

<h3 id="hey-what-is-a-file-by-the-way-">Hey, What is a File, By The Way ?</h3>

<p>If we want to simplify the concept, let’s consider the filesystem on POSIX platforms as a very simple <em>flat</em> storage manager, allowing to read/write data blobs and basic properties (such as the modified time) indexed by an <em>integer</em> number (hint: they sometimes call that the <em>inode number</em>).</p>

<p>For example, you may want to read the file #4242’s data. And later, write some data on file #1234.</p>

<p>To have a more convenient way to handle files (because “I need to send you the presentation number 155324” would not be really convenient in the real world), we use the <strong>filename/directory</strong> concepts. A file has a name, and it is <em>contained</em> within a <em>directory structure</em>. You may put files and directories in a directory, building a hierarchical structure. But everything rely on our previous basic data blobs to store both filename and the associated index.</p>

<p>As an example, reading the file <code class="language-plaintext highlighter-rouge">foo/bar.txt</code> (ie. the file <code class="language-plaintext highlighter-rouge">bar.txt</code> within the <code class="language-plaintext highlighter-rouge">foo</code> directory) will require to access the data blob associated with the directory <code class="language-plaintext highlighter-rouge">foo</code>. After parsing this opaque data blob, the system will fetch the entry for <code class="language-plaintext highlighter-rouge">bar.txt</code>, and open the associated data blob. (And yes, there is obviously a root entry, storing references to first-level entries, allowing to access any file top-down)</p>

<p>If I now want to create a new file named <code class="language-plaintext highlighter-rouge">foo/baz.txt</code>, it will require the system to access the data blob associated with the directory <code class="language-plaintext highlighter-rouge">foo</code>, add an entry named <code class="language-plaintext highlighter-rouge">baz.txt</code> with a new allocated index for the upcoming file, and write the updated directory blob back, and from this point, write to the newly allocated blob. The operation therefore involves <strong>two</strong> data structures: the <strong>directory entry</strong>, and the <strong>file</strong> itself.</p>

<h3 id="keeping-my-filename-safe">Keeping My File(name) Safe</h3>

<p>Let’s go back to our database problem: what is the impact of having <em>two</em> data structures for our files ?</p>

<p><strong>Atomicity</strong> and <strong>consistency</strong> of filenames are handled for us by the filesystem, so this is not really a bother.</p>

<p>What about <strong>durability</strong> ?</p>

<p>We know that <code class="language-plaintext highlighter-rouge">fsync()</code> provides guarantees related to data and meta-data sync’ing. But if you look closer to the specification, the only data involved are the one related to the file itself - not its directory entry. The “metadata” concept involves modified time, access time etc. - <strong>not</strong> the <em>directory entry</em> itself.</p>

<p>It would be cumbersome for a filesystem to provide this guarantee, by the way: on POSIX systems, you can have an arbitrary number of directory links to a filename (or to another directory entry). The most common case is <em>one</em>, of course. But you may delete a file being used (the file entry will be removed by the system when the file is closed) - the very reason why erasing a log file which is flooding a filesystem is a futile and deadly action - in such case, the number of links will be <em>zero</em>. And you may also create as many <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/link.html">hard-links</a> as you want for a given file/directory entry.</p>

<p>Therefore, in theory, you may create a file, write some data, synchronize it,  close the file, and see your <a href="https://upload.wikimedia.org/wikipedia/en/e/e0/Gollum.PNG">precious</a> file lost forever because of a power outage. Oh, the filesystem must guarantee consistency, of course, but not durability unless <em>explicitly</em> asked by the client - which means that a filesystem check <em>may</em> find your directory entry partially written, and decide to achieve consistency by taking the previous directory blob entry, <strong>wiping</strong> the unreferenced file entry (note: if you are “lucky” enough, the file will be expelled in <code class="language-plaintext highlighter-rouge">lost+found</code>)</p>

<p>The filesystem can, of course, decide to be gentle, and commit all filename operations when fsync’ing. It may also, such as for ext3, commit <a href="https://ext4.wiki.kernel.org/index.php/Ext3_Data%3DOrdered_vs_Data%3DWriteback_mode"><strong>everything</strong></a> when fsync’ing a file - causing the <a href="http://lwn.net/Articles/328363/">infamous</a> and <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=421482">horrendous</a> lags in firefox or thunderbird.</p>

<p>But if you need to have <strong>guarantees</strong>, and not just <em>hope</em> the filesystem “<em>will be gentle</em>”, and do not want to “<em>trust the filesystem</em>” (yes, someone actually told me that: you <em>need</em> to “trust the filesystem” - I swear it), you have to actually make sure that your <strong>filename</strong> entry is properly sync’ed on disk following the POSIX specification.</p>

<p>Oh, and by the way: according to <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/fsync.html">POSIX</a>, <em>The fsync() function is intended to force a physical write of data from the buffer cache, and to assure that after a system crash or other failure that all data up to the time of the fsync() call is recorded on the disk</em>.</p>

<p><em>But</em> things are sometimes a bit obscure on the implementation side
:</p>

<ul>
  <li>
    <p><a href="http://unixhelp.ed.ac.uk/CGI/man-cgi?fsync"><strong>Linux/ext3</strong></a>: <em>If the underlying hard disk has write caching enabled, then the data may not really be on permanent storage when fsync() / fdatasync() return.</em> (do’h!)</p>
  </li>
  <li>
    <p><a href="http://linux.die.net/man/2/fsync"><strong>Linux/ext4</strong></a>: <em>The fsync() implementations in older kernels and lesser used filesystems does not know how to flush disk caches.</em> (do’h!) - issue adressed quite <a href="http://lwn.net/Articles/270891/">recently</a></p>
  </li>
  <li>
    <p><a href="https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/fsync.2.html"><strong>OSX</strong></a>: <em>For applications that require tighter guarantees about the integrity of their data, Mac OS X provides the F_FULLFSYNC fcntl. The <code class="language-plaintext highlighter-rouge">F_FULLFSYNC</code> fcntl asks the drive to flush all buffered data to permanent storage</em> (hey, fsync was supposed to do that, no ? guys ?) <em>(Edit: no, fsync is actually not required to do that - thanks for the clarification Florent!)</em></p>
  </li>
</ul>

<p>But we may assume that on Linux with <a href="http://monolight.cc/2011/06/barriers-caches-filesystems/">ext4</a> (and OSX with proper flags ?) the system is properly propagating <a href="http://docs.fedoraproject.org/en-US/Fedora/14/html/Storage_Administration_Guide/writebarr.html">write barriers</a>.</p>

<p>On Windows, using <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/aa364439.aspx"><code class="language-plaintext highlighter-rouge">FlushFileBuffers()</code></a> is probably the way to go.</p>

<h3 id="syncing-filenames">Syncing Filenames</h3>

<p>I told you that a filesystem was actually a bunch of <em>flat</em> data blobs with associated metadata, and that a file had actually two parts: its directory entry (let’s assume there is only one directory entry for the sake of simplicity), and its actual data. We already know how to sync the later one ; do we have a way to do the same for the directory container itself ?</p>

<p>On POSIX, you may actually open a directory as if you were opening a file (hint: a directory is <a href="http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html"><em>a file that contains directory entries</em></a>). It means that <code class="language-plaintext highlighter-rouge">open()</code> <em>may</em> successfully open a directory entry. But on the other hand, you generally can not open a directory entry for <em>writing</em> (see POSIX <a href="http://pubs.opengroup.org/onlinepubs/009695299/functions/open.html">remark</a> regarding <code class="language-plaintext highlighter-rouge">EISDIR</code>: <em>The named file is a directory and oflag includes O_WRONLY or O_RDWR</em>), and this is perfectly logical: by directly writing to the internal directory entry, you may be able to mess up with the directory structure, ruining the filesystem <strong>consistency</strong>.</p>

<p>But can we fsync() written data using a file descriptor opened <em>only</em> for reading ? The question is… yes, or at least “<em>yes it should</em>” - even POSIX group had <em>editorial</em> inconsistencies regarding <a href="http://austingroupbugs.net/view.php?id=501">fdatasync</a> and <a href="http://austingroupbugs.net/view.php?id=671">aio_fsync()</a>, leading to incorrect <a href="http://cygwin.com/frysk/bugzilla/show_bug.cgi?id=15361">behavior</a> on various implementations. And the reason it should execute the operation is because requesting the completion of a write operation does not have to require actual write access - which have already been checked and enforced.</p>

<p>On Windows… err, there is no clear answer. You can not call <code class="language-plaintext highlighter-rouge">FlushFileBuffers()</code> on a directory handle as far as I can see.</p>

<p>Oh, a last funny note: how do you sync the <strong>content</strong> of a <em>symbolic link</em> (and its related meta-data), that is, the filename pointed by this link ? The answer is… you can’t. Nope. This is not possible with the current standard (hint: you can not <code class="language-plaintext highlighter-rouge">open()</code> a symbolic link). Which means that if you handle some kind of database generation update based on symbolic links (ie. changing a “last-version” symlink to the latest built generation file), you have zero guarantee over durability.</p>

<h3 id="conclusion">Conclusion</h3>

<p>Does it means that we need to call <code class="language-plaintext highlighter-rouge">fsync()</code> <em>twice</em>, one on the file data, and one on its parent directory ? When you need to achieve <em>durability</em>, the answer is obviously <strong>yes</strong>. (Remember that file file/filename will be sync’ed on disk anyway by the operating system, so you do not actually need to do that for every single file - only for those you want to have a durability guarantee at a given time)</p>

<p>However, the question is causing some headache on the POSIX standard, and as a follow-up to the <a href="http://comments.gmane.org/gmane.comp.standards.posix.austin.general/6952">austin-group</a> (ie. POSIX mailing-list) discussion, an <a href="http://austingroupbugs.net/view.php?id=672">editorial clarification request</a> is still pending and is waiting for feedback from various implementors. (you may also have a look at the <a href="http://unix.derkeiler.com/Newsgroups/comp.unix.programmer/2013-03/msg00016.html">comp.unix.programmer</a> discussion)</p>

<p><strong>TL;DR</strong>: syncing a file is not as simple as it seems!</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Coding is like Cooking]]></title>
    <link href="https://blog.httrack.com/blog/2013/11/11/coding-is-like-cooking/"/>
    <updated>2013-11-11T16:52:00+01:00</updated>
    <id>https://blog.httrack.com/blog/2013/11/11/coding-is-like-cooking</id>
    <content type="html"><![CDATA[<p>Many people try to compare coding (ie. “programming”) with more <em>understandable</em> concepts. Developers are sometimes compared to <em>rock stars</em> (some people would say <em>divas</em>, too), <a href="http://venturebeat.com/2013/10/19/841449/"><em>artists</em></a>, <em>rebels</em>, <a href="http://www.chrisaitchison.com/2011/05/03/you-are-not-a-software-engineer/"><em>gardeners</em></a> and sometimes, err, <em>obscure geeks typing on their keyboard</em> (yes, <em>peasants</em>, unlike the all-mighty business guys, cough cough cough.).</p>

<p>I consider myself as a <em>chef</em> - yes, as in <strong>cooking</strong>. My interest in <em>cuisine</em> (I am French, I admit) has probably something to do with this comparison, but I have always found it adequate for multiple reasons.</p>

<p>The ingredients.</p>

<ul>
  <li>python: want to cook something quickly and without too much hassle ? try ready-to-cook products, such as mexican-style food. you won’t win a cooking prize, but you did not have the time for something better anyway (and hey, you washed the tomatoes and the salad, so it’s technically <em>cooking</em>) (Note: I’m going to be flamed for this one - guys, I swear I love python too!)</li>
  <li>java: want to make a pie, but you don’t want to prepare the dough: use a ready-to-use pastry crust. yes, this is not as good as the real one, but hey, the blueberry tart is good anyway.</li>
  <li>c: you want to prepare everything, including the sauce, vegetable cutting/cooking, etc. after six hours of hard work, and having burnt twice your dough, you can at least eat a decent meal with pride.</li>
  <li>assembly: you want to make <em>your own flour</em> because, err, you don’t want to use someone else’s, and your flour will be better. and you want also yo make your own butter for the same reason. after one hard day of work, you have some butter, but did not have the time to grind the flour, but the cake will be the best one ever after you finish it next month.</li>
</ul>

<p>Of course, there’s always  a simpler solution:</p>

<ul>
  <li>configure &amp;&amp; make &amp;&amp; make install: hey, I don’t want to cook, let’s call the pizza delivery service! too bad I can’t order one with blue cheese.</li>
</ul>

<p>Some people will also find themselves unable to cook anything without specific instructions. Other will never follow any recipe, but will only write down rough steps of what they want to do.</p>

<p>But there is an even better reason for comparing coding and cooking: comparing the food industry with the IT industry.</p>

<p>There are basically two trends when you want to work in the food industry:</p>

<ul>
  <li>
    <p>The fast food chains: neither food nor salaries are great, but it’s a good way to start if you know nothing about cooking, and/or do not want to be cooking for the rest of your life. If you are good enough, you’ll start to do more interesting things as flipping burgers, such as management, running your own business, etc. And no, you won’t be a chef, ever.</p>
  </li>
  <li>
    <p>The restaurants: food is great (at least if this is a <em>decent</em> place), salaries CAN be great too (especially if you’re famous), but prepare yourself for hard work and pain. You’ll start peeling potatoes, and if you’re good enough, in few years you’ll become a useful aid inside a <a href="https://en.wikipedia.org/wiki/Brigade_de_cuisine"><em>brigade</em></a>. And maybe, one day, a great chef inside a three-star restaurant!</p>
  </li>
</ul>

<p>The two career paths are quite different, but most people understand that you may choose one or another depending on your motivation toward food. I mean, if you REALLY love food, you’ll try to become a chef. And nobody is going to consider this as a weird choice, or even less desirable. Most people will actually think the contrary.</p>

<p>In the IT industry, you may choose to work for a company focusing on management, business, and not at all on software engineering - think of contractors and integrators. You will attempt to stop coding as fast as possible, because, err, only looser still code after few years in these companies.</p>

<p>But for those who <strong>like</strong> to code, want to excel at finding new algorithms, discovering new technical challenges, the path is not quite the same, and the company profile radically different.</p>

<p>That’s why focusing on the <em>good types of companies</em> is absolutely vital for a newcomer in the IT industry. Working in small startups or in obscure software editors can be a great choice. You will learn, and you will learn by coding. And after few years, you’ll still code, design great algorithms, and lead incredible technical projects. And if you don’t feel like coding in few years, maybe you should choose the other path.</p>

<p><strong>TL;DR</strong>: comparisons are always a bit lame, but, err, this one is not that bad. Achieve excellence in your code as a chef can in cooking!</p>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Creating deletable and movable files on Windows]]></title>
    <link href="https://blog.httrack.com/blog/2013/10/05/creating-deletable-and-movable-files-on-windows/"/>
    <updated>2013-10-05T12:05:00+02:00</updated>
    <id>https://blog.httrack.com/blog/2013/10/05/creating-deletable-and-movable-files-on-windows</id>
    <content type="html"><![CDATA[<p>You probably stumbled upon this issue on Windows platforms: files created by <a href="http://msdn.microsoft.com/fr-fr/library/yeby3zcb%28v=vs.90%29.aspx"><code class="language-plaintext highlighter-rouge">fopen()</code></a> are by default locked for deletion or renaming.</p>

<p>Ie. if you do this:</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  <span class="kt">FILE</span> <span class="o">*</span><span class="n">my_file</span> <span class="o">=</span> <span class="n">fopen</span><span class="p">(</span><span class="s">"foo.bar"</span><span class="p">,</span> <span class="s">"wb"</span><span class="p">);</span>
  <span class="p">...</span>
</code></pre></div></div>

<p>then the <code class="language-plaintext highlighter-rouge">foo.bar</code> file will be locked until the process closes the file (or dies). No renaming. No deleting.</p>

<p>This default feature is rather different from what we have on <strong>POSIX</strong>-like systems, and on <strong>Linux</strong> especially, where any file can basically be renamed or deleted while being opened. Once a file is being opened, its directory reference has basically no influence, except the implicit reference count (ie. deleting the file will only decrement the refcount, and closing the last handle will remove the underlying inodes)</p>

<p>This is actually a <em>cool feature</em>: you can have a file opened, and immediately unlinked, to have an anonymous file where you may execute read/write operations without bothering handling the cleanup if the process is killed, for example. You can also create a log file, redirect stdout/stderr, and later want to rotate the file, without being hit by <code class="language-plaintext highlighter-rouge">permission denied</code> errors. This is especially true when all references to the file haven’t been closed yet - say, if you have stdio redirected to a log file, and if you have a child process spawned and logging in the same log file.</p>

<p>On, Windows, the infamous <code class="language-plaintext highlighter-rouge">ERROR_ACCESS_DENIED</code> will kick you in the face as soon as you try to do the same.</p>

<p>There is actually a quite simple solution if you want to get a <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/fopen.html"><code class="language-plaintext highlighter-rouge">fopen()</code></a> semantic which is closer to the POSIX one: use the low-level <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/aa363858%28v=vs.85%29.aspx"><code class="language-plaintext highlighter-rouge">CreateFile</code></a> function with the <code class="language-plaintext highlighter-rouge">FILE_SHARE_DELETE</code> sharing mode.</p>

<p>The documentation says:</p>

<ul>
  <li>FILE_SHARE_DELETE 
Enables subsequent open operations on a file or device to request delete access.
Otherwise, other processes cannot open the file or device if they request delete access.
If this flag is not specified, but the file or device has been opened for delete access, the function fails.
Note  Delete access allows both delete and rename operations.</li>
</ul>

<p>Hey, that’s exactly what we are looking for! But we want a nice <code class="language-plaintext highlighter-rouge">FILE</code> pointer, not an horrible <code class="language-plaintext highlighter-rouge">HANDLE</code> object. Is it possible ?</p>

<p><strong>Yes!</strong>:</p>

<ul>
  <li>
    <p>the <code class="language-plaintext highlighter-rouge">HANDLE</code> object can be attached to a low-level C library handle through <a href="http://msdn.microsoft.com/en-us/library/bdts1c9x.aspx"><code class="language-plaintext highlighter-rouge">_open_osfhandle()</code></a></p>
  </li>
  <li>
    <p>the low-level C library handle can be attached to a <code class="language-plaintext highlighter-rouge">FILE*</code> object through <a href="http://msdn.microsoft.com/en-us/library/dye30d82.aspx"><code class="language-plaintext highlighter-rouge">_fdopen()</code></a></p>
  </li>
</ul>

<p>You just have to take some care about the passed <code class="language-plaintext highlighter-rouge">fopen()</code> flags to open the file with proper flags, and your “compatible” function should be something like:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#include</span> <span class="cpf">&lt;stdio.h&gt;</span><span class="cp">
#include</span> <span class="cpf">&lt;fcntl.h&gt;</span><span class="cp">
#include</span> <span class="cpf">&lt;io.h&gt;</span><span class="cp">
#include</span> <span class="cpf">&lt;windows.h&gt;</span><span class="cp">
</span>
<span class="cm">/**
 * Opens the file whose name is the string pointed to by file and associates
 * a stream with it.
 **/</span>
<span class="k">static</span> <span class="kt">FILE</span><span class="o">*</span> <span class="nf">fopen_unixlike</span><span class="p">(</span><span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">file</span><span class="p">,</span> <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">mode</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">DWORD</span> <span class="n">dwDesiredAccess</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
  <span class="n">DWORD</span> <span class="n">dwCreationDisposition</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
  <span class="k">const</span> <span class="n">DWORD</span> <span class="n">dwFlagsAndAttributes</span> <span class="o">=</span> <span class="n">FILE_ATTRIBUTE_NORMAL</span><span class="p">;</span>
  <span class="k">const</span> <span class="n">DWORD</span> <span class="n">dwShareMode</span> <span class="o">=</span> <span class="n">FILE_SHARE_READ</span> <span class="o">|</span> <span class="n">FILE_SHARE_WRITE</span> <span class="o">|</span> <span class="n">FILE_SHARE_DELETE</span><span class="p">;</span>
  <span class="kt">size_t</span> <span class="n">i</span><span class="p">;</span>

  <span class="n">HANDLE</span> <span class="n">handle</span><span class="p">;</span>
  <span class="kt">int</span> <span class="n">fd</span><span class="p">;</span>

  <span class="cm">/* Infer flags. */</span>
  <span class="k">for</span><span class="p">(</span><span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">mode</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">!=</span> <span class="sc">'\0'</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">switch</span><span class="p">(</span><span class="n">mode</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="p">{</span>
      <span class="k">case</span> <span class="sc">'r'</span><span class="p">:</span>
        <span class="n">dwDesiredAccess</span> <span class="o">|=</span> <span class="n">GENERIC_READ</span><span class="p">;</span>
        <span class="n">dwCreationDisposition</span> <span class="o">=</span> <span class="n">OPEN_EXISTING</span><span class="p">;</span>
        <span class="k">break</span><span class="p">;</span>
      <span class="k">case</span> <span class="sc">'a'</span><span class="p">:</span>
        <span class="n">dwDesiredAccess</span> <span class="o">|=</span> <span class="n">GENERIC_WRITE</span><span class="p">;</span>
        <span class="n">dwCreationDisposition</span> <span class="o">=</span> <span class="n">CREATE_NEW</span><span class="p">;</span>
        <span class="k">break</span><span class="p">;</span>
      <span class="k">case</span> <span class="sc">'w'</span><span class="p">:</span>
        <span class="n">dwDesiredAccess</span> <span class="o">|=</span> <span class="n">GENERIC_WRITE</span><span class="p">;</span>
        <span class="n">dwCreationDisposition</span> <span class="o">=</span> <span class="n">CREATE_ALWAYS</span><span class="p">;</span>
        <span class="k">break</span><span class="p">;</span>
      <span class="k">case</span> <span class="sc">'+'</span><span class="p">:</span>
        <span class="n">dwDesiredAccess</span> <span class="o">|=</span> <span class="n">GENERIC_READ</span><span class="p">;</span>
        <span class="n">dwDesiredAccess</span> <span class="o">|=</span> <span class="n">GENERIC_WRITE</span><span class="p">;</span>
        <span class="k">break</span><span class="p">;</span>
    <span class="p">}</span>
  <span class="p">}</span>

  <span class="cm">/* Create the file. */</span>
  <span class="n">handle</span> <span class="o">=</span> <span class="n">CreateFileA</span><span class="p">(</span><span class="n">file</span><span class="p">,</span>
                       <span class="n">dwDesiredAccess</span><span class="p">,</span>
                       <span class="n">dwShareMode</span><span class="p">,</span>
                       <span class="nb">NULL</span><span class="p">,</span>
                       <span class="n">dwCreationDisposition</span><span class="p">,</span>
                       <span class="n">dwFlagsAndAttributes</span><span class="p">,</span>
                       <span class="nb">NULL</span><span class="p">);</span>
  <span class="k">if</span> <span class="p">(</span><span class="n">handle</span> <span class="o">==</span> <span class="n">INVALID_HANDLE_VALUE</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
  <span class="p">}</span>

  <span class="cm">/* Associates a C run-time file descriptor with a file HANDLE. */</span>
  <span class="n">fd</span> <span class="o">=</span> <span class="n">_open_osfhandle</span><span class="p">((</span><span class="kt">intptr_t</span><span class="p">)</span> <span class="n">handle</span><span class="p">,</span> <span class="n">_O_BINARY</span><span class="p">);</span>
  <span class="k">if</span> <span class="p">(</span><span class="n">fd</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">CloseHandle</span><span class="p">(</span><span class="n">handle</span><span class="p">);</span>
    <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
  <span class="p">}</span>

  <span class="cm">/* Associates a stream with a C run-time file descriptor. */</span>
  <span class="k">return</span> <span class="n">_fdopen</span><span class="p">(</span><span class="n">fd</span><span class="p">,</span> <span class="n">mode</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>

<p>You can test this function using the example below:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/* Comment to test. */</span>
<span class="cp">#define fopen fopen_unixlike
</span>
<span class="kt">int</span> <span class="nf">main</span><span class="p">(</span><span class="kt">int</span> <span class="n">argc</span><span class="p">,</span> <span class="kt">char</span><span class="o">*</span> <span class="n">argv</span><span class="p">[])</span> <span class="p">{</span>
  <span class="kt">FILE</span> <span class="o">*</span><span class="n">fp</span> <span class="o">=</span> <span class="n">fopen</span><span class="p">(</span><span class="s">"log.txt"</span><span class="p">,</span> <span class="s">"wb"</span><span class="p">);</span>
  <span class="kt">int</span> <span class="n">i</span><span class="p">;</span>
  <span class="k">for</span><span class="p">(</span><span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="mi">10</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">if</span> <span class="p">(</span><span class="n">fprintf</span><span class="p">(</span><span class="n">fp</span><span class="p">,</span> <span class="s">"this is the log line #%d</span><span class="se">\r\n</span><span class="s">"</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">0</span>
      <span class="o">&amp;&amp;</span> <span class="n">fflush</span><span class="p">(</span><span class="n">fp</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
      <span class="n">printf</span><span class="p">(</span><span class="s">"successfully written line #%d, press a key</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">i</span><span class="p">);</span>
      <span class="n">_fgetchar</span><span class="p">();</span>
    <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
      <span class="n">fprintf</span><span class="p">(</span><span class="n">stderr</span><span class="p">,</span> <span class="s">"error writting line #%d, press a key</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">i</span><span class="p">);</span>
      <span class="n">_fgetchar</span><span class="p">();</span>
      <span class="k">break</span><span class="p">;</span>
    <span class="p">}</span>
  <span class="p">}</span>
  <span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>

<p>The “log.txt” file will be deletable and movable without troubles, and you may even successfully write to the deleted file.</p>

<p><strong>TL;DR</strong>: I wish I had known sooner. Darn.</p>

]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[An unusual case of case (/switch)]]></title>
    <link href="https://blog.httrack.com/blog/2013/09/16/an-unusual-case-of-case-slash-switch/"/>
    <updated>2013-09-16T19:51:00+02:00</updated>
    <id>https://blog.httrack.com/blog/2013/09/16/an-unusual-case-of-case-slash-switch</id>
    <content type="html"><![CDATA[<p>I have been programming in C for something like 20 years, and I don’t consider myself as illiterate in coding. But time to time, I still discover little gems and curiosities, even in the very deep roots of the C language.</p>

<p>The latest gem <a href="https://github.com/davidlazar/base64/blob/master/base64encode.c">was</a> this code snippet:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  <span class="k">switch</span> <span class="p">(</span><span class="n">step</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">while</span> <span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
      <span class="k">case</span> <span class="n">step_A</span><span class="p">:</span>
      <span class="k">case</span> <span class="n">step_B</span><span class="p">:</span>
        <span class="p">...</span>
     <span class="p">}</span>
  <span class="p">}</span>
</code></pre></div></div>

<p>No, there’s no mistake - there’s a <code class="language-plaintext highlighter-rouge">while</code> interleaved between <code class="language-plaintext highlighter-rouge">switch</code> and <code class="language-plaintext highlighter-rouge">case</code>. Darn.</p>

<p>The fact is that <code class="language-plaintext highlighter-rouge">switch</code> is actually a disguised dispatch of <code class="language-plaintext highlighter-rouge">goto</code> (horror!) equivalent instructions, and <code class="language-plaintext highlighter-rouge">case</code> a disguised label case. As you can jump from/to nested loops or conditions, the code is perfectly valid (and logical).</p>

<p>The above code can be rewritten as something like:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  <span class="k">do</span> <span class="p">{</span>
    <span class="k">if</span> <span class="p">(</span><span class="n">step</span> <span class="o">==</span> <span class="n">step_A</span><span class="p">)</span> <span class="p">{</span>
      <span class="k">goto</span> <span class="n">A</span><span class="p">;</span>
    <span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="n">step</span> <span class="o">==</span> <span class="n">step_B</span><span class="p">)</span> <span class="p">{</span>
      <span class="k">goto</span> <span class="n">B</span><span class="p">;</span>
    <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
      <span class="k">goto</span> <span class="n">E</span><span class="p">;</span>
    <span class="p">}</span>
    <span class="k">while</span> <span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
      <span class="nl">A:</span>
      <span class="nl">B:</span>
        <span class="p">...</span>
    <span class="p">}</span>
    <span class="nl">E:</span>
  <span class="p">}</span> <span class="k">while</span><span class="p">(</span><span class="mi">0</span><span class="p">);</span>
</code></pre></div></div>

<p>This was <a href="https://groups.google.com/forum/#!topic/comp.lang.c/9Y35VN3AJp8">confirmed</a> by the specialists of <code class="language-plaintext highlighter-rouge">comp.lang.c</code>, with interesting remarks:</p>

<ul>
  <li>This coding technic is actually pretty classic in highly-optimized code (shame on me), and called <a href="https://en.wikipedia.org/wiki/Duff's_device">Duff’s device</a>, and allows, for example, to copy blocs of memory in a single row, inside a reduced loop, rather than copying the remainder outside the loop, and copying full chunks inside it:</li>
</ul>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">void</span> <span class="nf">somewhat_fast_memcpy</span><span class="p">(</span><span class="kt">char</span> <span class="o">*</span><span class="n">to</span><span class="p">,</span> <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">from</span><span class="p">,</span> <span class="k">const</span> <span class="kt">size_t</span> <span class="n">count</span><span class="p">)</span> <span class="p">{</span>
  <span class="kt">size_t</span> <span class="n">n</span> <span class="o">=</span> <span class="p">(</span><span class="n">count</span> <span class="o">+</span> <span class="mi">7</span><span class="p">)</span> <span class="o">/</span> <span class="mi">8</span><span class="p">;</span>
  <span class="k">switch</span><span class="p">(</span><span class="n">count</span> <span class="o">%</span> <span class="mi">8</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">case</span> <span class="mi">0</span><span class="p">:</span>
    <span class="k">do</span> <span class="p">{</span>
      <span class="o">*</span><span class="n">to</span> <span class="o">=</span> <span class="o">*</span><span class="n">from</span><span class="o">++</span><span class="p">;</span>
      <span class="k">case</span> <span class="mi">7</span><span class="p">:</span> <span class="o">*</span><span class="n">to</span> <span class="o">=</span> <span class="o">*</span><span class="n">from</span><span class="o">++</span><span class="p">;</span>
      <span class="k">case</span> <span class="mi">6</span><span class="p">:</span> <span class="o">*</span><span class="n">to</span> <span class="o">=</span> <span class="o">*</span><span class="n">from</span><span class="o">++</span><span class="p">;</span>
      <span class="k">case</span> <span class="mi">5</span><span class="p">:</span> <span class="o">*</span><span class="n">to</span> <span class="o">=</span> <span class="o">*</span><span class="n">from</span><span class="o">++</span><span class="p">;</span>
      <span class="k">case</span> <span class="mi">4</span><span class="p">:</span> <span class="o">*</span><span class="n">to</span> <span class="o">=</span> <span class="o">*</span><span class="n">from</span><span class="o">++</span><span class="p">;</span>
      <span class="k">case</span> <span class="mi">3</span><span class="p">:</span> <span class="o">*</span><span class="n">to</span> <span class="o">=</span> <span class="o">*</span><span class="n">from</span><span class="o">++</span><span class="p">;</span>
      <span class="k">case</span> <span class="mi">2</span><span class="p">:</span> <span class="o">*</span><span class="n">to</span> <span class="o">=</span> <span class="o">*</span><span class="n">from</span><span class="o">++</span><span class="p">;</span>
      <span class="k">case</span> <span class="mi">1</span><span class="p">:</span> <span class="o">*</span><span class="n">to</span> <span class="o">=</span> <span class="o">*</span><span class="n">from</span><span class="o">++</span><span class="p">;</span>
    <span class="p">}</span> <span class="k">while</span><span class="p">(</span><span class="o">--</span><span class="n">n</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">);</span>
  <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<ul>
  <li>You may end up with the same issues as for the <code class="language-plaintext highlighter-rouge">goto</code> equivalent ; such as uninitialized variables: (note that clang detect this case, but not my version of gcc)</li>
</ul>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  <span class="kt">int</span> <span class="n">i</span><span class="p">;</span>
  <span class="k">switch</span> <span class="p">(</span><span class="n">step</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">for</span> <span class="p">(</span><span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">loops</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
      <span class="k">case</span> <span class="n">step_A</span><span class="p">:</span> <span class="c1">// WARNING: uninitialized i</span>
      <span class="k">case</span> <span class="n">step_B</span><span class="p">:</span>
        <span class="p">...</span>
     <span class="p">}</span>
  <span class="p">}</span>
</code></pre></div></div>

<ul>
  <li>
    <p>There’s one constraint you have to follow (please take your breath before reading): <em>“If a switch statement has an associated case or default label within the scope of an identifier with a variably modified type, the entire switch statement shall be within the scope of that identifier.”</em>. Basically, it means that if you are foolish enough to use this coding style, <strong>AND</strong> foolish enough to use variable-length-variables (such as dynamic arrays), then the compiler has the right to put a contract on you.</p>
  </li>
  <li>
    <p>Abuse of interleaving of switch/case and other constructs may turn your code into a steaming pile of crap. Did I tell you that <code class="language-plaintext highlighter-rouge">switch</code>/<code class="language-plaintext highlighter-rouge">case</code> was somewhat equivalent to <code class="language-plaintext highlighter-rouge">goto</code>?</p>
  </li>
</ul>

<p><strong>TL;DR</strong>: take an aspirin and good nite.</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[Replacing JNI Crashes by Exceptions on Android]]></title>
    <link href="https://blog.httrack.com/blog/2013/08/23/catching-posix-signals-on-android/"/>
    <updated>2013-08-23T10:12:00+02:00</updated>
    <id>https://blog.httrack.com/blog/2013/08/23/catching-posix-signals-on-android</id>
    <content type="html"><![CDATA[<h2 id="to-report-or-not-to-report">To Report Or Not To Report</h2>

<p>You have a <a href="https://play.google.com/store/apps/details?id=com.httrack.android">nice</a> application available on the Google Android Store and, as a <strong>developer</strong>, you have access to nice features giving you basic statistics (the number of downloads, Android version breakdown, etc.), reviews, and a <em>Crashes &amp; ANRs</em> section allowing to audit user crash reports (and Application hangs - that’s the <em>ANR</em> thing).</p>

<p><img src="/images/httrack/google_play_console_httrack_crash_sample_50pc.png" alt="the crash report console" /></p>

<p>This is a rather basic feature (you do not have any details on the user’s phone, Android version, etc. - only a <em>Java stack trace</em> extract), but at least it allows you to quickly spot mistakes (such as <code class="language-plaintext highlighter-rouge">NullPointerException</code>, or in this case, a <code class="language-plaintext highlighter-rouge">NumberFormatException</code>), and the report is pretty straightforward for users (they only have click on the “Report” button in case of crash)</p>

<p>But what if your application is using <strong>native code</strong> (through <em>JNI</em>) ? In such case, the application will just crash <em>silently</em>, giving the user no opportunity to report the bug to the upstream developer (<em>you</em>), which is <strong>not cool</strong> (because the bug will remain unspotted, unless users are nice enough to email you, and have the know-how to provide you useful technical details, such as the address of the crash, which is kind of rare)</p>

<h2 id="catching-the-problem">Catching the Problem</h2>

<p>A first step is obviously to be able to <em>detect</em> common native crashes (<code class="language-plaintext highlighter-rouge">SIGSEGV</code>, <code class="language-plaintext highlighter-rouge">SIGBUS</code>, etc.) using signal handlers. On POSIX systems, this can be achieved by using <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/sigaction.html"><code class="language-plaintext highlighter-rouge">sigaction()</code></a>:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">struct</span> <span class="n">sigaction</span> <span class="n">sa</span><span class="p">;</span>
<span class="k">struct</span> <span class="n">sigaction</span> <span class="n">sa_old</span><span class="p">;</span>
<span class="n">memset</span><span class="p">(</span><span class="o">&amp;</span><span class="n">sa</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">sa</span><span class="p">));</span>
<span class="n">sigemptyset</span><span class="p">(</span><span class="o">&amp;</span><span class="n">sa</span><span class="p">.</span><span class="n">sa_mask</span><span class="p">);</span>
<span class="n">sa</span><span class="p">.</span><span class="n">sa_sigaction</span> <span class="o">=</span> <span class="n">my_handler</span><span class="p">;</span>
<span class="n">sa</span><span class="p">.</span><span class="n">sa_flags</span> <span class="o">=</span> <span class="n">SA_SIGINFO</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">sigaction</span><span class="p">(</span><span class="n">sig</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">sa</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">sa_old</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
  <span class="p">...</span>
<span class="p">}</span>
</code></pre></div></div>

<ul>
  <li>First problem: we are typically running on small systems (<em>Android</em> …), and one source of error is the <em>stack overflow</em>. When the stack is full (too much recursion, too large objects on stack), you hit the last guard page, and the system will raise the <code class="language-plaintext highlighter-rouge">SIGSEGV</code> signal handler, running by default on the… same full stack, raising one more time the signal. Fortunately, you may register in any thread an alternative stack through the use of <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/sigaltstack.html"><code class="language-plaintext highlighter-rouge">sigaltstack</code></a>, which basically reserve some space in case of emergency (ie. the system will switch the stack pointer to this one in case of trouble, letting you handler run on a “fresh” stack).</li>
</ul>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">stack_t</span> <span class="n">stack</span><span class="p">;</span>
<span class="n">memset</span><span class="p">(</span><span class="o">&amp;</span><span class="n">stack</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">stack</span><span class="p">));</span>
<span class="cm">/* Reserver the system default stack size. We don't need that much by the way. */</span>
<span class="n">stack</span><span class="p">.</span><span class="n">ss_size</span> <span class="o">=</span> <span class="n">SIGSTKSZ</span><span class="p">;</span>
<span class="n">stack</span><span class="p">.</span><span class="n">ss_sp</span> <span class="o">=</span> <span class="n">malloc</span><span class="p">(</span><span class="n">stack</span><span class="p">.</span><span class="n">ss_size</span><span class="p">);</span>
<span class="n">stack</span><span class="p">.</span><span class="n">ss_flags</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="cm">/* Install alternate stack size. Be sure the memory region is valid until you revert it. */</span>
<span class="k">if</span> <span class="p">(</span><span class="n">stack</span><span class="p">.</span><span class="n">ss_sp</span> <span class="o">!=</span> <span class="nb">NULL</span> <span class="o">&amp;&amp;</span> <span class="n">sigaltstack</span><span class="p">(</span><span class="o">&amp;</span><span class="n">stack</span><span class="p">,</span> <span class="nb">NULL</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
  <span class="p">...</span>
<span class="p">}</span>
</code></pre></div></div>

<ul>
  <li>Second problem: we’re hosted on a Java Virtual Machine, and some of these signals might already be caught. Typically, <code class="language-plaintext highlighter-rouge">SIGSEGV</code> might be regularly raised to address <code class="language-plaintext highlighter-rouge">NullPointerException</code> or as normal JIT processing (ie. executable pages might be flagged with a “no access” protection, and filled by the JIT compiler through a signal handler) - you have to make sure the original signal handler is called first, before messing up with it. If the signal was not processed, the original signal handler will generally return, or will call <code class="language-plaintext highlighter-rouge">abort()</code> (which is nice, because we have a last chance to catch it through a <code class="language-plaintext highlighter-rouge">SIGABRT</code> handler)</li>
</ul>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">static</span> <span class="kt">void</span> <span class="nf">my_handler</span><span class="p">(</span><span class="k">const</span> <span class="kt">int</span> <span class="n">code</span><span class="p">,</span> <span class="n">siginfo_t</span> <span class="o">*</span><span class="k">const</span> <span class="n">si</span><span class="p">,</span> <span class="kt">void</span> <span class="o">*</span><span class="k">const</span> <span class="n">sc</span><span class="p">)</span> <span class="p">{</span>
  <span class="cm">/* Call previous handler. */</span>
  <span class="n">old_handler</span><span class="p">.</span><span class="n">sa_sigaction</span><span class="p">(</span><span class="n">code</span><span class="p">,</span> <span class="n">si</span><span class="p">,</span> <span class="n">sc</span><span class="p">);</span>
<span class="p">...</span>
<span class="p">}</span>
</code></pre></div></div>

<ul>
  <li>Third problem: we’re running on a multi-threaded process, and ideally we do not want to catch crashes from threads we do not own. We can address this issue by using <a href="http://pubs.opengroup.org/onlinepubs/009696699/functions/pthread_setspecific.html"><code class="language-plaintext highlighter-rouge">pthread_getspecific()</code></a> to have a thread-specific context. Well, this is actually a <em>dirty</em> solution: <code class="language-plaintext highlighter-rouge">pthread_getspecific()</code> is <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/xsh_chap02_04.html">not</a> an <em>async-signal-safe</em> function, which means that if you are using it on a signal handler, you <em>may</em> have to <em>prepare for unforeseen consequences</em>. (I fail to see what could go wrong with this specific function, however - this is just a peek in a thread-specific address array. But yes, yes, we’re playing with fire, don’t kick me!)</li>
</ul>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">static</span> <span class="kt">void</span> <span class="nf">my_handler</span><span class="p">(</span><span class="k">const</span> <span class="kt">int</span> <span class="n">code</span><span class="p">,</span> <span class="n">siginfo_t</span> <span class="o">*</span><span class="k">const</span> <span class="n">si</span><span class="p">,</span> <span class="kt">void</span> <span class="o">*</span><span class="k">const</span> <span class="n">sc</span><span class="p">)</span> <span class="p">{</span>
  <span class="cm">/* Call previous handler. */</span>
  <span class="n">old_handler</span><span class="p">.</span><span class="n">sa_sigaction</span><span class="p">(</span><span class="n">code</span><span class="p">,</span> <span class="n">si</span><span class="p">,</span> <span class="n">sc</span><span class="p">);</span>

  <span class="cm">/* Get thread-specific context. */</span>
  <span class="n">my_struct</span> <span class="o">*</span><span class="n">s</span> <span class="o">=</span> <span class="p">(</span><span class="n">my_struct</span><span class="o">*</span><span class="p">)</span> <span class="n">pthread_getspecific</span><span class="p">(</span><span class="n">my_thread_var</span><span class="p">);</span>
  <span class="k">if</span> <span class="p">(</span><span class="n">s</span> <span class="o">!=</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
    <span class="p">...</span>
  <span class="p">}</span>
<span class="p">...</span>
<span class="p">}</span>
</code></pre></div></div>

<ul>
  <li>Fourth problem: we have to collect some basic information on the crash, especially the faulting address. Fortunately, the third argument of the <code class="language-plaintext highlighter-rouge">sigaction</code> callback is a pointer to a <code class="language-plaintext highlighter-rouge">ucontext_t</code> context collecting register values (and various other processor-specific details). On x86-64 architectures, the program counter will typically be saved in <code class="language-plaintext highlighter-rouge">uc_mcontext.gregs[REG_RIP]</code> ; on ARM, <code class="language-plaintext highlighter-rouge">uc_mcontext.arm_pc</code>. Unfortunately, on Android, the <code class="language-plaintext highlighter-rouge">ucontext_t</code> structure is not defined in any system headers, and you’ll have to import one by yourself (I shamelessly copied the one from Richard Quirk). You also have to find out what was the binary where the program counter was actually running, to find out this code base address in memory, because a randomized address is not very useful for audit and debugging. The Linux-specific <a href="http://linux.die.net/man/3/dladdr"><code class="language-plaintext highlighter-rouge">dladdr()</code></a> function is fortunately giving you this information, with useful other ones (namely the nearest symbol matching the address, and the module base address, to compute a relative offset address). (Note: you can also get this information on Linux by snooping in <code class="language-plaintext highlighter-rouge">/proc/self/maps</code>, and checking the address ranges - it will at least provide you the base address)</li>
</ul>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">Dl_info</span> <span class="n">info</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">dladdr</span><span class="p">(</span><span class="n">addr</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">info</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">0</span> <span class="o">&amp;&amp;</span> <span class="n">info</span><span class="p">.</span><span class="n">dli_fname</span> <span class="o">!=</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
  <span class="kt">void</span> <span class="o">*</span> <span class="k">const</span> <span class="n">nearest</span> <span class="o">=</span> <span class="n">info</span><span class="p">.</span><span class="n">dli_saddr</span><span class="p">;</span>
  <span class="k">const</span> <span class="kt">uintptr_t</span> <span class="n">addr_relative</span> <span class="o">=</span>
    <span class="p">((</span><span class="kt">uintptr_t</span><span class="p">)</span> <span class="n">addr</span> <span class="o">-</span> <span class="p">(</span><span class="kt">uintptr_t</span><span class="p">)</span> <span class="n">info</span><span class="p">.</span><span class="n">dli_fbase</span><span class="p">);</span>
  <span class="p">...</span>
<span class="p">}</span>
</code></pre></div></div>

<p>You have also the opportunity to catch a backtrace, with the same information, as long as you have a recent (ie. 4.1.1 or higher) Android version, using <a href="https://github.com/android/platform_system_core/tree/master/libcorkscrew"><code class="language-plaintext highlighter-rouge">libcorkscrew</code></a> library features. This library is not available on older Android releases, and besides, we do not want to get a backtrace of the <em>current</em> stack, but a backtrace of the stack provided in the crash context. Fortunately, we can dynamically load the <code class="language-plaintext highlighter-rouge">libcorkscrew.so</code> library to solve the first issue (using <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/dlopen.html"><code class="language-plaintext highlighter-rouge">dlopen</code></a> and <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/dlsym.html"><code class="language-plaintext highlighter-rouge">dlsym()</code></a>) and, for the second issue, import manually a nice function called <a href="https://github.com/android/platform_system_core/blob/master/libcorkscrew/backtrace-arch.h"><code class="language-plaintext highlighter-rouge">unwind_backtrace_signal_arch</code></a>, which does exactly what we want:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/*
* Describes a single frame of a backtrace.
*/</span>
<span class="k">typedef</span> <span class="k">struct</span> <span class="p">{</span>
    <span class="kt">uintptr_t</span> <span class="n">absolute_pc</span><span class="p">;</span> <span class="cm">/* absolute PC offset */</span>
    <span class="kt">uintptr_t</span> <span class="n">stack_top</span><span class="p">;</span> <span class="cm">/* top of stack for this frame */</span>
    <span class="kt">size_t</span> <span class="n">stack_size</span><span class="p">;</span> <span class="cm">/* size of this stack frame */</span>
<span class="p">}</span> <span class="n">backtrace_frame_t</span><span class="p">;</span>

<span class="kt">ssize_t</span> <span class="nf">unwind_backtrace_signal_arch</span><span class="p">(</span><span class="n">siginfo_t</span><span class="o">*</span> <span class="n">siginfo</span><span class="p">,</span> <span class="kt">void</span><span class="o">*</span> <span class="n">sigcontext</span><span class="p">,</span>
        <span class="k">const</span> <span class="n">map_info_t</span><span class="o">*</span> <span class="n">map_info_list</span><span class="p">,</span>
        <span class="n">backtrace_frame_t</span><span class="o">*</span> <span class="n">backtrace</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">ignore_depth</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">max_depth</span><span class="p">);</span>
</code></pre></div></div>

<p>(We also need to import <a href="https://github.com/android/platform_system_core/blob/master/include/corkscrew/map_info.h"><code class="language-plaintext highlighter-rouge">acquire_my_map_info_list()</code></a>)</p>

<p>We can also, when corkscrew is there, use the advanced <a href="https://github.com/android/platform_system_core/blob/master/include/corkscrew/backtrace.h"><code class="language-plaintext highlighter-rouge">get_backtrace_symbols</code></a> function to resolve symbols <em>and</em> demangle them:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/*
* Describes the symbols associated with a backtrace frame.
*/</span>
<span class="k">typedef</span> <span class="k">struct</span> <span class="p">{</span>
    <span class="kt">uintptr_t</span> <span class="n">relative_pc</span><span class="p">;</span> <span class="cm">/* relative frame PC offset from the start of the library, or the absolute PC if the library is unknown */</span>
    <span class="kt">uintptr_t</span> <span class="n">relative_symbol_addr</span><span class="p">;</span> <span class="cm">/* relative offset of the symbol from the start of the library or 0 if the library is unknown */</span>
    <span class="kt">char</span><span class="o">*</span> <span class="n">map_name</span><span class="p">;</span> <span class="cm">/* executable or library name, or NULL if unknown */</span>
    <span class="kt">char</span><span class="o">*</span> <span class="n">symbol_name</span><span class="p">;</span> <span class="cm">/* symbol name, or NULL if unknown */</span>
    <span class="kt">char</span><span class="o">*</span> <span class="n">demangled_name</span><span class="p">;</span> <span class="cm">/* demangled symbol name, or NULL if unknown */</span>
<span class="p">}</span> <span class="n">backtrace_symbol_t</span><span class="p">;</span>

<span class="cm">/*
* Gets the symbols for each frame of a backtrace.
* The symbols array must be big enough to hold one symbol record per frame.
* The symbols must later be freed using free_backtrace_symbols.
*/</span>
<span class="kt">void</span> <span class="nf">get_backtrace_symbols</span><span class="p">(</span><span class="k">const</span> <span class="n">backtrace_frame_t</span><span class="o">*</span> <span class="n">backtrace</span><span class="p">,</span> <span class="kt">size_t</span> <span class="n">frames</span><span class="p">,</span>
        <span class="n">backtrace_symbol_t</span><span class="o">*</span> <span class="n">backtrace_symbols</span><span class="p">);</span>
</code></pre></div></div>

<ul>
  <li>Fifth problem: you need to pass all these useful information back to the Java Virtual Machine, and not only by calling directly some kind of callback, because you need to propagate a <em>clean</em> <code class="language-plaintext highlighter-rouge">RuntimeException</code> and unwind all Java frames, to have your final exception being reported through the Android framework. The only way to achieve that is by storing the exit point in your own code, using <a href="http://pubs.opengroup.org/onlinepubs/009696799/functions/setjmp.html"><code class="language-plaintext highlighter-rouge">setjmp</code></a> (actually <a href="http://pubs.opengroup.org/onlinepubs/009695299/functions/sigsetjmp.html"><code class="language-plaintext highlighter-rouge">sigsetjmp</code></a>, because we’ll need to restore some masked signals), and using it in your signal handler to directly jump at the correct location. The <code class="language-plaintext highlighter-rouge">sigsetjmp</code>/<a href="http://pubs.opengroup.org/onlinepubs/009695299/functions/siglongjmp.html"><code class="language-plaintext highlighter-rouge">siglongjmp</code></a> functions are obviously not <em>async-signal-safe</em> (<a href="http://pubs.opengroup.org/onlinepubs/9699919799/functions/sigaction.html">see</a> the remark on Application Usage of <code class="language-plaintext highlighter-rouge">sigaction()</code>), so this is a highly risky bet. Typically, if the crash happened in the middle of a <code class="language-plaintext highlighter-rouge">malloc()</code> call (because, say, the linked list of free blocks has been corrupted), you may find yourself triggering another <code class="language-plaintext highlighter-rouge">SIGSEGV</code> (which is the lesser of the evils) or worse, deadlocked, which is rather embarrassing because the user will have to find a way to kill the application by himself. For this reason, an <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/alarm.html"><code class="language-plaintext highlighter-rouge">alarm()</code></a> call will be the first operation executed in case of emergency (and <em>yes</em>, <code class="language-plaintext highlighter-rouge">alarm()</code> is async-signal-safe - we can <em>safely</em> kill ourselves).</li>
</ul>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">static</span> <span class="kt">void</span> <span class="nf">my_handler</span><span class="p">(</span><span class="k">const</span> <span class="kt">int</span> <span class="n">code</span><span class="p">,</span> <span class="n">siginfo_t</span> <span class="o">*</span><span class="k">const</span> <span class="n">si</span><span class="p">,</span> <span class="kt">void</span> <span class="o">*</span><span class="k">const</span> <span class="n">sc</span><span class="p">)</span> <span class="p">{</span>
  <span class="cm">/* Call previous handler. */</span>
  <span class="n">old_handler</span><span class="p">.</span><span class="n">sa_sigaction</span><span class="p">(</span><span class="n">code</span><span class="p">,</span> <span class="n">si</span><span class="p">,</span> <span class="n">sc</span><span class="p">);</span>

  <span class="cm">/* Trigger a time bomb. */</span>
  <span class="p">(</span><span class="kt">void</span><span class="p">)</span> <span class="n">alarm</span><span class="p">(</span><span class="mi">30</span><span class="p">);</span>

  <span class="cm">/* Get thread-specific context. */</span>
  <span class="n">my_struct</span> <span class="o">*</span><span class="n">s</span> <span class="o">=</span> <span class="p">(</span><span class="n">my_struct</span><span class="o">*</span><span class="p">)</span> <span class="n">pthread_getspecific</span><span class="p">(</span><span class="n">my_thread_var</span><span class="p">);</span>
  <span class="k">if</span> <span class="p">(</span><span class="n">s</span> <span class="o">!=</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
    <span class="cm">/* Store crash context for later. */</span>
    <span class="n">s</span><span class="o">-&gt;</span><span class="n">code</span> <span class="o">=</span> <span class="n">code</span><span class="p">;</span>
    <span class="n">s</span><span class="o">-&gt;</span><span class="n">si</span> <span class="o">=</span> <span class="o">*</span><span class="n">si</span><span class="p">;</span>
    <span class="n">s</span><span class="o">-&gt;</span><span class="n">uc</span> <span class="o">=</span> <span class="o">*</span><span class="p">(</span><span class="n">ucontext_t</span><span class="o">*</span><span class="p">)</span> <span class="n">sc</span><span class="p">;</span>

    <span class="cm">/* Jump back to initial location. */</span>
    <span class="n">siglongjmp</span><span class="p">(</span><span class="n">t</span><span class="o">-&gt;</span><span class="n">ctx</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">);</span>
  <span class="p">}</span>
<span class="p">...</span>
<span class="p">}</span>
</code></pre></div></div>

<h2 id="is-it-working-">Is It Working ?</h2>

<p>Yes, and it’s much nicer to have <code class="language-plaintext highlighter-rouge">SIGSEGV</code> advertised through a clean stack:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FATAL EXCEPTION: AsyncTask #5
java.lang.RuntimeException: An error occured while executing doInBackground()
	at android.os.AsyncTask$3.done(AsyncTask.java:299)
	at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:352)
	at java.util.concurrent.FutureTask.setException(FutureTask.java:219)
	at java.util.concurrent.FutureTask.run(FutureTask.java:239)
	at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:230)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1080)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
	at java.lang.Thread.run(Thread.java:841)
Caused by: java.lang.Error: signal 11 (Address not mapped to object) at address 0x42 [at libhttrack.so:0xa024]
	at com.httrack.android.jni.HTTrackLib.main(Native Method)
	at com.httrack.android.HTTrackActivity$Runner.runInternal(HTTrackActivity.java:998)
	at com.httrack.android.HTTrackActivity$Runner.doInBackground(HTTrackActivity.java:919)
	at com.httrack.android.HTTrackActivity$Runner.doInBackground(HTTrackActivity.java:1)
	at android.os.AsyncTask$2.call(AsyncTask.java:287)
	at java.util.concurrent.FutureTask.run(FutureTask.java:234)
	... 4 more
Caused by: java.lang.Error: signal 11 (Address not mapped to object) at address 0x42 [at libhttrack.so:0xa024]
	at data.app_lib.com_httrack_android_2.libhttrack_so.0xa024(Native Method)
	at data.app_lib.com_httrack_android_2.libhttrack_so.0x705fc(hts_main2:0x8f74:0)
	at data.app_lib.com_httrack_android_2.libhtslibjni_so.0x4cc8(HTTrackLib_main:0xf8:0)
	at data.app_lib.com_httrack_android_2.libhtslibjni_so.0x52d8(Java_com_httrack_android_jni_HTTrackLib_main:0x64:0)
	at system.lib.libdvm_so.0x1dc4c(dvmPlatformInvoke:0x70:0)
	at system.lib.libdvm_so.0x4dcab(dvmCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*):0x18a:0)
	at system.lib.libdvm_so.0x385e1(dvmCheckCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*):0x8:0)
	at system.lib.libdvm_so.0x4f699(dvmResolveNativeMethod(unsigned int const*, JValue*, Method const*, Thread*):0xb8:0)
	at system.lib.libdvm_so.0x27060(Native Method)
	at system.lib.libdvm_so.0x2b580(dvmInterpret(Thread*, Method const*, JValue*):0xb8:0)
	at system.lib.libdvm_so.0x5fcbd(dvmCallMethodV(Thread*, Method const*, Object*, bool, JValue*, std::__va_list):0x124:0)
	at system.lib.libdvm_so.0x5fce7(dvmCallMethod(Thread*, Method const*, Object*, JValue*, ...):0x14:0)
	at system.lib.libdvm_so.0x54a6f(Native Method)
	at system.lib.libc_so.0xca58(__thread_entry:0x48:0)
	at system.lib.libc_so.0xcbd4(pthread_create:0xd0:0)
</code></pre></div></div>

<p><strong>Bonus</strong>: produce a relative address, and get the filename and line number for <strong>FREE</strong>!</p>

<p>In the above stack, we know that the crash occurred somewhere inside the <code class="language-plaintext highlighter-rouge">hts_main2</code> function, which is not very precise. We have actually another very useful information: the crash was spotted inside <code class="language-plaintext highlighter-rouge">libhttrack.so</code>, at the relative address <code class="language-plaintext highlighter-rouge">0xa024</code>. This is a <em>relative</em> address, computed earlier with <code class="language-plaintext highlighter-rouge">dladdr()</code>, which means that you can find out exactly the source location if you kept some debugging information. Most people do not want them, because it increases the binary size by a unreasonable factor (especially when running on small embedded devices with 3G+ connectivity priced above gold ingots levels), and thus either strip them silently, or keep another “debug” build.</p>

<p>You have an extremely simple alternative way: build once your libraries with all debugging symbols, including line numbers and macro information (<code class="language-plaintext highlighter-rouge">-g3</code>), and instead of stripping them, split the debugging sections on a separate file (say, a <code class="language-plaintext highlighter-rouge">.dbg</code> file). To let various tools such as <code class="language-plaintext highlighter-rouge">gdb</code> or <code class="language-plaintext highlighter-rouge">addr2line</code> behave gently, you have a way to “tell” them that the <code class="language-plaintext highlighter-rouge">.so</code> actually has a debug <code class="language-plaintext highlighter-rouge">.dbg</code> related file, through the <code class="language-plaintext highlighter-rouge">.gnu_debuglink</code> ELF section.</p>

<p>Here’s what you typically need to do to split your library into a stripped version plus a debug symbol file:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># copy all debugging sections to dbg file
objcopy --only-keep-debug mylib.so mylib.dbg
# strip debug sections
objcopy --strip-debug mylib.so
# wipe any existing ELF .gnu_debuglink section if any
objcopy --remove-section .gnu_debuglink mylib.so
# set the .gnu_debuglink to the dbg file
objcopy --add-gnu-debuglink=mylib.dbg mylib.so
</code></pre></div></div>

<p>The nice <code class="language-plaintext highlighter-rouge">.dbg</code> file can then be kept for debugging purpose:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cd /build-archives/httrack/armv7/3.47.99.35
./toolchains/arm-linux-androideabi-4.7/prebuilt/linux-x86_64/bin/arm-linux-androideabi-addr2line -C -f -e libhttrack.so 0xa024
fourty_two
src/htscoremain.c:111
</code></pre></div></div>

<h2 id="okay-but-is-it-really-safe-">Okay, but Is It Really Safe ?</h2>

<p>A typical crash may have a great variety of causes (Captain Obvious to the rescue!). <em>My</em> own experience, though, shows that NULL pointer dereferencing, dangling pointers, and other isolated crash spots due to bad code logic are a very common cause of crashes. Yes, you will still have troubles when dealing with corrupted allocators, or when breaking in the middle of a <em>async-signal-safe</em> call (leaving mutexes locked, and more generally a dirty state that will hit you back later), but the alternate solution is to die immediately (the default behavior), so trying to do gentle emergency steps can not do any harm.</p>

<h2 id="i-want-to-test-it">I Want To Test It!</h2>

<p>You can check out the “<strong>CoffeeCatch</strong>” library <a href="https://github.com/xroche/coffeecatch">code</a> on GitHub - you can either merge the <code class="language-plaintext highlighter-rouge">.c</code> file in your project(s), or build it as a standalone library ; this tiny library has no exotic external dependencies. Make sure all your libraries are built with the <a href="http://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html#index-funwind_002dtables-2380"><code class="language-plaintext highlighter-rouge">-funwind-tables</code></a> compiler flag to produce frame unwind information for all functions (add if necessary <code class="language-plaintext highlighter-rouge">LOCAL_CFLAGS := -funwind-tables</code> to your <code class="language-plaintext highlighter-rouge">Android.mk</code> file). Note that <code class="language-plaintext highlighter-rouge">-funwind-tables</code> does <strong>not</strong> <em>produce significant data size overhead</em> in normal situations - so rest well my friend.</p>

<p>The use is pretty straightforward - for JNI code, a simple macro <code class="language-plaintext highlighter-rouge">COFFEE_TRY_JNI</code>, taking the <code class="language-plaintext highlighter-rouge">JNIEnv</code> environment pointer as a first argument, and a code block as a second argument, will do the trick. For code outside JNI, you have to enclose protected code with COFFEE_TRY()/COFFEE_CATCH()/COFFEE_END() as in the example below.</p>

<p>In all cases, make sure the block is enclosed in a dedicated function without any local variables lying around (because the saved context does not include registers, AFAIK, especially volatile registers, which might be wiped before calling <code class="language-plaintext highlighter-rouge">sigsetjmp</code>, and whose saved values are in unknown location in the stack).</p>

<p><em>The JNI flavor:</em></p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/** The potentially dangerous function. **/</span>
<span class="n">jint</span> <span class="nf">call_dangerous_function</span><span class="p">(</span><span class="n">JNIEnv</span><span class="o">*</span> <span class="n">env</span><span class="p">,</span> <span class="n">jobject</span> <span class="n">object</span><span class="p">)</span> <span class="p">{</span>
  <span class="c1">// ... do dangerous things!</span>
  <span class="k">return</span> <span class="mi">42</span><span class="p">;</span>
<span class="p">}</span>

<span class="cm">/** Protected function stub. **/</span>
<span class="kt">void</span> <span class="nf">foo_protected</span><span class="p">(</span><span class="n">JNIEnv</span><span class="o">*</span> <span class="n">env</span><span class="p">,</span> <span class="n">jobject</span> <span class="n">object</span><span class="p">,</span> <span class="n">jint</span> <span class="o">*</span><span class="n">retcode</span><span class="p">)</span> <span class="p">{</span>
  <span class="cm">/* Try to call 'call_dangerous_function', and raise proper Java Error upon 
   * fatal error (SEGV, etc.). **/</span>
  <span class="n">COFFEE_TRY_JNI</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="o">*</span><span class="n">retcode</span> <span class="o">=</span> <span class="n">call_dangerous_function</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">object</span><span class="p">));</span>
<span class="p">}</span>

<span class="cm">/** Regular JNI entry point. **/</span>
<span class="n">jint</span> <span class="nf">Java_com_example_android_MyNative_foo</span><span class="p">(</span><span class="n">JNIEnv</span><span class="o">*</span> <span class="n">env</span><span class="p">,</span> <span class="n">jobject</span> <span class="n">object</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">jint</span> <span class="n">retcode</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
  <span class="n">foo_protected</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">object</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">retcode</span><span class="p">);</span>
  <span class="k">return</span> <span class="n">retcode</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>

<p><em>The standard flavor:</em></p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">static</span> <span class="nf">__attribute__</span> <span class="p">((</span><span class="n">noinline</span><span class="p">))</span> <span class="kt">void</span> <span class="n">demo</span><span class="p">(</span><span class="kt">int</span> <span class="o">*</span><span class="n">fault</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">COFFEE_TRY</span><span class="p">()</span> <span class="p">{</span>
    <span class="n">recurse_madness</span><span class="p">(</span><span class="mi">42</span><span class="p">);</span>
    <span class="o">*</span><span class="n">fault</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
  <span class="p">}</span> <span class="n">COFFEE_CATCH</span><span class="p">()</span> <span class="p">{</span>
    <span class="k">const</span> <span class="kt">char</span><span class="o">*</span><span class="k">const</span> <span class="n">message</span> <span class="o">=</span> <span class="n">coffeecatch_get_message</span><span class="p">();</span>
    <span class="n">snprintf</span><span class="p">(</span><span class="n">string_buffer</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">string_buffer</span><span class="p">),</span> <span class="s">"%s"</span><span class="p">,</span> <span class="n">message</span><span class="p">);</span>
    <span class="o">*</span><span class="n">fault</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
  <span class="p">}</span> <span class="n">COFFEE_END</span><span class="p">();</span>
<span class="p">}</span>
</code></pre></div></div>

<p><strong>TL;DR</strong>: so far, no native crash has been reported on Android for HTTrack. But I’m ready to collect them in case of need :)</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[HTTrack on Android]]></title>
    <link href="https://blog.httrack.com/blog/2013/08/13/httrack-on-android/"/>
    <updated>2013-08-13T11:40:00+02:00</updated>
    <id>https://blog.httrack.com/blog/2013/08/13/httrack-on-android</id>
    <content type="html"><![CDATA[<h2 id="discovering-a-new-world">Discovering a New World</h2>

<p>The first official Android release of HTTrack has just been <a href="https://play.google.com/store/apps/details?id=com.httrack.android">published</a> on the Android store.
<img src="http://android.httrack.com/snapshots/httrack_android_snapshot_welcome.png" alt="HTTrack on Android" title="HTTrack on Android" /></p>

<p>This was really fun to port HTTrack on this platform, actually. I never played too much with smartphones before (I usually don’t play at all with them, by the way, because everything I need to work or to connect to online resources is a real computer, not a smart-toy), but even if the httrack engine is 100% pure-C, the portage was not too difficult.</p>

<ul>
  <li>
    <p>the SDK provided by Google is <strong>clean</strong> enough do to complicated things without too much hassle (such as cross-compiling ARM bytecode on x86 machines) and is available on most platforms (at least Windows and Linux)</p>
  </li>
  <li>
    <p>the Eclipse-derivated developer environment (ADT) is rather <strong>well integrated</strong>, with all the nice features you would expect (including helping you tracking errors in obscure XML files, or starting gently device emulators)</p>
  </li>
  <li>
    <p>you do not even need any smartphone to start ; you can emulate on almost any type of devices (smartphones, tablets…) using <strong>the provided emulator</strong> and OS images</p>
  </li>
  <li>
    <p>the Android <strong>development API</strong> (in Java) is pretty straightforward, even when you start to mix native interface code (JNI)</p>
  </li>
</ul>

<p>To be clear, my first “hello, world” sample was created <strong>within minutes</strong>, and I then decided to directly start the httrack port.</p>

<h2 id="the-genesis">The Genesis</h2>

<p>I’m developing HTTrack on <strong>Linux</strong>, but I also use <strong>Windows</strong> as my main development platform, for practical reasons, and for the Windows release, which is currently the main “target” of HTTrack (and also, steam has more games on Windows, even if the Linux port is also quite excellent, but this is another story).</p>

<p>So I started by building the httrack engine code for android, of course. The first difficult task was to be able to build the original source code with the default build system (ie. Autotools/libtool). I could have created an Android mk file for that, but on the other hand I preferred to let libtool handle the dirty things (such as checking if v*printf was working, etc.) and have an unified build.</p>

<p>Another issue was the Android system itself: many common libraries, such as <code class="language-plaintext highlighter-rouge">iconv</code> or <code class="language-plaintext highlighter-rouge">openSSL</code>, are not available on this platform ; so I had to disable several features for the first releases.</p>

<p>After some tweaking, I finally got a working Makefile using something similar to:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./configure <span class="nt">-host</span><span class="o">=</span>arm-none-linux <span class="se">\</span>
	<span class="nt">--prefix</span><span class="o">=</span><span class="k">${</span><span class="nv">DEST_DIR</span><span class="k">}</span> <span class="se">\</span>
	<span class="nt">--enable-shared</span> <span class="nt">--disable-static</span> <span class="se">\</span>
	<span class="nt">--with-sysroot</span><span class="o">=</span><span class="k">${</span><span class="nv">ANDROID_SDK</span><span class="k">}</span>/platforms/android-5/arch-arm <span class="se">\</span>
	<span class="nt">--with-zlib</span><span class="o">=</span><span class="k">${</span><span class="nv">ANDROID_SDK</span><span class="k">}</span>/platforms/android-5/arch-arm/usr <span class="se">\</span>
	<span class="nt">--includedir</span><span class="o">=</span><span class="k">${</span><span class="nv">ANDROID_SDK</span><span class="k">}</span>/platforms/android-5/arch-arm/usr/include <span class="se">\</span>
	<span class="nv">CC</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">ANDROID_SDK</span><span class="k">}</span><span class="s2">/toolchains/arm-linux-androideabi-4.6/prebuilt/linux-x86_64/bin/arm-linux-androideabi-gcc --sysroot=</span><span class="k">${</span><span class="nv">ANDROID_SDK</span><span class="k">}</span><span class="s2">/platforms/android-5/arch-arm -I</span><span class="k">${</span><span class="nv">ANDROID_SDK</span><span class="k">}</span><span class="s2">/platforms/android-5/arch-arm/usr/include -L</span><span class="k">${</span><span class="nv">ANDROID_SDK</span><span class="k">}</span><span class="s2">/platforms/android-5/arch-arm/usr/lib"</span> <span class="se">\</span>
	<span class="nv">CFLAGS</span><span class="o">=</span><span class="s2">"-DANDROID -D_ANDROID -DLIBICONV_PLUG -I</span><span class="k">${</span><span class="nv">DEST_DIR</span><span class="k">}</span><span class="s2">/include -L</span><span class="k">${</span><span class="nv">DEST_DIR</span><span class="k">}</span><span class="s2">/lib"</span> <span class="se">\</span>
 <span class="o">&amp;&amp;</span> <span class="nb">sed</span> <span class="nt">-i</span> <span class="nt">-e</span> <span class="s1">'s/CPPFLAGS = .*/CPPFLAGS =/'</span> Makefile <span class="se">\</span>
 <span class="o">&amp;&amp;</span> <span class="nb">sed</span> <span class="nt">-i</span> <span class="nt">-e</span> <span class="s1">'s/CPPFLAGS = .*/CPPFLAGS =/'</span> src/Makefile <span class="se">\</span>
 <span class="o">&amp;&amp;</span> find <span class="nb">.</span> <span class="nt">-name</span> Makefile <span class="nt">-exec</span> <span class="nb">sed</span> <span class="nt">-i</span> <span class="nt">-e</span> <span class="s2">"s/</span><span class="se">\(</span><span class="s2">.*LDFLAGS = </span><span class="se">\)</span><span class="s2">-version-info .*/</span><span class="se">\1</span><span class="s2"> -avoid-version/"</span> <span class="o">{}</span> <span class="se">\;</span> <span class="se">\</span>
 <span class="o">&amp;&amp;</span> make <span class="nt">-j8</span> <span class="o">&amp;&amp;</span> make <span class="nb">install </span><span class="nv">DESTDIR</span><span class="o">=</span><span class="k">${</span><span class="nv">DEST_DIR</span><span class="k">}</span> <span class="se">\</span>
 <span class="o">&amp;&amp;</span> make <span class="nb">install</span>
</code></pre></div></div>

<p>Yuk, yes, I know, this is a little verbose, but, believe me, this is not as horrible as it seems.</p>

<p>The good thing is that I got rather quickly a <code class="language-plaintext highlighter-rouge">libhttrack.so</code> library containing everything I wanted (yay), without having two build systems. I was not too anxious, though: httrack is successfully built in a variety of platforms on <a href="http://packages.debian.org/search?keywords=httrack">Debian</a>, including ARM architectures.</p>

<h2 id="the-gui">The GUI</h2>

<p>The second step was to create some kind of interface (GUI) with the Android libraries, because you can not have (easily) a command line interface to play with. The first version only had a very minimalistic screen to play with (mostly a “Start” button).</p>

<p>Later, I added more cosmetic features, such as Next/Previous-style navigation, option panel with tabs, and some nice idiot-proof features (detecting common obvious mistakes, such as empty project name, empty URL, etc.)</p>

<p>One of the recurring complaint about the Windows version is the relative ugliness of its GUI. I must admit that my UI-design-skills are far from being perfect, and the original Windows GUI version took an insanely large amount of time to be developed (especially for an ugly result - but hey, I’m not developing a photo gallery, I’m developing a system utility).</p>

<p>In comparison, the Android GUI was much easier to develop (some people will probably find it likewise ugly - okay - but at least I did not spend two months on it), and much nicer in term of extensibility (adding tabs, scrollviews, etc. is pretty straightforward).</p>

<h2 id="the-dependencies">The Dependencies</h2>

<p>As I said earlier, neither iconv nor OpenSSL are officially provided on Android platforms.</p>

<ul>
  <li>
    <p>iconv is not provided at all (even in <code class="language-plaintext highlighter-rouge">libc</code>)</p>
  </li>
  <li>
    <p>OpenSSL is provided as binary-only (no include files in ther SDK)</p>
  </li>
</ul>

<p>iconv is the easier one to build ; Danilo Giulianelli <a href="http://danilogiulianelli.blogspot.fr/2012/12/how-to-cross-compile-libiconv-for.html">wrote</a> a very clear entry on how to build iconv on Android.</p>

<p>OpenSSL was more difficult, and I basically used the <a href="http://guardianproject.info/">Guardian Project</a> patched source at the beginning, only to discover that <code class="language-plaintext highlighter-rouge">/system/lib/libcrypto.so</code> and <code class="language-plaintext highlighter-rouge">/system/lib/libssl.so</code> are actually provided - you just need to get the include files, and hope the ABI is more or less stable. My experience with OpenSSL suggest that this should be true (I used to dynamically load OpenSSL in httrack on POSIX platforms - probing any available release - in a foolish attempt not to be tied to crazy exports regulations), and the rather small subset of OpenSSL functions I am using makes me confident that this… daring solution should not cause too much trouble.</p>

<h2 id="the-native-interface">The Native Interface</h2>

<p>In parallel, I started to write JNI interface code to be able to start the engine from the Java code.</p>

<p>This part was a bit more tricky, for a variety of reasons:</p>

<ul>
  <li>JNI code is not difficult, but very strict over <strong>global references</strong> handling, or the number of <strong>local objects references</strong> you may have on the stack etc.</li>
</ul>

<p>Therefore, if you do:</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">jclass</span> <span class="n">myClass</span> <span class="o">=</span> <span class="p">(</span><span class="o">*</span><span class="n">env</span><span class="p">)</span><span class="o">-&gt;</span><span class="n">FindClass</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">name</span><span class="p">);</span>
</code></pre></div></div>

<p>then you have to remember than <code class="language-plaintext highlighter-rouge">myClass</code> is only a <em>local</em> object reference, and if you keep it in some global variable, you’re gonne have a very bad time (ie. random crashes).</p>

<p>Therefore, you basically need to create a global reference:</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">static</span> <span class="n">jclass</span> <span class="nf">findClass</span><span class="p">(</span><span class="n">JNIEnv</span> <span class="o">*</span><span class="n">env</span><span class="p">,</span> <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span><span class="n">name</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">jclass</span> <span class="n">localClass</span> <span class="o">=</span> <span class="p">(</span><span class="o">*</span><span class="n">env</span><span class="p">)</span><span class="o">-&gt;</span><span class="n">FindClass</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">name</span><span class="p">);</span>
  <span class="cm">/* "Note however that the jclass is a class reference and must be protected
   * with a call to NewGlobalRef " -- DARN! */</span>
  <span class="k">if</span> <span class="p">(</span><span class="n">localClass</span> <span class="o">!=</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">return</span> <span class="p">(</span><span class="o">*</span><span class="n">env</span><span class="p">)</span><span class="o">-&gt;</span><span class="n">NewGlobalRef</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">localClass</span><span class="p">);</span>
  <span class="p">}</span>
  <span class="k">return</span> <span class="nb">NULL</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>

<p>quite obvious, yes, but forget this obvious point and you are doomed!</p>

<p>Another detail, by the way: when you throw an exception, remember that the string passed has to be valid during the lifetime of the underlying String object. The small details are often the most annoying things…</p>

<p>You have also to make sure you don’t have too many local reference objects on the current stack, because the default one is pretty small (512 references at most). Each time you create an object (including strings), each time you get a member from a structure, you create an additional local reference which needs to be deleted as soon as possible when running in inner JNI code loops.</p>

<p>For example, converting a <code class="language-plaintext highlighter-rouge">String[]</code> into a <code class="language-plaintext highlighter-rouge">char*[]</code> should look like:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  <span class="cm">/* Create array */</span>
  <span class="kt">size_t</span> <span class="n">i</span><span class="p">;</span>
  <span class="k">for</span> <span class="p">(</span><span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">argc</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
    <span class="cm">/* Note: a local reference is created here */</span>
    <span class="n">jstring</span> <span class="n">str</span> <span class="o">=</span> <span class="p">(</span><span class="n">jstring</span><span class="p">)(</span><span class="o">*</span><span class="n">env</span><span class="p">)</span><span class="o">-&gt;</span><span class="n">GetObjectArrayElement</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">stringArray</span><span class="p">,</span> <span class="n">i</span><span class="p">);</span>
    <span class="k">const</span> <span class="kt">char</span> <span class="o">*</span> <span class="k">const</span> <span class="n">utf_string</span> <span class="o">=</span> <span class="p">(</span><span class="o">*</span><span class="n">env</span><span class="p">)</span><span class="o">-&gt;</span><span class="n">GetStringUTFChars</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">str</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
    <span class="n">argv</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">strdup</span><span class="p">(</span><span class="n">utf_string</span> <span class="o">!=</span> <span class="nb">NULL</span> <span class="o">?</span> <span class="n">utf_string</span> <span class="o">:</span> <span class="s">""</span><span class="p">);</span>
    <span class="p">(</span><span class="o">*</span><span class="n">env</span><span class="p">)</span><span class="o">-&gt;</span><span class="n">ReleaseStringUTFChars</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">str</span><span class="p">,</span> <span class="n">utf_string</span><span class="p">);</span>
    <span class="p">(</span><span class="o">*</span><span class="n">env</span><span class="p">)</span><span class="o">-&gt;</span><span class="n">DeleteLocalRef</span><span class="p">(</span><span class="n">env</span><span class="p">,</span> <span class="n">str</span><span class="p">);</span>
  <span class="p">}</span>
  <span class="n">argv</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="nb">NULL</span><span class="p">;</span>
</code></pre></div></div>

<p>Of course, you also have the solution to create a temporary local frame (something Java do when calling functions, for example) which will allow to wipe all created local references using <code class="language-plaintext highlighter-rouge">PopLocalFrame</code>:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  <span class="cm">/* create a new local frame for local objects */</span>
  <span class="k">if</span> <span class="p">((</span><span class="o">*</span><span class="n">t</span><span class="o">-&gt;</span><span class="n">env</span><span class="p">)</span><span class="o">-&gt;</span><span class="n">PushLocalFrame</span><span class="p">(</span><span class="n">t</span><span class="o">-&gt;</span><span class="n">env</span><span class="p">,</span> <span class="n">capacity</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
    <span class="p">...</span>
    <span class="p">(</span><span class="kt">void</span><span class="p">)</span> <span class="p">(</span><span class="o">*</span><span class="n">t</span><span class="o">-&gt;</span><span class="n">env</span><span class="p">)</span><span class="o">-&gt;</span><span class="n">PopLocalFrame</span><span class="p">(</span><span class="n">t</span><span class="o">-&gt;</span><span class="n">env</span><span class="p">,</span> <span class="nb">NULL</span><span class="p">);</span>
  <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
    <span class="p">...</span> <span class="n">throw</span> <span class="n">something</span>
  <span class="p">}</span>
</code></pre></div></div>

<ul>
  <li>Crashes are difficult to track when playing with native code ; the ADT debugger provided is not always able to track where the crash did occur - you’d better have a clean code. I personally use an abundant amount of assert(), which are providing some bits of information before chocking:</li>
</ul>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/* Our own assert version. */</span>
<span class="k">static</span> <span class="kt">void</span> <span class="nf">assert_failure</span><span class="p">(</span><span class="k">const</span> <span class="kt">char</span><span class="o">*</span> <span class="n">exp</span><span class="p">,</span> <span class="k">const</span> <span class="kt">char</span><span class="o">*</span> <span class="n">file</span><span class="p">,</span> <span class="kt">int</span> <span class="n">line</span><span class="p">)</span> <span class="p">{</span>
  <span class="cm">/* FIXME TODO: pass the getExternalStorageDirectory() in init. */</span>
  <span class="kt">FILE</span> <span class="o">*</span><span class="k">const</span> <span class="n">dumpFile</span> <span class="o">=</span> <span class="n">fopen</span><span class="p">(</span><span class="s">"/mnt/sdcard/Download/HTTrack/error.txt"</span><span class="p">,</span> <span class="s">"wb"</span><span class="p">);</span>
  <span class="k">if</span> <span class="p">(</span><span class="n">dumpFile</span> <span class="o">!=</span> <span class="nb">NULL</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">fprintf</span><span class="p">(</span><span class="n">dumpFile</span><span class="p">,</span> <span class="s">"assertion '%s' failed at %s:%d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">exp</span><span class="p">,</span> <span class="n">file</span><span class="p">,</span> <span class="n">line</span><span class="p">);</span>
    <span class="n">fclose</span><span class="p">(</span><span class="n">dumpFile</span><span class="p">);</span>
  <span class="p">}</span>
  <span class="n">abort</span><span class="p">();</span>
<span class="p">}</span>
<span class="cp">#undef assert
#define assert(EXP) (void)( (EXP) || (assert_failure(#EXP, __FILE__, __LINE__), 0) )
</span></code></pre></div></div>

<p>I could also display some fancy message to the user directly, by calling the appropriate android java code…</p>

<ul>
  <li>You do not have a <code class="language-plaintext highlighter-rouge">LD_LIBRARY_PATH</code> ready with all your beloved libraries ; it means basically that you have to load all libraries and dependencies in reverse topological order, using <code class="language-plaintext highlighter-rouge">System.loadLibrary()</code> calls in static initializer. I also realized that some libraries, such as <code class="language-plaintext highlighter-rouge">OpenSSL</code>, are <em>actually</em> present on the system - wthout any headers being provided, though. Therefore, you have to dig a little to sort things out…</li>
</ul>

<p>The typical HTTrack library initializer is then looking like:</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  <span class="cm">/** OpenSSL (for HTTPS); taken from the /system libraries **/</span>
  <span class="n">System</span><span class="p">.</span><span class="n">loadLibrary</span><span class="p">(</span><span class="s">"crypto"</span><span class="p">);</span>
  <span class="n">System</span><span class="p">.</span><span class="n">loadLibrary</span><span class="p">(</span><span class="s">"ssl"</span><span class="p">);</span>

  <span class="cm">/** Iconv (Unicode). **/</span>
  <span class="n">System</span><span class="p">.</span><span class="n">loadLibrary</span><span class="p">(</span><span class="s">"iconv"</span><span class="p">);</span>

  <span class="cm">/** HTTrack core engine. **/</span>
  <span class="n">System</span><span class="p">.</span><span class="n">loadLibrary</span><span class="p">(</span><span class="s">"httrack"</span><span class="p">);</span>

  <span class="cm">/**
   * HTTrack Java plugin (note: dlopen()'ed by HTTrack, has symbol
   * dependencies to libhttrack.so).
   **/</span>
  <span class="n">System</span><span class="p">.</span><span class="n">loadLibrary</span><span class="p">(</span><span class="s">"htsjava"</span><span class="p">);</span>

  <span class="cm">/** HTTrack Android JNI layer. **/</span>
  <span class="n">System</span><span class="p">.</span><span class="n">loadLibrary</span><span class="p">(</span><span class="s">"htslibjni"</span><span class="p">);</span>
</code></pre></div></div>

<h2 id="gluing-every-pieces">Gluing Every Pieces</h2>

<p>Connecting all pieces together took a bit more time, too. On Android, you have to handle a variety of cases:</p>

<ul>
  <li>
    <p><strong>Permissions</strong>: this is probably one of the beginner’s first headache source. If you need to write to the external SDCard, for example, you will need to declare some special permission in your <code class="language-plaintext highlighter-rouge">Android.xml</code> application file (in this case, <code class="language-plaintext highlighter-rouge">android.permission.WRITE_EXTERNAL_STORAGE</code>) so that the system can let you do what you want. You won’t be able to connect to the outside world without <code class="language-plaintext highlighter-rouge">android.permission.INTERNET</code>, either.</p>
  </li>
  <li>
    <p><strong>Orientation change</strong>: when an user is changing the screen orientation (by rotating the device), your <a href="http://developer.android.com/reference/android/app/Activity.html">Activity</a> is by default killed, and recreated. This was a bit surprising for me, but fortunately you can put all working jobs on an isolated “<a href="http://developer.android.com/guide/components/fragments.html">fragment</a>” container, and let the system know that you may gently save and restore the state of your GUI (Activity) if needed:</p>
  </li>
</ul>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  <span class="cm">/* First attempt to reattach to an existing background job */</span>
  <span class="n">final</span> <span class="n">FragmentManager</span> <span class="n">fm</span> <span class="o">=</span> <span class="n">getSupportFragmentManager</span><span class="p">();</span>
  <span class="n">runner</span> <span class="o">=</span> <span class="p">(</span><span class="n">RunnerFragment</span><span class="p">)</span> <span class="n">fm</span><span class="p">.</span><span class="n">findFragmentByTag</span><span class="p">(</span><span class="n">id</span><span class="p">);</span>

  <span class="cm">/* Create a new "fragment" (our background job runner) */</span>
  <span class="k">if</span> <span class="p">(</span><span class="n">runner</span> <span class="o">==</span> <span class="n">null</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">final</span> <span class="n">FragmentManager</span> <span class="n">fm</span> <span class="o">=</span> <span class="n">getSupportFragmentManager</span><span class="p">();</span>
    <span class="n">runner</span> <span class="o">=</span> <span class="n">new</span> <span class="n">RunnerFragment</span><span class="p">();</span>
    <span class="n">runner</span><span class="p">.</span><span class="n">setParent</span><span class="p">(</span><span class="n">this</span><span class="p">);</span>
    <span class="n">fm</span><span class="p">.</span><span class="n">beginTransaction</span><span class="p">().</span><span class="n">add</span><span class="p">(</span><span class="n">runner</span><span class="p">,</span> <span class="n">id</span><span class="p">).</span><span class="n">commit</span><span class="p">();</span>
  <span class="p">}</span>
</code></pre></div></div>

<p>You also need to overrivde <code class="language-plaintext highlighter-rouge">onSaveInstanceState</code> and <code class="language-plaintext highlighter-rouge">onRestoreInstanceState</code> in your Activity to be able to retrieve your current context.</p>

<ul>
  <li>
    <p>Your application <strong>can be killed at any time</strong>, for example if the user hits the “back” button to exit your main GUI. You have to take care of saving necessary settings, and stop running background jobs (especially native ones!) gently, and if necessary suggest to restart an interrupted job later.</p>
  </li>
  <li>
    <p><strong>Limited amount of resources</strong>: especially when playing with JNI, you have to take extreme care of how many objects, how many local references etc. you are using (see above remarks). You also have to take care on where you are storing data - writing directly on the SDcard root is considered a bad practice ; you also may not have any SDcard at all on the device, etc.</p>
  </li>
</ul>

<h2 id="at-last">At Last</h2>

<p>It took me basically two weeks to have a final release. Oh, I’m not saying that the first release is perfect, and you may experience bugs and problems, but the first tests were quite successful.</p>

<p>And it was a rather pleasant time - considering my total lack of experience on this field, the portage was rather straightforward, even with a project involving not trivial tasks (JNI, background jobs, etc.)</p>

<p>If you want to check out the code, feel free to <a href="http://code.google.com/p/httrack/source/browse/#svn%2Fandroid%2Ftrunk">browse it</a>.</p>

<p><strong>TL;DR</strong>: developing applications on Android, even including native code, is rather straightforward!</p>
]]></content>
  </entry>
  
  <entry>
    <title type="html"><![CDATA[MD5 is Your Friend]]></title>
    <link href="https://blog.httrack.com/blog/2013/07/19/md5-is-your-friend/"/>
    <updated>2013-07-19T18:29:00+02:00</updated>
    <id>https://blog.httrack.com/blog/2013/07/19/md5-is-your-friend</id>
    <content type="html"><![CDATA[<h2 id="md5-is-your-friend-really-">MD5 is Your Friend, Really ?</h2>

<p>Yes, <a href="https://en.wikipedia.org/wiki/MD5">MD5</a> is your friend. Believe me.</p>

<h2 id="why-is-md5-your-friend-">Why is MD5 Your Friend ?</h2>

<p>My first encounter with MD5 was around fifteen years ago, when I implemented the first <a href="/blog/2013/07/12/so-i-made-a-hashtable/">hashtable</a> in httrack. The code was not really glorious, but at least I quickly discovered something: <em>if you design your own hashing function, you’re gonna have a bad time</em>.</p>

<p>Here’s what I first implemented (please do not scream):</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">unsigned</span> <span class="kt">long</span> <span class="kt">int</span> <span class="nf">hash_for_dummies</span><span class="p">(</span><span class="k">const</span> <span class="kt">unsigned</span> <span class="kt">char</span><span class="o">*</span> <span class="n">s</span><span class="p">)</span> <span class="p">{</span>
  <span class="kt">unsigned</span> <span class="kt">long</span> <span class="n">sum</span><span class="p">;</span>
  <span class="kt">unsigned</span> <span class="kt">int</span> <span class="n">i</span><span class="p">;</span>
  <span class="k">for</span><span class="p">(</span><span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="n">sum</span> <span class="o">=</span> <span class="mi">0</span> <span class="p">;</span> <span class="o">*</span><span class="n">s</span> <span class="o">!=</span> <span class="sc">'\0'</span> <span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">sum</span> <span class="o">+=</span> <span class="mi">1</span><span class="p">;</span>
    <span class="n">sum</span> <span class="o">+=</span> <span class="p">(</span><span class="kt">unsigned</span> <span class="kt">int</span><span class="p">)</span> <span class="o">*</span><span class="n">s</span><span class="p">;</span>
    <span class="n">sum</span> <span class="o">*=</span> <span class="p">(</span><span class="kt">unsigned</span> <span class="kt">int</span><span class="p">)</span> <span class="o">*</span><span class="p">(</span><span class="n">s</span><span class="o">++</span><span class="p">);</span>
    <span class="n">sum</span> <span class="o">+=</span> <span class="n">i</span><span class="p">;</span>
  <span class="p">}</span>
  <span class="k">return</span> <span class="n">sum</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Well, on several (I mean, on one of two) examples, it was kind of a cool hashing function actually. Of course, once being used inside my hashtable, things did not go so well, and on several sites with special URL patterns, the table exploded (in term of complexity).</p>

<p>Oh yes, I forgot to mention: the hashing function was meant to be used in a hashtable, and the keys being hashed were URLs. The funny thing with URLs is that you may have any pattern:</p>

<ul>
  <li>totally randomly distributed URLs (with nice random-looking <code class="language-plaintext highlighter-rouge">/session/?5209d5f396ca0d46</code>)</li>
  <li>not so random URLs (plain text URLs, such as <code class="language-plaintext highlighter-rouge">/I-made-a-huge-mistake</code>)</li>
  <li>really not very random URLs (<code class="language-plaintext highlighter-rouge">/page/1</code>, <code class="language-plaintext highlighter-rouge">/page/2</code>, … <code class="language-plaintext highlighter-rouge">/page/999999</code>)</li>
</ul>

<p>When hashing with a specific function, you may have very good results for one of specific pattern, or for several families, but time to time you discover that the hashing function betrayed you in specific cases.</p>

<h2 id="so-md5-was-the-solution-">So, MD5 was the Solution ?</h2>

<p>Yes, yes, but you ruined the plot! Okay, that was expected, but yet, after discovering my poor hashing abilities, I decided to use the thermo-nuclear option: MD5, the cryptographic hashing function, which has been studied during long years, and which is considered an excellent solution for hashing anything.</p>

<p>And it was good. Hardly no collisions. Perfect distribution of keys. No nightmares about specific URL patterns that would ruin everything. MD5 was perfect.</p>

<h2 id="and-costly-switch-to-fnv-1-or-murmur">And Costly. Switch to FNV-1 or Murmur!</h2>

<p>Ah, yes, the cost. <a href="http://code.google.com/p/httrack/source/browse/trunk/src/md5.c">MD5</a> has a bad reputation of being over-costly. If you look at the internal loop, you will discover an intimidating block of 64 MD5STEP macros such as:</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#define F1(x, y, z) (z ^ (x &amp; (y ^ z)))
#define MD5STEP(f, w, x, y, z, data, s) \
( w += f(x, y, z) + data,  w = w&lt;&lt;s | w&gt;&gt;(32-s),  w += x )
</span><span class="p">...</span>
  <span class="n">MD5STEP</span><span class="p">(</span><span class="n">F1</span><span class="p">,</span> <span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">c</span><span class="p">,</span> <span class="n">d</span><span class="p">,</span> <span class="n">in</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">+</span> <span class="mh">0xd76aa478</span><span class="p">,</span> <span class="mi">7</span><span class="p">);</span>
  <span class="n">MD5STEP</span><span class="p">(</span><span class="n">F1</span><span class="p">,</span> <span class="n">d</span><span class="p">,</span> <span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">c</span><span class="p">,</span> <span class="n">in</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">+</span> <span class="mh">0xe8c7b756</span><span class="p">,</span> <span class="mi">12</span><span class="p">);</span>
<span class="c1">// ... during 64 lines</span>
</code></pre></div></div>

<p>If you look <em>closer</em>, the internal loop function is actually processing 64 bytes at once (with final zero-padding) ; which mean that for each byte, a dozen simple (<code class="language-plaintext highlighter-rouge">add</code>, <code class="language-plaintext highlighter-rouge">xor</code>, shift…) operations is executed. That’s a lot, and not so much at the same time. Most operations can be executed in one cycle or less (pipelined) by a modern processor - and it has to be compared to more expensive operations (such as <code class="language-plaintext highlighter-rouge">mul</code>) used in other hashes.</p>

<p>So yes, MD5 is probably slower. Well, actually it is:</p>

<table>
  <thead>
    <tr>
      <th>MD5</th>
      <th>FNV-1</th>
      <th>Murmur 3</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>6.0s</td>
      <td>4.40s</td>
      <td>1.54s</td>
    </tr>
  </tbody>
</table>

<p>(<em>Tests executed on a 3801493504 bytes file, on a Xeon(R) CPU E5-1620 0 @ 3.60GHz</em>)</p>

<p>Basically <code class="language-plaintext highlighter-rouge">50%</code> slower than some <a href="https://en.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo_hash_function">FNV-1</a> variant, and <code class="language-plaintext highlighter-rouge">four times</code> slower than <a href="https://en.wikipedia.org/wiki/MurmurHash">Murmur3</a>.</p>

<h2 id="thats-horrible">That’s Horrible!</h2>

<p>No, actually not. Remember one thing: we’re not hashing the universe, we’re hashing small <em>keys</em> (No, keys are not supposed to be huge binary blobs). We have plenty of work to do after that (like, uh, storing the value in the hashtable, and probably many other interesting things with them), and the keys hashing is only a small subset of the code.</p>

<p>If you do more extensive tests, such as, inserting and reading to a hashtable, things are less clear:</p>

<table>
  <thead>
    <tr>
      <th>FNV-1</th>
      <th>MD5</th>
      <th>Murmur</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>1m13s</td>
      <td>1m54</td>
      <td>1m13s</td>
    </tr>
  </tbody>
</table>

<p>(<em>Tests executed on 10000000 keys progressively, with keys of length between 80 and 100 bytes, with insert/overwrite/delete patterns.</em>)</p>

<p>Yep, basically MD5 is only increasing <em>marginally</em> (well, yes, I call 30% “<em>marginally</em>” - I know everybody would love to earn marginally more) the overall hashtable cost.</p>

<h2 id="but-still-it-is-slower">But Still, It Is Slower</h2>

<p>Yes. And <strong>I don’t care</strong>. Remember the URLs pattern I was talking about previously ? Well, with MD5, I can sleep quietly. I don’t have to worry whether the hash function has been studied correctly. MD5 has. And even if on several tests Murmur has proven to be a reliable hashing function (with the same number of collisions than MD5, basically), I’m still reluctant in exchanging a little CPU cost with a bit of algorithmic uncertainty.</p>

<p><strong>TL;DR</strong>: I stopped worrying and learned to love MD5.</p>

<p><em>Note: during my various tests, I used 64-bit hashing functions variants, with shift+mask to isolate the number of bits needed (ie. the hashtable bucket size was a power of two). The hashes were always processed by computing 128-bit hashing data, XOR-folded into a 64 bit version.</em></p>
]]></content>
  </entry>
  
</feed>
