Jekyll2024-01-10T19:01:59+00:00https://arteymix.github.io/feed.xmlA Day in a Pile of WorkMy personal Web development blogGuillaume Poirier-Morency <guillaumepoiriermorency@gmail.com>Starting a Consulting Company2021-04-20T00:00:00+00:002021-04-20T00:00:00+00:00https://arteymix.github.io/2021/04/20/starting-a-consulting-company<p>Here we go. I just registered my new venture, a sole proprietorship named
“Guillaume Poirier-Morency Consulting Co.”. It will focus on providing
consulting services in my areas of expertise such as computer science,
programming, bioinformatics, and statistical modelling.</p>
<p>I have a few core ideas:</p>
<ul>
<li>Apply my expertise to help out people make the most out of computer science
in their projects. That could range from better understanding and explaining
the challenges underlying their scientific questions to building working
prototypes and solutions.</li>
<li>Use cloud services to power solutions and avoid as much as possible the
burdens of maintenance.</li>
<li>Provide solutions for customers and release as much as possible under free
software licenses.</li>
<li>Use free software for business operations (i.e. GNU Cash, LibreOffice, etc.).</li>
</ul>
<p>It’s also important for me to avoid competing interests with my work and make
sure that I preserve an overall sane work-life balance. To that end, I will pick
projects carefully.</p>
<p>I might also deviate a bit into uncharted territories such as personal income
tax. I’ve been helping friends and family members recently with their tax
returns, and I know a fair bit about the available deductions, credits, etc. to
help out people.</p>
<p>No logo, graphics, or website yet. Just an idea. That’s it.</p>Guillaume Poirier-Morency <guillaumepoiriermorency@gmail.com>Here we go. I just registered my new venture, a sole proprietorship named “Guillaume Poirier-Morency Consulting Co.”. It will focus on providing consulting services in my areas of expertise such as computer science, programming, bioinformatics, and statistical modelling.Zero-copy FASTA Parser2017-08-10T00:00:00+00:002017-08-10T00:00:00+00:00https://arteymix.github.io/2017/08/10/zero-copy-fasta-parser<p>Where I work, we deal often deal with large datasets from which we copy the
relevant entries into the program memory. However, doing so typically incurs
a very large usage of memory, which could leads to memory-bound parallelism if
multiple instances are launched.</p>
<p>The memory-bound parallelism issue is arising when a system cannot execute more
tasks due to a lack of available memory. It is essentially wasting of all other
available resources such as CPU time.</p>
<p>To address this kind of issue, I’ll describe in this post a strategy using
memory-mapped files and on-demand processing over a very common data format in
bioinformatics: FASTA. The use case is pretty simple: we want to query small
and arbitrary subsequences without having to precondition them in allocated
memory.</p>
<h3 id="about-virtual-memory-space">About Virtual Memory Space</h3>
<p>The virtual address space is large, very. Think of all the addresse values a 64
bit pointer can take. That’s about 18 quitillions of addressable bytes, which
is enough to never be bothered with.</p>
<p>Understandbly, no computer can hold that much of memory. Instead, the operating
system partitions the virtual memory into pages and the physical memory into
frames. It uses a cache algorithm and load addressed pages into physical
frames. Unused pages are stored on disk, in the available swap partitions or
compressed into physical memory if you use Zswap<sup id="fnref:zswap" role="doc-noteref"><a href="#fn:zswap" class="footnote" rel="footnote">1</a></sup>.</p>
<p>The <code class="language-plaintext highlighter-rouge">mmap</code><sup id="fnref:mmap" role="doc-noteref"><a href="#fn:mmap" class="footnote" rel="footnote">2</a></sup> system call makes a correspondance between a file and pages in
virtual memory. Addressing the memory where the file has been mapped will
result in the kernel fetching its content dynamically. Moreover, if multiple
processes map the same file, the same frames (i.e. physical memory) will be
used across all of them.</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">void</span> <span class="o">*</span> <span class="nf">mmap</span> <span class="p">(</span><span class="kt">void</span> <span class="o">*</span><span class="n">addr</span><span class="p">,</span>
<span class="kt">size_t</span> <span class="n">length</span><span class="p">,</span>
<span class="kt">int</span> <span class="n">prot</span><span class="p">,</span>
<span class="kt">int</span> <span class="n">flags</span><span class="p">,</span>
<span class="kt">int</span> <span class="n">fd</span><span class="p">,</span>
<span class="kt">off_t</span> <span class="n">offset</span><span class="p">);</span>
</code></pre></div></div>
<p>Where <code class="language-plaintext highlighter-rouge">addr</code> hint the operating system for a memory location, <code class="language-plaintext highlighter-rouge">length</code> indicates
the size of the mapping, <code class="language-plaintext highlighter-rouge">prot</code> indicates permissions on the region, <code class="language-plaintext highlighter-rouge">flags</code>
holds various options, <code class="language-plaintext highlighter-rouge">fd</code> is a file descriptor and <code class="language-plaintext highlighter-rouge">offset</code> is a byte offset
from the file content. The returned value is the mapped address.</p>
<p>We can use this feature at our advantage by loading our data once and
transparently share them across all the instances of our program.</p>
<p>I’m using GLib, a portable C library, and its providen <code class="language-plaintext highlighter-rouge">GMappedFile</code> to
carefully wrap <code class="language-plaintext highlighter-rouge">mmap</code> with reference counting.</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">g_auto</span> <span class="p">(</span><span class="n">GMappedFile</span><span class="p">)</span> <span class="n">fasta_map</span> <span class="o">=</span> <span class="n">g_mapped_file_new</span> <span class="p">(</span><span class="s">"hg38.fa"</span><span class="p">,</span>
<span class="nb">false</span><span class="p">);</span>
</code></pre></div></div>
<h3 id="our-use-case">Our Use Case</h3>
<p>To be more specific our use case only require to view small windows (~7
nucleotides) of the sequence at once. If we assume 80 nucleotides per line, we
have 80 possible windows from which 73 are free of newlines. The probability
for a random subsequence of length 7 of landing on a newline is thus
approximately 8.75%.</p>
<p>For the great majority of cases, assuming uniformly distributed subsequence
requests, we can simply return the address from the mapped memory.</p>
<p>From now on, we assume that the in-memory mapped document has already been
indexed by bookeeping the beginnings of each sequences, which can be easily
done with <code class="language-plaintext highlighter-rouge">memchr</code><sup id="fnref:memchr" role="doc-noteref"><a href="#fn:memchr" class="footnote" rel="footnote">3</a></sup>. The <code class="language-plaintext highlighter-rouge">sequence</code> pointer points to some start of
a sequence and <code class="language-plaintext highlighter-rouge">sequence_len</code> indicate the length before the next one.</p>
<p>To work efficiently, it is worth to index the newlines. For this purpose, we
use a <code class="language-plaintext highlighter-rouge">GPtrArray</code>, which is a simple pointer array implementation that we
populate with the addresses of the newlines in the mapped buffer.</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">const</span> <span class="n">gchar</span> <span class="o">*</span><span class="n">sequence</span> <span class="o">=</span> <span class="s">"ACTG</span><span class="se">\n</span><span class="s">ACTG"</span><span class="p">;</span>
<span class="n">gsize</span> <span class="n">sequence_len</span> <span class="o">=</span> <span class="mi">9</span><span class="p">;</span>
<span class="n">g_autoptr</span> <span class="p">(</span><span class="n">GPtrArray</span><span class="p">)</span> <span class="n">sequence_skips</span> <span class="o">=</span>
<span class="n">g_ptr_array_sized_new</span> <span class="p">(</span><span class="n">sequence_len</span> <span class="o">/</span> <span class="mi">80</span><span class="p">);</span> <span class="c1">// line feed every 80 characters</span>
<span class="k">const</span> <span class="n">gchar</span><span class="o">*</span> <span class="n">seq</span> <span class="o">=</span> <span class="n">sequence</span><span class="p">;</span>
<span class="k">while</span> <span class="p">((</span><span class="n">seq</span> <span class="o">=</span> <span class="n">memchr</span> <span class="p">(</span><span class="n">seq</span><span class="p">,</span> <span class="sc">'\n'</span><span class="p">,</span> <span class="n">sequence_len</span> <span class="o">-</span> <span class="p">(</span><span class="n">seq</span> <span class="o">-</span> <span class="n">sequence</span><span class="p">))))</span>
<span class="p">{</span>
<span class="n">g_ptr_array_add</span> <span class="p">(</span><span class="n">sequence_skips</span><span class="p">,</span> <span class="p">(</span><span class="n">gpointer</span><span class="p">)</span> <span class="n">seq</span><span class="p">);</span>
<span class="n">seq</span><span class="o">++</span><span class="p">;</span> <span class="c1">// jump right after the line feed</span>
<span class="p">}</span>
</code></pre></div></div>
<p>A newline can either preceed, follow or land within the subsequence.</p>
<ul>
<li>all thoses preceeding the desired subsquence shifts the sequence to the right</li>
<li>all those within the subsequence must be stripped</li>
<li>the remaining newlines can be safely ignored</li>
</ul>
<p>If only the first or last condition apply, we’re in the 92.5% of the cases as
we can simply return the corresponding memory address.</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">gsize</span> <span class="n">subsequence_offset</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="n">gsize</span> <span class="n">subsequence_len</span> <span class="o">=</span> <span class="mi">7</span><span class="p">;</span>
</code></pre></div></div>
<p>We first position our subsequence at its initial location.</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">const</span> <span class="n">gchar</span> <span class="o">*</span><span class="n">subsequence</span> <span class="o">=</span> <span class="n">sequence</span> <span class="o">+</span> <span class="n">subsequence_offset</span><span class="p">;</span>
</code></pre></div></div>
<p>We need some bookkeeping for filling a fixed-width buffer if a newline land
within our subsequence.</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">static</span> <span class="n">gchar</span> <span class="n">subsequence_buffer</span><span class="p">[</span><span class="mi">64</span><span class="p">];</span>
<span class="n">gsize</span> <span class="n">subsequence_buffer_offset</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
</code></pre></div></div>
<p>Now, for each linefeed we’ve collected, we’re going to test our three
conditions and either move the subsequence right or fill the static buffer.</p>
<p>The second condition require some work. Using the indexed newlines, we
basically trim the sequence into a static buffer that is returned. Although we
lose thread safety working this way, it will be mitigated by process-level
parallelism.</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">gint</span> <span class="n">i</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span><span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">sequence_skips</span><span class="o">-></span><span class="n">len</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">const</span> <span class="n">gchar</span> <span class="o">*</span><span class="n">linefeed</span> <span class="o">=</span> <span class="n">g_ptr_array_index</span> <span class="p">(</span><span class="n">sequence_skips</span><span class="p">,</span> <span class="n">i</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">linefeed</span> <span class="o"><=</span> <span class="n">subsequence</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">subsequence</span><span class="o">++</span><span class="p">;</span> <span class="c1">// move the subsequence right</span>
<span class="p">}</span>
<span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="n">linefeed</span> <span class="o"><</span> <span class="n">subsequence</span> <span class="o">+</span> <span class="n">subsequence_len</span><span class="p">)</span>
<span class="p">{</span>
<span class="c1">// length until the next linefeed</span>
<span class="n">gsize</span> <span class="n">len_to_copy</span> <span class="o">=</span> <span class="n">linefeed</span> <span class="o">-</span> <span class="n">subsequence</span><span class="p">;</span>
<span class="n">memcpy</span> <span class="p">(</span><span class="n">subsequence_buffer</span> <span class="o">+</span> <span class="n">subsequence_buffer_offset</span><span class="p">,</span>
<span class="n">subsequence</span><span class="p">,</span>
<span class="n">len_to_copy</span><span class="p">);</span>
<span class="n">subsequence_buffer_offset</span> <span class="o">+=</span> <span class="n">len_to_copy</span><span class="p">;</span>
<span class="n">subsequence</span> <span class="o">+=</span> <span class="n">len_to_copy</span> <span class="o">+</span> <span class="mi">1</span><span class="p">;</span> <span class="c1">// jump right after the linefeed</span>
<span class="p">}</span>
<span class="k">else</span>
<span class="p">{</span>
<span class="k">break</span><span class="p">;</span> <span class="c1">// linefeed supersedes the subsequence</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Lastly we check whether or not we’ve used the static buffer, in which case we
copy any trailing sequence.</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="p">(</span><span class="n">subsequence_buffer_offset</span> <span class="o">></span> <span class="mi">0</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">subsequence_buffer_offset</span> <span class="o"><</span> <span class="n">subsequence_len</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">memcpy</span> <span class="p">(</span><span class="n">subsequence_buffer</span> <span class="o">+</span> <span class="n">subsequence_buffer_offset</span><span class="p">,</span>
<span class="n">subsequence</span><span class="p">,</span>
<span class="n">subsequence_len</span> <span class="o">-</span> <span class="n">subsequence_buffer_offset</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">subsequence_buffer</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">else</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">subsequence</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>It’s possible to use a binary search strategy to obtain the range of newlines
affecting the position of the requested subsequence, but since the number of
newlines is considerably small, I ignored this optimization so far.</p>
<p>Here we are with our zero-copy FASTA parser that efficiently look for small
subsequences.</p>
<p>P.S.: This technique has been used for the C rewrite of miRBooking<sup id="fnref:mirbooking" role="doc-noteref"><a href="#fn:mirbooking" class="footnote" rel="footnote">4</a></sup>
I’ve been working on these past weeks.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:zswap" role="doc-endnote">
<p><a href="https://www.kernel.org/doc/Documentation/vm/zswap.txt">https://www.kernel.org/doc/Documentation/vm/zswap.txt</a> <a href="#fnref:zswap" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:mmap" role="doc-endnote">
<p><a href="http://man7.org/linux/man-pages/man2/mmap.2.html">http://man7.org/linux/man-pages/man2/mmap.2.html</a> <a href="#fnref:mmap" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:memchr" role="doc-endnote">
<p><a href="http://man7.org/linux/man-pages/man3/memchr.3.html">http://man7.org/linux/man-pages/man3/memchr.3.html</a> <a href="#fnref:memchr" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:mirbooking" role="doc-endnote">
<p><a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4538818/">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4538818/</a> <a href="#fnref:mirbooking" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Guillaume Poirier-Morency <guillaumepoiriermorency@gmail.com>Where I work, we deal often deal with large datasets from which we copy the relevant entries into the program memory. However, doing so typically incurs a very large usage of memory, which could leads to memory-bound parallelism if multiple instances are launched.Valadoc.org Rewrite and More!2017-02-25T00:00:00+00:002017-02-25T00:00:00+00:00https://arteymix.github.io/valum/2017/02/25/valadoc-org-and-more<p>The rewrite of <a href="https://valadoc.org/">valadoc.org</a> in Vala using Valum has been
completed and should be deployed eventually be elementary OS team (see <a href="https://github.com/Valadoc/valadoc-org/pull/40">pull
#40</a>). There’s a couple of interesting stuff there too:</p>
<ul>
<li>experimental search API using JSON via the <code class="language-plaintext highlighter-rouge">/search</code> endpoint</li>
<li><a href="https://github.com/chergert/glrucache">GLruCache</a> now has Vala bindings and an improved API</li>
<li>an eventual GMysql wrapper around the C client API if extracting the classes
I wrote is worth it</li>
</ul>
<p>In the meantime, you can test it at <a href="https://valadoc2.elementary.io/">valadoc2.elementary.io</a>
and report any regression on the pull-request.</p>
<p>Valum 0.3 has been patched and improved while I have been working on the 0.4
feature set. There’s a <a href="https://github.com/valum-framework/valum/pull/206">work-in-progress WebSocket middleware</a>,
VSGI 1.0 and support for PyGObject planned.</p>
<p>If everything goes as planned, I should finish the AJP backend and maybe
consider Lwan.</p>
<p>On the top, there’s Windows support coming, although the most difficult part is
to test it. I might need some help there to setup AppVeyor CI.</p>
<p>I’m aware of the harsh discussions about the state of Vala and whether or not
it will just end into an abysmal void. I would advocate inertia here: the
current state of the language still make it an excelllent candidate for
writing GNOME-related software and this is not expected to change.</p>Guillaume Poirier-Morency <guillaumepoiriermorency@gmail.com>The rewrite of valadoc.org in Vala using Valum has been completed and should be deployed eventually be elementary OS team (see pull #40). There’s a couple of interesting stuff there too:Announcing Valum 0.32016-11-11T00:00:00+00:002016-11-11T00:00:00+00:00https://arteymix.github.io/valum/2016/11/11/announcing-valum-0-3<p>The first release candidate for Valum 0.3 has been launched today!</p>
<p><a href="https://github.com/valum-framework/valum/releases/tag/v0.3.0-rc">Get</a> it, test it and be the first to find a bug! The final release will
come shortly after along with various Linux distributions packages.</p>
<p>This post review the changes and features that have been introduced since the
<code class="language-plaintext highlighter-rouge">0.2</code>. There’s been a lot of work, so take a comfortable seat and brew yourself
a strong coffee.</p>
<p>The most significant change has probably been the introduction of Meson as
a build system and all the new deployment strategy it now makes possible.</p>
<p>If you prefer avoiding a full install, it’s not possible to use it as
a subproject. These are defined as subdirectories of <code class="language-plaintext highlighter-rouge">subprojects</code>, which you
can conveniently track using <a href="https://git-scm.com/docs/git-submodule">git submodules</a>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">project</span><span class="p">(</span><span class="s">''</span><span class="p">,</span> <span class="s">'c'</span><span class="p">,</span> <span class="s">'vala'</span><span class="p">)</span>
<span class="n">glib</span> <span class="o">=</span> <span class="n">dependency</span><span class="p">(</span><span class="s">'glib-2.0'</span><span class="p">)</span>
<span class="n">gobject</span> <span class="o">=</span> <span class="n">dependency</span><span class="p">(</span><span class="s">'gobject-2.0'</span><span class="p">)</span>
<span class="n">gio</span> <span class="o">=</span> <span class="n">dependency</span><span class="p">(</span><span class="s">'gio-2.0'</span><span class="p">)</span>
<span class="n">soup</span> <span class="o">=</span> <span class="n">dependency</span><span class="p">(</span><span class="s">'libsoup-2.4'</span><span class="p">)</span>
<span class="n">vsgi</span> <span class="o">=</span> <span class="n">subproject</span><span class="p">(</span><span class="s">'valum'</span><span class="p">).</span><span class="n">get_variable</span><span class="p">(</span><span class="s">'vsgi'</span><span class="p">)</span>
<span class="n">valum</span> <span class="o">=</span> <span class="n">subproject</span><span class="p">(</span><span class="s">'valum'</span><span class="p">).</span><span class="n">get_variable</span><span class="p">(</span><span class="s">'valum'</span><span class="p">)</span>
<span class="n">executable</span><span class="p">(</span><span class="s">'app'</span><span class="p">,</span> <span class="s">'app.vala'</span><span class="p">,</span>
<span class="n">dependencies</span><span class="p">:</span> <span class="p">[</span><span class="n">glib</span><span class="p">,</span> <span class="n">gobject</span><span class="p">,</span> <span class="n">gio</span><span class="p">,</span> <span class="n">soup</span><span class="p">,</span> <span class="n">vsgi</span><span class="p">,</span> <span class="n">valum</span><span class="p">])</span>
</code></pre></div></div>
<p>Once installed, however, all that is needed is to pass <code class="language-plaintext highlighter-rouge">--pkg=valum-0.3</code> to the
Vala compiler.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vala <span class="nt">--pkg</span><span class="o">=</span>valum-0.3 app.vala
</code></pre></div></div>
<p>In <code class="language-plaintext highlighter-rouge">app.vala</code>,</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">using</span> <span class="nn">Valum</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">VSGI</span><span class="p">;</span>
<span class="k">public</span> <span class="kt">int</span> <span class="nf">main</span> <span class="p">(</span><span class="kt">string</span><span class="p">[]</span> <span class="n">args</span><span class="p">)</span> <span class="p">{</span>
<span class="kt">var</span> <span class="n">app</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">Router</span> <span class="p">();</span>
<span class="n">app</span><span class="p">.</span><span class="k">get</span> <span class="p">(</span><span class="s">"/"</span><span class="p">,</span> <span class="p">(</span><span class="n">req</span><span class="p">,</span> <span class="n">res</span><span class="p">)</span> <span class="p">=></span> <span class="p">{</span>
<span class="k">return</span> <span class="n">res</span><span class="p">.</span><span class="nf">expand_utf8</span> <span class="p">(</span><span class="s">"Hello world!"</span><span class="p">);</span>
<span class="p">});</span>
<span class="k">return</span> <span class="n">Server</span><span class="p">.</span><span class="nf">@new</span> <span class="p">(</span><span class="s">"http"</span><span class="p">,</span> <span class="n">handler</span><span class="p">:</span> <span class="n">app</span><span class="p">)</span>
<span class="p">.</span><span class="nf">run</span> <span class="p">(</span><span class="n">args</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>There’s been a lot of new features and I hope I won’t miss any!</p>
<p>There’s a new <code class="language-plaintext highlighter-rouge">url_for</code> utility in <code class="language-plaintext highlighter-rouge">Router</code> that comes with named route. It
basically allow one to reverse URLs patterns defined with rules and raw paths.</p>
<p>All that is needed is to pass a name to <code class="language-plaintext highlighter-rouge">rule</code>, <code class="language-plaintext highlighter-rouge">path</code> or any method helper.</p>
<p>I discovered the <code class="language-plaintext highlighter-rouge">:</code> notation for named varidic arguments if they alternate
between strings and values. This is typically used to initialize <code class="language-plaintext highlighter-rouge">GLib.Object</code>.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">using</span> <span class="nn">Valum</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">VSGI</span><span class="p">;</span>
<span class="kt">var</span> <span class="n">app</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">Router</span> <span class="p">();</span>
<span class="n">app</span><span class="p">.</span><span class="k">get</span> <span class="p">(</span><span class="s">"/"</span><span class="p">,</span> <span class="p">(</span><span class="n">req</span><span class="p">,</span> <span class="n">res</span><span class="p">)</span> <span class="p">=></span> <span class="p">{</span>
<span class="k">return</span> <span class="s">"<a href=\"%s\">View profile of %s</a>"</span><span class="p">.</span><span class="nf">printf</span> <span class="p">(</span>
<span class="n">app</span><span class="p">.</span><span class="nf">url_for</span> <span class="p">(</span><span class="s">"user"</span><span class="p">,</span> <span class="n">id</span><span class="p">:</span> <span class="s">"5"</span><span class="p">),</span> <span class="s">"John Doe"</span><span class="p">);</span>
<span class="p">});</span>
<span class="n">app</span><span class="p">.</span><span class="k">get</span> <span class="p">(</span><span class="s">"/users/<int:id>"</span><span class="p">,</span> <span class="p">(</span><span class="n">req</span><span class="p">,</span> <span class="n">res</span><span class="p">,</span> <span class="n">next</span><span class="p">,</span> <span class="n">ctx</span><span class="p">)</span> <span class="p">=></span> <span class="p">{</span>
<span class="kt">var</span> <span class="n">id</span> <span class="p">=</span> <span class="n">ctx</span><span class="p">[</span><span class="s">"id"</span><span class="p">].</span><span class="nf">get_string</span> <span class="p">();</span>
<span class="k">return</span> <span class="n">res</span><span class="p">.</span><span class="nf">expand_utf8</span> <span class="p">(</span><span class="s">"Hello %s!"</span><span class="p">.</span><span class="nf">printf</span> <span class="p">(</span><span class="n">id</span><span class="p">));</span>
<span class="p">},</span> <span class="s">"user"</span><span class="p">);</span>
</code></pre></div></div>
<p>In <code class="language-plaintext highlighter-rouge">Router</code>, we also have:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">asterisk</code> to handle <code class="language-plaintext highlighter-rouge">*</code> URI</li>
<li><code class="language-plaintext highlighter-rouge">once</code> for performing initialization</li>
<li><code class="language-plaintext highlighter-rouge">path</code> for a path-based route</li>
<li><code class="language-plaintext highlighter-rouge">rule</code> to replace <code class="language-plaintext highlighter-rouge">method</code></li>
<li><code class="language-plaintext highlighter-rouge">register_type</code> rather than a <code class="language-plaintext highlighter-rouge">GLib.HashTable<string, Regex></code></li>
</ul>
<p>Another significant change is that the previous state stack has been replaced
by a context tree with recursive key resolution. It pretty much maps <code class="language-plaintext highlighter-rouge">string</code>
to <code class="language-plaintext highlighter-rouge">GLib.Value</code> in a non-destructive way.</p>
<p>In terms of new middlewares, you’ll be glad to see all the built-in
functionnalities we now support:</p>
<ul>
<li>authentication with support for the <code class="language-plaintext highlighter-rouge">Basic</code> scheme via <code class="language-plaintext highlighter-rouge">authenticate</code></li>
<li>content negotiation via <code class="language-plaintext highlighter-rouge">negotiate</code>, <code class="language-plaintext highlighter-rouge">accept</code> and more!</li>
<li>static resource delivery from <code class="language-plaintext highlighter-rouge">GLib.File</code> and <code class="language-plaintext highlighter-rouge">GLib.Resource</code> bundles</li>
<li><code class="language-plaintext highlighter-rouge">basic</code> to strip the <code class="language-plaintext highlighter-rouge">Router</code> responsibilities</li>
<li><code class="language-plaintext highlighter-rouge">subdomain</code></li>
<li><code class="language-plaintext highlighter-rouge">basepath</code> to prefix URLs</li>
<li><code class="language-plaintext highlighter-rouge">cache_control</code> to set the <code class="language-plaintext highlighter-rouge">Cache-Control</code> header</li>
<li>branch on raised <code class="language-plaintext highlighter-rouge">status</code> codes</li>
<li>perform work <code class="language-plaintext highlighter-rouge">safely</code> and don’t let any error leak!</li>
<li>stream events with <code class="language-plaintext highlighter-rouge">stream_events</code></li>
</ul>
<p>Now, which one to cover?</p>
<p>The <code class="language-plaintext highlighter-rouge">basepath</code> is my personal favourite, because it allow one to create
prefix-agnostic routers.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">var</span> <span class="n">app</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">Router</span> <span class="p">();</span>
<span class="kt">var</span> <span class="n">api</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">Router</span> <span class="p">();</span>
<span class="c1">// matches '/api/v1/'</span>
<span class="n">api</span><span class="p">.</span><span class="k">get</span> <span class="p">(</span><span class="s">"/"</span><span class="p">,</span> <span class="p">(</span><span class="n">req</span><span class="p">,</span> <span class="n">res</span><span class="p">)</span> <span class="p">=></span> <span class="p">{</span>
<span class="k">return</span> <span class="n">res</span><span class="p">.</span><span class="nf">expand_utf8</span> <span class="p">(</span><span class="s">"Hello world!"</span><span class="p">);</span>
<span class="p">});</span>
<span class="n">app</span><span class="p">.</span><span class="nf">use</span> <span class="p">(</span><span class="nf">basepath</span> <span class="p">(</span><span class="s">"/api/v1"</span><span class="p">,</span> <span class="n">api</span><span class="p">.</span><span class="n">handle</span><span class="p">));</span>
</code></pre></div></div>
<p>The only missing feature is to retranslate URLs directly from the body. I think
we could use some <code class="language-plaintext highlighter-rouge">GLib.Converter</code> here.</p>
<p>The <code class="language-plaintext highlighter-rouge">negotiate</code> middleware and it’s derivatives are really handy for declaring
the available representations of a resource.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">app</span><span class="p">.</span><span class="k">get</span> <span class="p">(</span><span class="s">"/"</span><span class="p">,</span> <span class="nf">accept</span> <span class="p">(</span><span class="s">"text/html; text/plain"</span><span class="p">,</span> <span class="p">(</span><span class="n">req</span><span class="p">,</span> <span class="n">res</span><span class="p">,</span> <span class="n">next</span><span class="p">,</span> <span class="n">ctx</span><span class="p">,</span> <span class="n">ct</span><span class="p">)</span> <span class="p">=></span> <span class="p">{</span>
<span class="k">switch</span> <span class="p">(</span><span class="n">ct</span><span class="p">)</span> <span class="p">{</span>
<span class="k">case</span> <span class="s">"text/html"</span><span class="p">:</span>
<span class="k">return</span> <span class="n">res</span><span class="p">.</span><span class="nf">expand_utf8</span> <span class="p">(</span><span class="s">""</span><span class="p">);</span>
<span class="k">case</span> <span class="s">"text/plain"</span><span class="p">:</span>
<span class="k">return</span> <span class="s">"Hello world!"</span><span class="p">;</span>
<span class="k">default</span><span class="p">:</span>
<span class="nf">assert_not_reached</span> <span class="p">();</span>
<span class="p">}</span>
<span class="p">}))</span>
</code></pre></div></div>
<p>There’s a lot of stuff happening in each of them so refer to the docs!</p>
<p>Quick review into <code class="language-plaintext highlighter-rouge">Request</code> and <code class="language-plaintext highlighter-rouge">Response</code>, we now have the following helpers:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">lookup_query</code> to fetch a query item and deal with its <code class="language-plaintext highlighter-rouge">null</code> case</li>
<li><code class="language-plaintext highlighter-rouge">lookup_cookie</code> and <code class="language-plaintext highlighter-rouge">lookup_signed_cookie</code> to fetch a cookie</li>
<li><code class="language-plaintext highlighter-rouge">cookies</code> to get cookies from a request and response</li>
<li><code class="language-plaintext highlighter-rouge">convert</code> to apply a <code class="language-plaintext highlighter-rouge">GLib.Converter</code></li>
<li><code class="language-plaintext highlighter-rouge">append</code> to append a chunk into the response body</li>
<li><code class="language-plaintext highlighter-rouge">expand</code> to write a buffer into the response body</li>
<li><code class="language-plaintext highlighter-rouge">expand_stream</code> to pipe a stream</li>
<li><code class="language-plaintext highlighter-rouge">expand_file</code> to pipe a file</li>
<li><code class="language-plaintext highlighter-rouge">end</code> to end a response properly</li>
<li><code class="language-plaintext highlighter-rouge">tee</code> to tee the response body into an additional stream</li>
</ul>
<p>All the utilities to write the body come in <code class="language-plaintext highlighter-rouge">_bytes</code> and <code class="language-plaintext highlighter-rouge">_utf8</code> variants. The
latter properly set the content charset when appliable.</p>
<p>Back into <code class="language-plaintext highlighter-rouge">Server</code>, implementation have been modularized with <code class="language-plaintext highlighter-rouge">GLib.Module</code> and
are now dynamically loaded. What used to be a <code class="language-plaintext highlighter-rouge">VSGI.<server></code> namespace now has
become simply <code class="language-plaintext highlighter-rouge">Server.new ("<name>")</code>. Implementations are installed in
<code class="language-plaintext highlighter-rouge">${prefix}/${libdir}/vsgi-0.3/servers</code>, which can be overwritten by the
<code class="language-plaintext highlighter-rouge">VSGI_SERVER_PATH</code> environment variable.</p>
<p>The VSGI specification is not yet <code class="language-plaintext highlighter-rouge">1.0</code>, so please, don’t write a custom server
for now or if you do so, please submit it for inclusion. There’s some
work-in-progress for <a href="https://lwan.ws/">Lwan</a> and <a href="http://tomcat.apache.org/tomcat-3.3-doc/AJPv13.html">AJP</a> as I speak if you have some time to spend.</p>
<p>Options have been moved into <code class="language-plaintext highlighter-rouge">GLib.Object</code> properties and a new <code class="language-plaintext highlighter-rouge">listen</code> API
based on <code class="language-plaintext highlighter-rouge">GLib.SocketAddress</code> makes it more convenient than ever.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">using</span> <span class="nn">VSGI</span><span class="p">;</span>
<span class="kt">var</span> <span class="n">tls_cert</span> <span class="p">=</span> <span class="k">new</span> <span class="n">TlsCertificate</span><span class="p">.</span><span class="nf">from_files</span> <span class="p">(</span><span class="s">"localhost.cert"</span><span class="p">,</span>
<span class="s">"localhost.key"</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">http_server</span> <span class="p">=</span> <span class="n">Server</span><span class="p">.</span><span class="k">new</span> <span class="p">(</span><span class="s">"http"</span><span class="p">,</span> <span class="n">https</span><span class="p">:</span> <span class="k">true</span><span class="p">,</span>
<span class="n">tls_certificate</span><span class="p">:</span> <span class="n">tls_cert</span><span class="p">);</span>
<span class="n">http_server</span><span class="p">.</span><span class="nf">set_application_callback</span> <span class="p">((</span><span class="n">req</span><span class="p">,</span> <span class="n">res</span><span class="p">)</span> <span class="p">=></span> <span class="p">{</span>
<span class="k">return</span> <span class="n">res</span><span class="p">.</span><span class="nf">expand_utf8</span> <span class="p">(</span><span class="s">"Hello world!"</span><span class="p">);</span>
<span class="p">});</span>
<span class="n">http_server</span><span class="p">.</span><span class="nf">listen</span> <span class="p">(</span><span class="k">new</span> <span class="nf">InetSocketAddress</span> <span class="p">(</span><span class="k">new</span> <span class="n">InetAddress</span><span class="p">.</span><span class="nf">loopback</span> <span class="p">(</span><span class="n">SocketFamily</span><span class="p">.</span><span class="n">IPV4</span><span class="p">),</span> <span class="m">3003</span><span class="p">));</span>
<span class="k">new</span> <span class="nf">MainLoop</span> <span class="p">().</span><span class="nf">run</span> <span class="p">();</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">GLib.Application</code> code has been extracted into the new <code class="language-plaintext highlighter-rouge">VSGI.Application</code>
cushion used when calling <code class="language-plaintext highlighter-rouge">run</code>. It parses the CLI, set the logger and handle
<code class="language-plaintext highlighter-rouge">SIGTERM</code> into a graceful shutdown.</p>
<p>Server can also <code class="language-plaintext highlighter-rouge">fork</code> to scale on multicore architectures. I’ve backtracked on
the <code class="language-plaintext highlighter-rouge">Worker</code> class to deal with IPC communication, but if anyone is interested
into building a nice clustering system, I would be glad to look into it.</p>
<p>That wraps it up, the rest can be discovered in the <a href="http://docs.valum-framework.org/en/latest/">updated docs</a>. The API
docs should be available shortly via <a href="http://valadoc.org/valum-0.3/index.htm">valadoc.org</a>.</p>
<p>I manage to cover this exhaustively with <a href="https://sourceware.org/libabigail/manual/abidiff.html">abidiff</a>, a really nice tool to
diff two ELF files.</p>
<h2 id="long-term-notes">Long-term notes</h2>
<p>Here’s some long-term notes for things I couldn’t put into this release or that
I plan at a much longer term.</p>
<ul>
<li>multipart streams</li>
<li>digest authentication</li>
<li>async delegates</li>
<li>epoll and kqueue with <a href="https://mail.gnome.org/archives/commits-list/2014-February/msg06024.html">wip/pollcore</a></li>
<li>schedule future release with the GNOME project</li>
<li>GIR introspection and typelibs for <a href="https://wiki.gnome.org/Projects/PyGObject">PyGObject</a> and <a href="https://wiki.gnome.org/Projects/Gjs">Gjs</a></li>
</ul>
<p>The GIR and typelibs stuff might not be suitable for Valum, but VSGI could have
a bright future with Python or JavaScript bindings.</p>
<p>Coming releases will be much less time-consuming as there’s been a big step to
make to have something actually usable. Maybe every 6 months or so.</p>Guillaume Poirier-Morency <guillaumepoiriermorency@gmail.com>The first release candidate for Valum 0.3 has been launched today!What is Meson?2016-09-18T00:00:00+00:002016-09-18T00:00:00+00:00https://arteymix.github.io/2016/09/18/what-is-meson<p>I have discovered <a href="http://mesonbuild.com/">Meson</a> a couple of years back and since then use it for most
of my projects written in Vala. This post is an attempt at describing the good,
bad and ugly of the build system.</p>
<p>So, what is Meson?</p>
<ul>
<li>a build system</li>
<li>portable (see Python portability)</li>
<li>a Ninja generator</li>
<li>use case oriented</li>
<li>fast</li>
<li>opiniated</li>
</ul>
<p>What it’s not?</p>
<ul>
<li>a general purpose build system</li>
<li>a Turing-complete language</li>
<li>extensible (only in Python)</li>
</ul>
<p>It handle 80% of the cases nicely and elegantly.</p>
<p>Since it is use case oriented, features are introduced on need. It keeps
a tight balance between conciseness, generality and features.</p>
<p>It mixes configure and build step so that the build essentially become one big
tree. Then, the build system determine what goes into the configuration and what
goes into the build.</p>
<p>The cognitive load is very low, which means it’s very easy to learn the basics
and make actual usage of it. This is critical, because all the time spent on
setting the build hardly contribute to the project goal.</p>
<p>The following is a basic build that check for dependencies (using
<a href="https://www.freedesktop.org/wiki/Software/pkg-config/">pkg-config</a>) and build an executable:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">project</span><span class="p">(</span><span class="s">'Meson Example'</span><span class="p">,</span> <span class="s">'c'</span><span class="p">,</span> <span class="s">'vala'</span><span class="p">)</span>
<span class="n">glib</span> <span class="o">=</span> <span class="n">dependency</span><span class="p">(</span><span class="s">'glib-2.0'</span><span class="p">)</span>
<span class="n">gobject</span> <span class="o">=</span> <span class="n">dependency</span><span class="p">(</span><span class="s">'gobject-2.0'</span><span class="p">)</span>
<span class="n">executable</span><span class="p">(</span><span class="s">'app'</span><span class="p">,</span> <span class="s">'app.vala'</span><span class="p">,</span> <span class="n">dependencies</span><span class="p">:</span> <span class="p">[</span><span class="n">glib</span><span class="p">,</span> <span class="n">gobject</span><span class="p">])</span>
</code></pre></div></div>
<p>Building becomes a piece of cake:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir </span>build <span class="o">&&</span> <span class="nb">cd </span>build
meson ..
ninja
</code></pre></div></div>
<p>Only a few keywords are sufficient for most builds:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">executable</code></li>
<li><code class="language-plaintext highlighter-rouge">library</code> with <code class="language-plaintext highlighter-rouge">shared_library</code> and <code class="language-plaintext highlighter-rouge">static_library</code></li>
<li><code class="language-plaintext highlighter-rouge">dependency</code></li>
<li><code class="language-plaintext highlighter-rouge">declare_dependency</code></li>
</ul>
<p>Built-in benchmarks and tests, just pass the executable to either <code class="language-plaintext highlighter-rouge">benchmark</code>
or <code class="language-plaintext highlighter-rouge">test</code>.</p>
<p>The main downside is that if what you want to do is not <em>supported</em>, you either
have to hack things or wait until the feature gets into the build system.</p>
<p>The system is very opiniated. It’s both a good and bad thing. Good since you
don’t need to write a lot to get most jobs done. Bad because you might hit
a wall eventually.</p>
<p>There’s also the Python question. It requires at least 3.4. This is becoming
less an problematic as old distributions progressively die out, but still can
prevent you now. Here’s a few ideas to remedy this problem:</p>
<ul>
<li>build a dependency-free zipball (see <a href="https://github.com/mesonbuild/meson/issues/588">issue #588</a>)</li>
<li>backport Meson to older Python version</li>
</ul>
<p>Meson is getting better over time and so far has managed to become the best
build system for Vala. This is why I highly recommend it.</p>Guillaume Poirier-Morency <guillaumepoiriermorency@gmail.com>I have discovered Meson a couple of years back and since then use it for most of my projects written in Vala. This post is an attempt at describing the good, bad and ugly of the build system.Merged GModule branch!2016-06-05T00:00:00+00:002016-06-05T00:00:00+00:00https://arteymix.github.io/valum/2016/06/05/merged-gmodule-branch<p>Valum now support dynamically loadable server implementation with <a href="https://developer.gnome.org/glib/stable/glib-Dynamic-Loading-of-Modules.html">GModule</a>!</p>
<p>Server are typically looked up in <code class="language-plaintext highlighter-rouge">/usr/lib64/vsgi/servers</code> with the
<code class="language-plaintext highlighter-rouge">libvsgi-<name>.so</code> pattern (although this is highly system-dependent).</p>
<p>This works by setting the <code class="language-plaintext highlighter-rouge">RPATH</code> to <code class="language-plaintext highlighter-rouge">$ORIGIN/vsgi/servers</code> of the VSGI shared
library so that it looks into that folder first.</p>
<p>The <code class="language-plaintext highlighter-rouge">VSGI_SERVER_PATH</code> environment variable can be set as well to explicitly
provide a directory containing implementations.</p>
<p>To implement a compliant VSGI server, all you need is a <code class="language-plaintext highlighter-rouge">server_init</code> symbol
which complies with <code class="language-plaintext highlighter-rouge">ServerInitFunc</code> delegate like the following:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="n">ModuleInit</span><span class="p">]</span>
<span class="k">public</span> <span class="n">Type</span> <span class="nf">server_init</span> <span class="p">(</span><span class="n">TypeModule</span> <span class="n">type_module</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">typeof</span> <span class="p">(</span><span class="n">VSGI</span><span class="p">.</span><span class="n">Custom</span><span class="p">.</span><span class="n">Server</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">VSGI</span><span class="p">.</span><span class="n">Custom</span><span class="p">.</span><span class="n">Server</span> <span class="p">:</span> <span class="n">VSGI</span><span class="p">.</span><span class="n">Server</span> <span class="p">{</span>
<span class="c1">// ...</span>
<span class="p">}</span>
</code></pre></div></div>
<p>It has to return a type that is derived from <code class="language-plaintext highlighter-rouge">VSGI.Server</code> and instantiable
with <code class="language-plaintext highlighter-rouge">GLib.Object.new</code>. The Vala compiler will automatically generate the code
to register class and interfaces into the <code class="language-plaintext highlighter-rouge">type_module</code> parameter.</p>
<p>Some code from CGI has been moved into VSGI to provide uniform handling of its
environment variables. If the protocol you want complies with that, just
subclass (or directly use) <code class="language-plaintext highlighter-rouge">VSGI.CGI.Request</code> and it will perform all the
required initialization.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">VSGI</span><span class="p">.</span><span class="n">Custom</span><span class="p">.</span><span class="n">Request</span> <span class="p">:</span> <span class="n">VSGI</span><span class="p">.</span><span class="n">CGI</span><span class="p">.</span><span class="n">Request</span> <span class="p">{</span>
<span class="k">public</span> <span class="nf">Request</span> <span class="p">(</span><span class="n">IOStream</span> <span class="n">connection</span><span class="p">,</span> <span class="kt">string</span><span class="p">[]</span> <span class="n">environment</span><span class="p">)</span> <span class="p">{</span>
<span class="k">base</span> <span class="p">(</span><span class="n">connection</span><span class="p">,</span> <span class="n">environment</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>For more flexibility, servers can be loaded with <code class="language-plaintext highlighter-rouge">ServerModule</code> directly,
allowing one to specify an explicit lookup directory and control when the
module should be loaded or unloaded.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">var</span> <span class="n">cgi_module</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">ServerModule</span> <span class="p">(</span><span class="k">null</span><span class="p">,</span> <span class="s">"cgi"</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(!</span><span class="n">cgi_module</span><span class="p">.</span><span class="nf">load</span> <span class="p">())</span> <span class="p">{</span>
<span class="nf">assert_not_reached</span> <span class="p">();</span>
<span class="p">}</span>
<span class="kt">var</span> <span class="n">server</span> <span class="p">=</span> <span class="n">Object</span><span class="p">.</span><span class="k">new</span> <span class="p">(</span><span class="n">cgi_module</span><span class="p">.</span><span class="n">server_type</span><span class="p">);</span>
</code></pre></div></div>
<p>I received very useful support from <a href="https://github.com/nirbheek">Nirbheek Chauhan</a> and <a href="https://github.com/tp-m">Tim-Philipp
Müller</a> for setting the <a href="https://github.com/mesonbuild/meson/issues/580">necessary build configuration</a> for that feature.</p>Guillaume Poirier-Morency <guillaumepoiriermorency@gmail.com>Valum now support dynamically loadable server implementation with GModule!Content Negotiation2016-05-14T00:00:00+00:002016-05-14T00:00:00+00:00https://arteymix.github.io/valum/2016/05/14/content-negotiation<p>I recently finished and merged <a href="https://github.com/valum-framework/valum/pull/150">support for content negotiation</a>.</p>
<p>The implementation is really simple: one provide a header, a string describing
expecations and a callback invoked with the negotiated representation. If no
expectation is met, a <code class="language-plaintext highlighter-rouge">406 Not Acceptable</code> is raised.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">app</span><span class="p">.</span><span class="k">get</span> <span class="p">(</span><span class="s">"/"</span><span class="p">,</span> <span class="nf">negotiate</span> <span class="p">(</span><span class="s">"Accept"</span><span class="p">,</span> <span class="s">"text/xml; application/json"</span><span class="p">,</span>
<span class="p">(</span><span class="n">req</span><span class="p">,</span> <span class="n">res</span><span class="p">,</span> <span class="n">next</span><span class="p">,</span> <span class="n">ctx</span><span class="p">,</span> <span class="n">content_type</span><span class="p">)</span> <span class="p">=></span> <span class="p">{</span>
<span class="c1">// produce according to 'content_type'</span>
<span class="p">}));</span>
</code></pre></div></div>
<p>Content negotiation is a nice feature of the HTTP protocol allowing a client
and a server to negotiate the representation (eg. content type, language,
encoding) of a resource.</p>
<p>One very nice part allows the user agent to state a preference and the server
to express quality for a given representation. This is done by specifying the
<code class="language-plaintext highlighter-rouge">q</code> parameter and the negotiation process attempt to maximize the product of
both values.</p>
<p>The following example express that the XML version is poor quality, which is
typically the case when it’s not the source document. JSON would be favoured –
implicitly <code class="language-plaintext highlighter-rouge">q=1</code> – if the client does not state any particular preference.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">accept</span> <span class="p">(</span><span class="s">"text/xml; q=0.1, application/json"</span><span class="p">,</span> <span class="p">()</span> <span class="p">=></span> <span class="p">{</span>
<span class="p">});</span>
</code></pre></div></div>
<p>Mounted as a top-level middleware, it provide a nice way of setting
a <code class="language-plaintext highlighter-rouge">Content-Type: text/html; charset=UTF-8</code> header and filter out
non-compliant clients.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">using</span> <span class="nn">Tmpl</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">Valum</span><span class="p">;</span>
<span class="kt">var</span> <span class="n">app</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">Router</span> <span class="p">();</span>
<span class="n">app</span><span class="p">.</span><span class="nf">use</span> <span class="p">(</span><span class="nf">accept</span> <span class="p">(</span><span class="s">"text/html"</span><span class="p">,</span> <span class="p">()</span> <span class="p">=></span> <span class="p">{</span>
<span class="k">return</span> <span class="nf">next</span> <span class="p">();</span>
<span class="p">}));</span>
<span class="n">app</span><span class="p">.</span><span class="nf">use</span> <span class="p">(</span><span class="nf">accept_charset</span> <span class="p">(</span><span class="s">"UTF-8"</span><span class="p">,</span> <span class="p">()</span> <span class="p">=></span> <span class="p">{</span>
<span class="k">return</span> <span class="nf">next</span> <span class="p">();</span>
<span class="p">}));</span>
<span class="kt">var</span> <span class="n">home</span> <span class="p">=</span> <span class="k">new</span> <span class="n">Template</span><span class="p">.</span><span class="nf">from_path</span> <span class="p">(</span><span class="s">"templates/home.html"</span><span class="p">);</span>
<span class="n">app</span><span class="p">.</span><span class="k">get</span> <span class="p">(</span><span class="s">"/"</span><span class="p">,</span> <span class="p">(</span><span class="n">req</span><span class="p">,</span> <span class="n">res</span><span class="p">)</span> <span class="p">=></span> <span class="p">{</span>
<span class="n">home</span><span class="p">.</span><span class="nf">expand</span> <span class="p">(</span><span class="n">res</span><span class="p">.</span><span class="n">body</span><span class="p">,</span> <span class="k">null</span><span class="p">);</span>
<span class="p">});</span>
</code></pre></div></div>
<p>This is another step forward a <code class="language-plaintext highlighter-rouge">0.3</code> release!</p>Guillaume Poirier-Morency <guillaumepoiriermorency@gmail.com>I recently finished and merged support for content negotiation.Fork!2016-04-15T00:00:00+00:002016-04-15T00:00:00+00:00https://arteymix.github.io/valum/2016/04/15/fork<p>Ever heard of <code class="language-plaintext highlighter-rouge">fork</code>?</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">using</span> <span class="nn">GLib</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">VSGI.HTTP</span><span class="p">;</span>
<span class="kt">var</span> <span class="n">server</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">Server</span> <span class="p">(</span><span class="s">""</span><span class="p">,</span> <span class="p">()</span> <span class="p">=></span> <span class="p">{</span>
<span class="k">return</span> <span class="n">res</span><span class="p">.</span><span class="nf">expand_utf8</span> <span class="p">(</span><span class="s">"Hello world!"</span><span class="p">);</span>
<span class="p">});</span>
<span class="n">server</span><span class="p">.</span><span class="nf">listen</span> <span class="p">(</span><span class="k">new</span> <span class="nf">VariantDict</span> <span class="p">().</span><span class="nf">end</span> <span class="p">());</span>
<span class="n">server</span><span class="p">.</span><span class="nf">fork</span> <span class="p">();</span>
<span class="k">new</span> <span class="nf">MainLoop</span> <span class="p">().</span><span class="nf">run</span> <span class="p">();</span>
</code></pre></div></div>
<p><em>Yeah, there’s a new API for listening and forking with custom options…</em></p>
<p>The <code class="language-plaintext highlighter-rouge">fork</code> system call will actually copy the whole process into a new process,
running the exact same program.</p>
<p>Although memory is not shared, file descriptors are, so you can have workers
listening on common interfaces.</p>
<p>I notably tested the whole thing on our cluster at IRIC. It’s a 64 cores Xeon
Core i7 setup.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wrk <span class="nt">-c</span> 1024 <span class="nt">-t</span> 32 http://0.0.0.0:3003/hello
</code></pre></div></div>
<p>With a single worker:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Running 10s test @ http://0.0.0.0:3003/hello
32 threads and 1024 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 54.35ms 95.96ms 1.93s 98.78%
Req/Sec 165.81 228.28 2.04k 86.08%
41741 requests in 10.10s, 5.89MB read
Socket errors: connect 35, read 0, write 0, timeout 13
Requests/sec: 4132.53
Transfer/sec: 597.28KB
</code></pre></div></div>
<p>With 63 forks (64 workers):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Running 10s test @ http://0.0.0.0:3003/hello
32 threads and 1024 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 60.83ms 210.70ms 2.00s 93.58%
Req/Sec 2.99k 797.97 7.44k 70.33%
956577 requests in 10.10s, 135.02MB read
Socket errors: connect 35, read 0, write 0, timeout 17
Requests/sec: 94720.20
Transfer/sec: 13.37MB
</code></pre></div></div>
<p>It’s about 1500 req/sec per worker and an speedup of a factor of 23. The
latency is almost not affected.</p>Guillaume Poirier-Morency <guillaumepoiriermorency@gmail.com>Ever heard of fork?Work on Memcached-GLib2016-04-05T00:00:00+00:002016-04-05T00:00:00+00:00https://arteymix.github.io/2016/04/05/work-on-libmemcached-glib<p>The past few days, I’ve been working on a really nice <a href="http://libmemcached.org/libMemcached.html">libmemcached</a>
GLib wrapper.</p>
<ul>
<li>main loop integration</li>
<li>fully asynchronous API</li>
<li>error handling</li>
</ul>
<p>The whole code is available under the LGPLv3 from <a href="https://github.com/arteymix/libmemcached-glib/tree/master/benchmarks">arteymix/libmemcached-glib</a>.</p>
<p>It should reach 1.0 very quickly, only a few features are missing:</p>
<ul>
<li>a couple of function wrappers</li>
<li>integration for libmemcachedutil</li>
<li>async I/O improvements</li>
</ul>
<p>Once released, it might be interesting to build a GTK UI for Memcached upon
that work. Meanwhile, it will be a very useful tool to build fast web
applications with <a href="valum">Valum</a>.</p>Guillaume Poirier-Morency <guillaumepoiriermorency@gmail.com>The past few days, I’ve been working on a really nice libmemcached GLib wrapper.Proposal for asynchronous delegates in Vala2016-03-30T00:00:00+00:002016-03-30T00:00:00+00:00https://arteymix.github.io/2016/03/30/async-delegates<p>This post describe a feature I will attempt to implement this summer.</p>
<p>The declaration of async delegate is simply extending a traditional delegate
with the <code class="language-plaintext highlighter-rouge">async</code> trait.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">async</span> <span class="k">delegate</span> <span class="k">void</span> <span class="nf">AsyncDelegate</span> <span class="p">(</span><span class="n">GLib</span><span class="p">.</span><span class="n">OutputStream</span> <span class="n">@out</span><span class="p">);</span>
</code></pre></div></div>
<p>The syntax of callback is the same. It’s not necessary to add anything since
the async trait is infered from the type of the variable holding it.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">AsyncDelegate</span> <span class="n">d</span> <span class="p">=</span> <span class="p">(</span><span class="n">@out</span><span class="p">)</span> <span class="p">=></span> <span class="p">{</span>
<span class="k">yield</span> <span class="n">@out</span><span class="p">.</span><span class="nf">write_all_async</span> <span class="p">(</span><span class="s">"Hello world!"</span><span class="p">.</span><span class="n">data</span><span class="p">,</span> <span class="k">null</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Just like regular callback, asynchronous callbacks are first-class citizen.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">async</span> <span class="k">void</span> <span class="nf">test_async</span> <span class="p">(</span><span class="n">AsyncDelegate</span> <span class="n">callback</span><span class="p">,</span>
<span class="n">OutputStream</span> <span class="n">@out</span><span class="p">)</span> <span class="p">{</span>
<span class="k">yield</span> <span class="nf">callback</span> <span class="p">(</span><span class="n">@out</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>It’s also possible to pass an asynchronous function which is type-compatible
with the delegate signature:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">async</span> <span class="k">void</span> <span class="nf">hello_world_async</span> <span class="p">(</span><span class="n">OutputStream</span> <span class="n">@out</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">yield</span> <span class="n">@out</span><span class="p">.</span><span class="nf">write_all_async</span> <span class="p">(</span><span class="s">"Hello world!"</span><span class="p">.</span><span class="n">data</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">yield</span> <span class="nf">test_async</span> <span class="p">(</span><span class="n">hello_world_async</span><span class="p">,</span> <span class="n">@out</span><span class="p">);</span>
</code></pre></div></div>
<h2 id="chaining">Chaining</h2>
<p>I still need to figure out how to handle chaining for async lambda. Here’s
a few ideas:</p>
<ul>
<li>refer to the callback using <code class="language-plaintext highlighter-rouge">this</code> (weird..)</li>
<li>introduce a <code class="language-plaintext highlighter-rouge">callback</code> keyword</li>
</ul>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">AsyncDelegate</span> <span class="n">d</span> <span class="p">=</span> <span class="p">(</span><span class="n">@out</span><span class="p">)</span> <span class="p">=></span> <span class="p">{</span>
<span class="n">Idle</span><span class="p">.</span><span class="k">add</span> <span class="p">(</span><span class="k">this</span><span class="p">.</span><span class="n">callback</span><span class="p">);</span>
<span class="k">yield</span><span class="p">;</span>
<span class="p">};</span>
<span class="n">AsyncDelegate</span> <span class="n">d</span> <span class="p">=</span> <span class="p">(</span><span class="n">@out</span><span class="p">)</span> <span class="p">=></span> <span class="p">{</span>
<span class="n">Idle</span><span class="p">.</span><span class="k">add</span> <span class="p">(</span><span class="n">callback</span><span class="p">);</span>
<span class="k">yield</span><span class="p">;</span>
<span class="p">};</span>
</code></pre></div></div>
<h2 id="how-it-would-end-up-for-valum">How it would end-up for Valum</h2>
<p>Most of the framework could be revamped with the <code class="language-plaintext highlighter-rouge">async</code> trait in
<code class="language-plaintext highlighter-rouge">ApplicationCallback</code>, <code class="language-plaintext highlighter-rouge">HandlerCallback</code> and <code class="language-plaintext highlighter-rouge">NextCallback</code>.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">app</span><span class="p">.</span><span class="nf">@get</span> <span class="p">(</span><span class="s">"/me"</span><span class="p">,</span> <span class="p">(</span><span class="n">req</span><span class="p">,</span> <span class="n">res</span><span class="p">,</span> <span class="n">next</span><span class="p">)</span> <span class="p">=></span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">req</span><span class="p">.</span><span class="nf">lookup_signed_cookies</span> <span class="p">(</span><span class="s">"session"</span><span class="p">)</span> <span class="p">==</span> <span class="k">null</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">yield</span> <span class="nf">next</span> <span class="p">(</span><span class="n">req</span><span class="p">,</span> <span class="n">res</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="k">yield</span> <span class="n">res</span><span class="p">.</span><span class="nf">extend_utf8_async</span> <span class="p">(</span><span class="s">"Hello world!"</span><span class="p">.</span><span class="n">data</span><span class="p">);</span>
<span class="p">});</span>
</code></pre></div></div>
<p>The semantic for the return value would simply state if the request has been
handled instead of being <em>eventually</em> handled.</p>Guillaume Poirier-Morency <guillaumepoiriermorency@gmail.com>This post describe a feature I will attempt to implement this summer.