Umberto Raimondi's Blog

Interesting projects on GitHub/April

2012-05-01T00:00:00+02:00

A few noteworthy open source projects i started following on GitHub in april:

Meteor, uncovered last month and discussed profusely everywhere, is now covered by an MIT license. Looks still a bit scary for the amount of communication protocol details it hides from the developer and for the complete lack of access control on requests performed from the web client to the application DB. Also, it seems that only MongoDB is supported at the moment. link
Heist, small scheme interpreter written in Ruby. link
Repository for testingbenfordslaw.com, nice project about the Benford's Law, not been updated in a while. link
Slick Grid, as the author says "a lightning fast javascript grid/spreadsheet", we'll see if one day i'll find a use for this. link

Interesting projects on GitHub/March

2012-04-01T00:00:00+02:00

A few noteworthy open source projects i started following on GitHub in march:

Misaki, a Jekyll inspired static site generator built in clojure link
JQuery Scroll Path, a plugin to define custom scroll paths, check the demo link
Mechanize, nice ruby web client library to automate interaction with websites link
Induction, generic database client for Mac OS X link

Interesting projects on GitHub/February

2012-03-01T00:00:00+01:00

A few noteworthy open source projects i started following on GitHub in february:

Small footprint C JSON Parser link
Luvit, and interesting lua node.js "clone" link
SocketRocket, an Objective-C websocket client library from Square link
Tinycon, a js library to manipulate favicons (see compatibility notes) link
Filtrr, a javascript image filter library link
Resty, small Java JSON client library link

From VIM Theme to Pygments CSS

2011-09-21T00:00:00+02:00

I have just converted a few VIM themes to CSS that can be used with Pygments using the python script described here (GitHub repo).

If anyone is interested (you'll need this if you are using Jekyll), the GitHub repository with the CSS files can be found here and at the moment it contains the following VIM themes: desert, mustang, no_quarter, peaksea, railscasts, rdark, slate, wombat, nuvola.

Feel free to fork the repository and add your own converted themes, an example of the result using the rdark theme can be seen below, enjoy!

                                                            
public static class IntTimes
              implements ScalaObject
{

    public void times(Function1 what)
    {
        Predef$.MODULE$.intWrapper(1).to(n).foreach$mVc$sp(what);
    }
    private final int n;

    public IntTimes(int n)
    {
        this.n = n;
        super();
    }
}

First steps in Scala: Implementing Ruby's Integer.times

2011-09-21T00:00:00+02:00

After reading Seven Languages in Seven Weeks:A Pragmatic Guide to Learning Programming Languages i was left with the desire to know more about a couple of languages, namely Scala and Clojure. While i had some experiences with functional programming (Standard ML, Scheme) and other programming paradigms (Prolog) during the university, now some years have passed.

I strongly suggest you to buy the book even if you already know some of the described languages (Ruby, Io, Prolog, Scala, Erlang, Clojure, Haskell), it only gives a gist of these seven languages in a set of lessons ideally completed in three days (that with some individual unguided experimentation will fill a full week) but with its style will make you hungry for more.

If you need a suggestion on a Scala book, at the moment i'm going through Odersky's Programming in Scala and up until now it seems a good book even for those without prior experience in functional(ish) languages. Also, unlike other books, the first chapters are free of enigmatic examples that will only confuse beginners.

I'm practicing Scala solving some problems at Project Euler and for one of those i decided to implement something that resembled Ruby's .times and check the produced java bytecode to get a glimpse of scala's inner workings.

object scalatimes extends App{
  implicit def int2inttimes(n:Int)=new IntTimes(n)
  
  class IntTimes(n:Int){ 
       def times(what:Int=>Unit){
           (1 to n).foreach(what);
       }
  }

  1000.times { n =>
      val n2=n+1;
      println(n2);
  }

}

The implementation of times is quite trivial, this code makes use of implicit conversions to add to the Int type the new method through the implicit conversion of Int objects to the new IntTimes in the scope of the scalatimes object.

The times function simply uses the given function object of type Int=>Unit within a foreach cycle, the current index will be accessible inside the function body (i suppose that there are better ways to do this). But what kind of java code does the scala compiler produces for this example? A quick check shows that six class files are built:

scalatimes
scalatimes$
scalatimes$$anonfun$1
scalatimes$delayedInit$body
scalatimes$Int2
scalatimes$IntTimes

These files follow the usual java .class naming conventions,so the scala compiler produces a scalatimes class with a few inner classes. The class files can be decompiled with Jad but the decompiled result will be only partially complete. But anyway, this will help to understand what is happeing behind the scene. As expected, the body of IntTimes is contained in an inner class with the same name:

                                                            
public static class IntTimes
              implements ScalaObject
{

    public void times(Function1 what)
    {
        Predef$.MODULE$.intWrapper(1).to(n).foreach$mVc$sp(what);
    }

    private final int n;

    public IntTimes(int n)
    {
        this.n = n;
        super();
    }
}

The times method expects a Function1 object as parameter, the body of the function passed to 1000.times() as parameter has been wrapped in an anonymous inner class (as per Jad comment):

                                                            
/* anonymous class */ 

public static final class anonfun.cls1 extends scala.runtime.AbstractFunction1.mcVI.sp
                    implements Serializable
{

    public final void apply(int n)
    {
        apply$mcVI$sp(n);
    }

    public void apply$mcVI$sp(int v1)
    {
        int n2 = v1 + 1;
        Predef$.MODULE$.println(BoxesRunTime.boxToInteger(n2));
    }

    public final volatile Object apply(Object v1)
    {
        apply(BoxesRunTime.unboxToInt(v1));
        return BoxedUnit.UNIT;
    }

    public static final long serialVersionUID;

    static 
    {
        0L;
        serialVersionUID = 0L;
    }

}

The two lines of code that compose the function object body passed to 1000.times() are clearly visible in apply$mcVI$sp(int v1). The main method of the App can be identified in the body of the apply() method of a static class that extends AbstractFunction0:

public static final class on0 extends AbstractFunction0
                    implements ScalaObject{

    public final Object apply()
    {
        $outer.t1_$eq(System.currentTimeMillis());
        $outer.int2inttimes(1000).outer(new ());
        Predef$.MODULE$.println((new StringBuilder()).append("Time:").append(BoxesRunTime.boxToLong(System.currentTimeMillis() - $outer.t1())).toString());
        return BoxedUnit.UNIT;
    }

    private final scalatimes$ $outer;

    public on1.mcVI.sp(scalatimes$ $outer)
    {
        if($outer == null){
            throw new NullPointerException();
        } else {
            this.$outer = $outer;
            super();
            return;
        }
    }
}

Function0, like the Fuction1 above, is a scala framework object, likely related to the class extended above, and some helper methods in scalatimes hints to the fact that it will be loaded using a delayedInit() call (by an App loader?). And this ends for now my brief and superficial investigation on the structure of the code produced by the Scala compiler.

SSH Productivity Tips: Multiplexed connection and more

2011-08-19T00:00:00+02:00

With tools like SSH, Git and RSync there are always new tips you don't know about or some clever scripts that can increase your productivity or simply save you a few keystrokes. Today's tips come from a presentation given at YAPC Europe 2011 that have been summed up in this nice post.

The best one is obviously connection sharing, this allows you to share a single ssh connection channel with more than one ssh session, saving the few seconds needed to establish a new connection. Just two config lines that you need to put in your ~/.ssh/config:

ControlMaster auto
ControlPath /tmp/ssh_mux_%h_%p_%r

As stated here, every user (e.g. root) that has access to your control path file will be potentially able to use your established connection, be aware of this. Furthermore, it looks like that without additional Control directives, the disconnection of the master session who opened the first connection also disconnects the other slave sessions, to fix this, as described here you just need to let the connection persist for some time (any timeout would do i guess)

ControlPersist 10m

Glad to see that the suggestion to use SSH key authentication is there too, just remember to use a strong key password (and if you haven't done so already disable root login).

Be sure to checkout the Jumping Through Servers section, a good use of netcat.

On a related note, see Senko's Blog for a script that using inotify launches a command when a file in a directory has been changed. He has built it to automatically perform an rsync on selected directories when changes are detected, nice.

Configuring NGINX for Jekyll on Ubuntu

2011-08-10T00:00:00+02:00

After a few years of shared hosting, a few days ago i moved this site on a Linode VPS and so far i am really happy with it, the performances are impressive, especially for what concern disk i/o, way better then what you get with other VPS providers.

With the new hosting i decided it was time to finally drop Wordpress and the boatload of plugins i was using and try an alternative solution that was more appropriate for a site with a small number of pages and no dynamic content like mine.

The old site was already using a caching plugin for WordPress, W3 Total Cache, to produce static pages (regenerated periodically), on the new site i am giving a try to Jekyll to remove some moving parts and simply serve a static site built offline with a generator.

So far i have no complains, i would have preferred if a basic layout with archive/categories/tags/etc... was bundled with Jekyll but checking the numerous examples available on github i've found my way through anyway, and built an initial skeletal layout in a few days than then evolved in what you are seeing now.

As my HTTP server of choice, i have opted for NGINX. Various independent benchmarks performed with different environments and configurations i have read lately seems to confirm that, with the right configuration, NGINX is one of the best http server for static content serving.

If interested you can read some of them here, here and here.

This kind of tests should be taken with a grain of salt, after all the only meaningful benchmark for you is the one you'll perform on your system with your configuration, here is an interesting discussion on the rules of benchmarking.

This is the configuration i am using at the moment, it's optimized for small files and it enables the file cache to reduce disk access and the number of i/o syscalls. I have performed a few tests on my own and this seems to perform quite well on the first two VPS tier of Linode.

user www-data;
worker_processes  2;

events {
    worker_connections  1024;
}

http {

    include mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
        '$status $body_bytes_sent "$http_referer" '
        '"$http_user_agent" "$http_x_forwarded_for"';

    open_file_cache max=1000 inactive=20s;
    open_file_cache_valid 30s;
    open_file_cache_min_uses 2;

    sendfile off;
    tcp_nodelay on;

    keepalive_timeout  65;

    server {
        listen 80;
        server_name <yourdomain>;
        access_log /var/www/logs/<yourdomain>.access.log  main;
        error_log /var/www/logs/<yourdomain>.error.log;
        root /var/www/sites/<yourdomain>;
        index index.html index.htm;
    }
}

On the NGINX wiki you can find a nice init.d script to launch the server but to manage its lifecycle i suggest to use upstart (installed by default on ubuntu) instead, it will manage your NGINX instance restarting it if it dies unexpectedly.

Create an upstart configuration file at /etc/init/nginx.conf:

description "nginx http daemon"

start on (filesystem and net-device-up IFACE=lo)
stop on runlevel [!2345]

env DAEMON=/usr/local/nginx/sbin/nginx

expect fork
respawn

pre-start script
    $DAEMON -t
    if [ $? -ne 0 ]
        then exit $?
    fi
end script

exec $DAEMON -c /etc/nginx/nginx.conf

Verify that upstart has located your new configuration file, if the output of the following command is empty you should check /var/log/daemon.log for errors in the config file:

initctl list | grep nginx

Now you can start NGINX with:

initctl start nginx

Your NGINX instance will now be respawned if it dies unexpectedly and this can easily be verified with:

killall nginx
ps -Al | grep nginx

Upgrading the ATI drivers and configuring OpenCL in Ubuntu

2010-09-06T00:00:00+02:00

I'm planning a series of post about OpenCL basics and its use ( simple kernels, smoke modelling with Navier-Stokes equation and usage with python) , but before that, a short post on upgrading drivers and basic environment configuration could be useful. Especially when done in conjunction with a kernel upgrade, the update of the ATI graphic drivers is a bit tricky.Unlike CUDA or Stream, OpenCL library is CPU/GPU architecture-independent, so this will be the only post where you'll read something limited to ATI card owners.

We'll use ATI Radeon 57xx logical schematics in the next post to understand how modern GPUs are structured internally but every consideration will also apply to GPUs from other vendors.

The recommended procedure to perform a driver update consists of three steps: uninstalling the old drivers, building custom .deb packages/installing them, build the initial configuration.So, first of all, uninstall the drivers and delete the obsolete fglrx packages:

sudo sh /usr/share/ati/fglrx-uninstall.sh
sudo apt-get remove --purge fglrx_* xorg-driver-fglrx

Download from ati.com the latest driver, generate .deb and install (using 10.04, same procedure applies to other recent builds):

sh ati-driver-installer-10-8-x86.x86_64.run --buildpkg Ubuntu/lucid
sudo dpkg -i fglrx*.deb

Initialize /etc/X11/xorg.conf (a backup is generated) and after this reboot/restart X:

sudo aticonfig --initial -f

To verify that everything has gone as expected (this will display the current driver version):

./fglrxinfo

Regarding OpenCL environment configuration, the installation is easy and fast and the official ATI documentation does a good job.

Get the Stream SDK and the OpenCL ICD configuration (new in 2.2) from the developer site and follow these steps.Untar the SDK in a directory of your choice and add to your .bash_profile (or equivalent) these lines, customizing where needed:

export ATISTREAMSDKROOT=<...>/ati-stream-sdk-v2.2-lnx64export LD_LIBRARY_PATH=$ATISTREAMSDKROOT/lib/x86_64:$LD_LIBRARY_PATH

Untar the ICD configuration archive (will put vendor-specific configuration files in /etc/OpenCL/vendors):

cd /
sudo tar xfz icd-registration.tgz

To verify the successful installation, reload your .bash_profile (source ~/.bash_profile) go to .../ati-stream-sdk-v2.2-lnx64/samples/opencl , compile the opencl samples with make and then run from .../ati-stream-sdk-v2.2-lnx64/samples/opencl/bin/x86_64:

./CLInfo

Verify that both CL_DEVICE_TYPE_CPU and CL_DEVICE_TYPE_GPU are present in the output.As last test and if you want to qualitatively evaluate performance differences, try Mandelbrot with --device cpu and --device gpu.

Amazon AWS, Cloud Computing and economies of scale

2010-04-10T00:00:00+02:00

[![Get Microsoft Silverlight](http://go.microsoft.com/fwlink/?LinkId=108181)](http://go.microsoft.com/fwlink/?LinkID=124807)

A really interesting presentation given at MIX10 (Microsoft sponsored conference) by James Hamilton regarding Cloud computing and its implementation in Amazon Web Services. After a brief introduction about the reasons why AWS is economically sustainable and why it was mandatory for Amazon considering the large amount of underutilized servers they had, he provides a sneak peek on how an Amazon Data Center is structured and what kind of problem them or anyone else building a "cloud" have to address (power/cooling/maximize utilization). Well worth one hour of your time. You'll need Silverlight(win) or Moonlight(lnx/firefox) to play this one, if you don't feel inclined to install it, go straight to the site and download the mp4 recording.

HTC Magic ADB USB driver install instructions

2009-07-06T00:00:00+02:00

If you diligently followed the android driver install instruction from the Android Developer site but your HTC Magic is still discovered only as an "HTC Android Phone USB Device" and an unknown device named "ADB" because the driver contained in the last SDK (android-sdk-windows-1.5_r2\usb_driver\x86) is not recognized, follow these steps:

Open the Device Manager, select the ADB Device and open the Details tab. Look for "Device Instance Id", it should be something like vid_0bb4&pid_0c0*. If your * is not 2, these instructions will probably solve your issue.
Disconnect your device and uninstall the two discovered devices (i used usbdeview but you can do it manually from the Device Manager).
Open with a text editor android-sdk-windows-1.5_r2\usb_driver\x86\android_usb.inf. Replace all the vid_0bb4&pid_0c02 occurrences with vid_0bb4&pid_0c0* (e.g. vid_0bb4&pid_0c03).
Connect your phone and when asked for driver choose to specify manually a path and select _android-sdk-windows-1.5_r2\usb_driver\x86_.
If the installation is successful, to verify that all is ok execute android-sdk-windows-1.5_r2\tools\adb devices, your phone should be listed in the device table.

Thanks to Wysie, for the full discussion see the original thread. After following this procedure even HTC Sync should work.

UPDATE: This procedure should likely work for HTC Hero, Touch and Samsung Galaxy too, just substitute where needed the “Device Instance Id” found at step 1.

UPDATE 4/10/10: Confirmed, this procedure still works for newer phones like the Desire, just use a correct Device Id.

UPDATE 2011: I'm still receiving a good amount of visitors for this page so i guess that installing the android driver under windows is still not straightforward as it should be. As an update, please use the driver provided here instead of the one suggested above. Furthermore, feel free to share your opinions below if these instruction helped you or if you feel the steps suggested are not relevant anymore (from 2009 things could have changed a bit).

DNS-323/CH3SNAS Packages recap

2008-12-15T00:00:00+01:00

Here is a brief recap of the new packages i built for the DNS-323/321/343/CH3SNAS and available here (fonz funplug 0.5 is required):

Lynx 2.8.6: The textual web browser;
Alpine 2.0: Successor of Pine, textual mail reader;
Aria2 1.0: Multi-threaded file download, it should support also bittorrent.

UPDATE: Added Htop 0.8.3 , a nice top-like process manager with tree view and other goodies (simpler arrow navigation , easily accessible kill/nice). As usual, you can install it using:

funpkg -i htop-0.8.3.tgz

Edit: The full binaries repository has now been moved to GitHub

DNS-323/CH3SNAS: A low cost NAS and embedded development

2008-12-13T00:00:00+01:00

If you are looking for a low cost NAS you surely will come across with three devices: the Linksys NSLU2, the Iomega MyBook WE and the D-Link DNS-323 (or its sibling CH3SNAS pitcured on left).Devices with similar functionality (all of them support Samba/FTP/ITunesServer/UPNPAVServer/etc...) and that in a similar fashion have been greatly extended thanks to the community. The NSLU2 with its custom firmware, linux distributions and the optware package repository, the MyBook WE with a first customized firmware and now with a full fledged community and the DNS-323/CH3SNAS with the fonz's fun_plug and the huge amount of modifications that came after that.Something similar to what always happens to devices based on FOSS and when the opportunity to enhance them is given to the community and something, in my opinion, that always boosts the "saleability" of such products, just think to the famous Linksys WRT54G.I got my own Conceptronic CH3SNAS last year, and since then i enjoyed the progress made by the community and contributed my little share when possible.You can now find all my DNS323/CH3SNAS binaries here. Feel free to try them and call for support if needed.

After this long preamble about opensource and NASes, let's talk briefly about Linux and embedded development.There are many ways to start learning Linux for those who come from a MSWindows background such as installing a random distribution, maybe on a virtual machine, but what the DNS323 and similar devices give you is the opportunity to get a taste of what embedded development is.

The hardware platform reminds what you could find in your average pocket-pc and is good enough for this use: a Marvell SoC with an ARM926Tj at 500Mhz and 64Mb of RAM.Don't limit your experience to installing a different UPnP AV Server or the latest Transmission/mldonkey, try to compile some software yourself and experiment.

Browse the WIKI to discover what a toolchain is, how you can cross-compile software for different CPUs (in this case an ARM926) on a x86 host and which are the main differences among glibc and libraries like uclibc, eglibc, etc...

This post should contain enough pointers for you to start!

Edit: The full binaries repository has now been moved to GitHub

Google: The early days

2008-09-12T00:00:00+02:00

In handwritten letters it said, “Google World Headquarters”.

Ubergizmo publishes this touching interview from 1998 and this reminds me of the great book "The Search" by John Battelle , a vivid and detailed description of the search market in the pre-bubble era (plus some chapters with debatable content about the future of the search engine and the search market in general).Maybe it's the right time to start "Founders at Work" or to buy the recent "Once You're Lucky, Twice You're Good".All Books about great people and successful ventures, someone should really write something similar to Banvard's Folly (GREAT book with awesome stories) set in the bubble era, possibly non following the Startup.com documentary lousy approach...

SUN Spot

2008-08-03T00:00:00+02:00

Open-source electronics platforms for rapid prototyping have already been around for a while, i can think of Arduino board but surely there are more of them.The low cost, the wide variety of inputs/outputs, the ability to interact with sensors ot to control motors, the ease of use and the community, make these platforms a perfect fit for hobbyist's and artist's projects.Regarding the ease of use, Arduino can be programmed in a c-like language ( additional straightforward libraries to access the board's functions are provided ) and has a development environment based on the famous Processing.And now, SUN Spot.I've spotted the video embedded below on the googletechtalks youtube channel.The platform developed by SUN Laboratories seems awesome:

Completely open source (hardware/software)
180 MHz 32 bit ARM920T core (with jazelle?) - 512K RAM/4M Flash
2.4 GHz IEEE 802.15.4 radio with integrated antenna
USB interface (host?)
2G/6G 3-axis accelerometer
Temperature sensor
Light sensor
8 tri-color LEDs
6 analog inputs
2 momentary switches
5 general purpose I/O pins and 4 high current output pins
100 mt max distance between Spots (more info on this on the sunspotworld.com forum, the devices together form a meshed network, so multiple stations can be used to extend the network civerage, but the maximum distance between two spots seems limited to that)

Missing agps and only 4mb flash? Looks like someone already built a custom gps module and SD module for their projects. Some new addons have been shown at JavaOne2008.The SUN Spot can be programmed enterely in Java, and has a development platform based on NetBeans (Eclipse can also be used).Refer to the embedded video for some example projects. Above all, watch @00:21 for the cool Sun Blackbox monitoring project.It comes in a developer kit with 1 Spot base station and 2 standard Spots for under $700, a kit that reminds me of the early openmoko developers kit.The two things that i found most intriguing are the ability to upload new "software packages" wirelessly (in this presentation is done through a management interface, could this be done dinamically using the java api provided?) and the way the meshed wireless network formed by Spots is managed (i should take a look at this sooner or later).Additional videos about Sun Spot and projects based on it can be found on youtube.