Clean up maildir folders with python

Time to clean up some older mails for certain folders in my maildir directory, I thought. It didn’t need to be something fancy, just that I wanted to put different requirements for days to keep per folder. First I thought I’d go looking in the maildir and look per file for file-timestamps and clean up directly from there, but Python has the ‘mailbox’ module for dealling with mailboxes in mbox/maildir formats.

So after throwing some code together, here’s a basic result.

#!/usr/bin/env python3
# Clean up script for certain old mail folders

import mailbox
import os
import time

# Maildir main folder
mailpath = os.environ.get('HOME') + '/Mail/'

# List of tuples with (Maildir subfolder, number of days to keep)
cleanup_paths = [
    ('0_MAYBE_SPAM', 3),
    ('0_SPAM', 3),
    ('Trash', 3),
    ('Internet.CR-Net_Log', 7),
    ('CR-Net.Promotions', 31),

def main():
    now = time.time()
    print("Cleaning up old mails...")
    maildir = mailbox.Maildir(mailpath)
    for (cl_path, keep_days) in cleanup_paths:
        print("*** Processing {} discarding older than {} days".format(cl_path, keep_days))
        keep_time = now - keep_days * 86400
        cleanfolder = maildir.get_folder(cl_path)
        for filename, msg in cleanfolder.items():
            # If epoch timestamp of the msg is smaller than keep_time epoch timestamp
            if msg.get_date() < keep_time:
                print("Filename:", filename, "\nSubject:", msg['subject'], "\nDate", msg['date'], "\n----------------------------------")

if __name__ == '__main__':

You can also find the code at Bitbucket and Github

DRM-Free e-books

Recently I have bought a new e-reader, the Kobo Aura HD. I’m very happy with it. My previous reading device was a 7.7″ Android tablet (Samsung Galaxy 7.7), which is, by itself, an excellent device, but where it fails (just like all other tablets), is when it comes to reading outside in the sunlight. It can not cope with the external light, the glare and reflection of the screen, and does not deliver enough contrast.
So enters the e-paper e-reader..

Having such a new toy also made me start reading again on a higher pace, so I continued with the “A Song of Ice And Fire” series. I have the bundle with the 4 books which I bought through Amazon. Of course, that (like most of their books), came with DRM installed.
After a little hassle to have the DRM stripped, so I could import the book into Calibre, which runs on my Linux PC, it reads perfectly on my new device. But that raised me the following question for my future reading:

What e-book shop offers A Dance of Dragons DRM-free?

Both the Kobo webshop and Amazon have Adobe DRM, which is an extra hassle to import in Calibre.

Shops without DRM have my preference. Shops that “Do it well”TM are O’Reilly and InformIT which offer DRM-free books. O’Reilly even pushes them automatically to Kindle and Google Drive, and both offer plain downloads from their website in various formats (Kindle, PDF, EPUB, .. ).

That’s how it should be, let the consumer choose his reading device. Walled gardens are inconvenient. The music industry has realized that as well. Next should be book and movie/series publishers.

In short. I prefer e-book shops that make my lazy life easier. Get to work, and you have an extra customer.

Playing with KVM virtualisation – part 1.1

Looks like I bumped into a small issue when updating one of the VMs (Debian) I was playing with: Shortage of diskspace on the root volume. Disaster when running dist-upgrade!

# df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/vm2-root    1.5G  1.4G     0 100% /
udev                     10M     0   10M   0% /dev
tmpfs                    25M  224K   25M   1% /run
tmpfs                   5.0M     0  5.0M   0% /run/lock
tmpfs                    49M     0   49M   0% /run/shm
/dev/vda1               228M   58M  158M  27% /boot
/dev/mapper/vm2-home    2.1G  3.6M  2.0G   1% /home

Fortunately, it’s an LVM so it’s not so difficult to extend this, even on-the-fly.

Let’s create our new disk first

virsh# vol-create-as MyVolume vm2-2.img 2G --format raw
Vol vm2-2.img created

virsh # vol-list --details MyVolume
Name                 Path                                  Type   Capacity  Allocation
Android-x86-4.0.img  /virtualmachines/Android-x86-4.0.img  file   4,00 GiB  498,75 MiB
cvm3.img             /virtualmachines/cvm3.img             file  16,00 GiB   61,81 MiB
vm2-2.img            /virtualmachines/vm2-2.img            file   2,00 GiB    2,00 GiB
vm2.img              /virtualmachines/vm2.img              file   4,00 GiB    1,83 GiB

Now that we have created the volume, let’s attach it to the VM. It is possible while it’s running live.

virsh # list
 Id    Name                           State
 4     vm2                            running

virsh # attach-disk vm2 /virtualmachines/vm2-2.img vdc
Disk attached successfully

Within the VM it’s automatically detected as well.

vm2# tail /var/log/kern.log
Dec 21 11:58:00 vm2 kernel: [ 3978.289939]  vdc: unknown partition table

The new disk is there, but it has not been partitioned yet. So let’s do that

root@vm2:/# fdisk /dev/vdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xc9f8eafe.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): m
Command action
a toggle a bootable flag
b edit bsd disklabel
c toggle the dos compatibility flag
d delete a partition
l list known partition types
m print this menu
n add a new partition
o create a new empty DOS partition table
p print the partition table
q quit without saving changes
s create a new empty Sun disklabel
t change a partition's system id
u change display/entry units
v verify the partition table
w write table to disk and exit
x extra functionality (experts only)

Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-4095999, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-4095999, default 4095999):
Using default value 4095999

Command (m for help): t
Selected partition 1

Hex code (type L to list codes): 8e

8e is the code for Linux LVM

Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Done! Now we need to add it to our LVM. First let’s check the volume group (vm2 here).

# vgdisplay
  --- Volume group ---
  VG Name               vm2
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               3.76 GiB
  PE Size               4.00 MiB
  Total PE              962
  Alloc PE / Size       962 / 3.76 GiB
  Free  PE / Size       0 / 0   
  VG UUID               ksC1ha-y05g-E6gx-JshG-b09N-dw6q-kcF2t7

As you can see, no Free space left (Free PE / Size). We are ready to add our new disk and its freshly created partition

# vgextend vm2 /dev/vdb1
No physical volume label read from /dev/vdb1
Physical volume "/dev/vdb1" successfully created
Volume group "vm2" successfully extended

# vgdisplay
  --- Volume group ---
  VG Name               vm2
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  6
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               5.71 GiB
  PE Size               4.00 MiB
  Total PE              1461
  Alloc PE / Size       1218 / 4.76 GiB
  Free  PE / Size       243 / 972.00 MiB
  VG UUID               ksC1ha-y05g-E6gx-JshG-b09N-dw6q-kcF2t7

Great! The new space has been allocated (VG Size, Free PE / Size). Now we are ready to exptend the root volume

# lvextend -L+1G /dev/vm2/root
Extending logical volume root to 2.47 GiB
Logical volume root successfully resized
# df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/vm2-root    2.5G  1.4G  945M  60% /
udev                     10M     0   10M   0% /dev
tmpfs                    25M  228K   25M   1% /run
tmpfs                   5.0M     0  5.0M   0% /run/lock
tmpfs                    49M     0   49M   0% /run/shm
/dev/vda1               228M   58M  158M  27% /boot
/dev/mapper/vm2-home    2.1G  3.6M  2.0G   1% /home

Just extending the logical volume does not automatically add the free space to the filesystem, that needs to be resized as well.

# resize2fs -p /dev/vm2/root 
resize2fs 1.42.8 (20-Jun-2013)
Filesystem at /dev/vm2/root is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/vm2/root is now 648192 blocks long.

root@vm2:/dev# df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/vm2-root    2.5G  1.4G  945M  60% /
udev                     10M     0   10M   0% /dev
tmpfs                    25M  224K   25M   1% /run
tmpfs                   5.0M     0  5.0M   0% /run/lock
tmpfs                    49M     0   49M   0% /run/shm
/dev/vda1               228M   58M  158M  27% /boot
/dev/mapper/vm2-home    2.1G  3.6M  2.0G   1% /home

Great success! One last look to the volume group stats shows that not all space has been allocated yet.

# vgdisplay
  --- Volume group ---
  VG Name               vm2
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  6
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               5.71 GiB
  PE Size               4.00 MiB
  Total PE              1461
  Alloc PE / Size       1218 / 4.76 GiB
  Free  PE / Size       243 / 972.00 MiB
  VG UUID               ksC1ha-y05g-E6gx-JshG-b09N-dw6q-kcF2t7

In case we need some tweaking to other logical volumes, it can be added accordingly.

Playing with KVM virtualisation

When I attended Loadays 2013 back in the beginning of April, I was introduced to Ansible, an interesting configuration management system. In order to properly play with it I wanted to have a couple of hosts that could be messed with. Virtual machines are perfect for that. For the exercise I opted for KVM. No particular reason, other options are just as fine for this. This is also mainly a beginning reference for myself, so the setup can probably be handled in a much better and efficient way (don’t hesitate to tell me, of course).

For the impatient, the source files as example below can be found in this Bitbucket repository


Create bridge interface

First of all, in order to have different IP addresses assigned, and you want to have the VMs connect directly on your physical network, you can create a bridge interface br0, which all the VMs can use for networking. In the example scripts below, this way is used.

A good starting point for more information on setting this up on Debian can be found on the Debian Wiki BridgeNetworkConnections page.


For this setup, we’re using libVirt

Webserver to host files

For testing, I installed a simple Apache Webserver which hosted the files in its root directory. YMMV (which is also recommended)

Create virtual machines

Of course, this can all be done through the virt-manager GUI as well, but when working remote, without an X-server or VNC at your disposal, it’s handy to know some command-line as well. And to reduce typing the same thing over and over (although it’s not bad to get the commands in your muscle memory), let’s put it in a small script. (Inspiration was found here)

Creation scripts

Note: The example scripts are for testing purposes, and are meant to generate a small image. Your usage requirements will most probably differ.

# A script to install Debian Wheezy on a KVM guest
# using automatic installation from installable distribution
# image on an HTTP server

# Defined MAC range 52:54:00:8E:FF:00 - 52:54:00:8E:FF:FF
# Choose the last number to your own preference
if [ $# -ne 1 ]
echo "Usage: $0 guest-name"
exit 1

virt-install \
--connect=qemu:///system \
--virt-type kvm \
--name=${1} \
--ram=${RAM} \
--vcpus=${CPUs} \
--disk path=/var/lib/libvirt/images/${1}.img,size=${size} \
-l \
--vnc \
--os-variant debianwheezy \
--noautoconsole \
--hvm \
--network bridge=br0,mac=${VMMAC} \
--extra-args="auto=true hostname=${1} domain=${domain} url=${wwwhost}/preseed-debian_vm.txt text console=tty1 console=ttyS0,115200"

Call this script like this

# vm1

and it will result in a call to libvirt to create a new virtual machine, with 2 CPUs, 512MB RAM and 2 GB image on disk. 1 CPU and 256MB also work fine, you can easily define that in the script (or later in the generated xml config file for the VM, located at /etc/libvirt/qemu/vm1.xml). It also has a console that you can attach to from within virsh.

To see the progress of the installation, type

# virsh console vm1

The -l parameter in the script tells us to connect to the Debian mirror to fetch the online installation image for Wheezy. It also adds some kernel parameters with –extra-args which tells the installer to fetch a pre-configured preseed file from the wwwhost you defined above.

Headless installation files

An example preseed.txt file has been put in the www subdirectory of the repository. It will create a minimal install of Debian Wheezy, with a belgian mirror configured. Its filesystem is only 1 partition without LVM. No accessible root user is configured, but the pre-defined “myuser” has sudo rights for root permissions. Some useful (YMMV) packages are also automatically installed.

CentOS headless installation

I’ve also added VM-creation script for CentOS. There is a corresponding Kickstart file which gives a hardened CentOS image. The main source of inspiration was found in RackerHackers Github repository

VM fun

Listing the VMs
# virsh list --all
 Id    Name                           State
 1     vm1                            running
 2     vm2                            shut off
Starting the VM

# virsh start vm1
Domain vm1 started

Stopping the VM

# virsh shutdown vm1
Domain vm1 is being shutdown

virsh commands and help
# virsh help 
 Virsh itself (help keyword 'virsh'):
    cd                             change the current directory
    echo                           echo arguments
    exit                           quit this interactive terminal
    help                           print help
    pwd                            print the current directory
    quit                           quit this interactive terminal

Cloning fun

If you want to backup or clone your VM, you can use the virt-clone command.

# virt-clone -o vm1 -n vm2 -m 52:54:00:8E:FF:02 -f /var/lib/libvirt/images/vm2.img

A new VM will be created, exactly the same as the original image, except for the name, MAC address and image-file. Also note that the ssh-host keys will be identical, so they’d need to be regenerated.

The End

With enough tiny clones, you are ready for the other fun stuff: orchestration. Those first steps will probably be dealt with in a future blogpost about Ansible and/or Puppet

And to end, for those who want to go even further in this:

And most of all, have fun, if you feel something needs to be added or corrected, let me know!

Continuum vs TOR

A short attempt to a little technobabble in a new series called “Continuum“.
Looks like Hollywood does have some hopes to crack into the TOR network, somehow :-)

(edit: Looks like the series is Canadian, so no Hollywood)

Transcript (although it has a few gaps or mistakes)

"I wanted to let you know I've been doing some digging. The Portland PD database was hacked by a third-party server."
"That's.. good?"
"Not bad.. depending on who did the hacking.. I've been following the [..] trail back to the originating server. He or she is using TOR, to encrypt their TCP streams and cover their tracks. I've been funneling the gaps between incoming and outgoing packets.. and they're narrowing" *smile*
"So you're going to find out who's behind the curtain, right?"
"Pretty soon he's gonna overstay his welcome, and I'm gonna nail his ass."

Get swap usage per process.

A few months ago, I had the need to check what the swap usage was on my Linux system.

(Update 2014-10-26: This can also be checked with the excellent ‘smem‘ tool)

I encountered a little bash script by a guy called Erik Ljungstrom, who had written a small script that iterates through the /proc system, takes the name for the PID and adds up the swap space.

Here’s the original script.

# Get current swap usage for all running processes
# Erik Ljungstrom 27/05/2011
for DIR in `find /proc/ -maxdepth 1 -type d | egrep "^/proc/[0-9]"` 
  PID=`echo $DIR | cut -d / -f 3`
  PROGNAME=`ps -p $PID -o comm --no-headers`
  for SWAP in `grep Swap $DIR/smaps 2>/dev/null| awk '{ print $2 }'`
    let SUM=$SUM+$SWAP
  echo "PID=$PID - Swap used: $SUM - ($PROGNAME )"
echo "Overall swap used: $OVERALL"

It’s not the fasted script, but it does its job, and you can do a

# ./ | sort -n -k 5

to have the values sorted in ascending order.

And as it was a free day, I needed to brush up some basic Perl skillz again.

Here’s the result:

#!/usr/bin/perl -w
# Get current swap usage for all running processes
# Based on a bash script by Erik Ljungstrom 27/05/2011
# Author = Christopher Ranschaert - 07/07/2012 
# Parses pids in /proc, collects the name from /proc/$pid/comm and 
# calculates the swapsize from /proc/$pid/smaps
# Warning: Needs root permissions to read swap values, will otherwise return 0.
# If you don't trust this, learn some Perl and check the code :-)
# Something I'd suggest you to do anyway!
use strict;
my $overall = 0;
my $dirname = "/proc";
my %names = ();
my %swaps = ();
my $processname = '';
my $pidnumber = '';
my $swapsizenumber = '';
# Open /proc directory and parse for PIDs and store them in hash.
opendir(DIR, $dirname) or die "Couldn't open $dirname: $!";
while ( defined ( my $pid = readdir DIR )){
  next unless $pid =~ /^\d+$/;  # PIDs are digits only, skip the rest
  if (-d "$dirname/$pid") {
    $names{$pid} = &read_processname_from($pid);
    $swaps{$pid} = &get_swapsize_from($pid);

format STDOUT =
@<<<<<< | @<<<<< | @<<<<<<<<<<<<<<< |

print "-------------------------------------\n";
print "Processes and their swapsizes\n";
print "-------------------------------------\n";
print " PID | SWAP | PROCESSNAME |\n";
print "-------------------------------------\n";
my @pids = sort { $swaps{$a} <=> $swaps{$b} } (keys %swaps);
foreach my $pid ( @pids ){
  $pidnumber = $pid;
  $processname = $names{$pid};
  $swapsizenumber = $swaps{$pid};
  $overall += $swapsizenumber;
print "-------------------------------------\n";
print "Overall Swap used: $overall KiB\n";
exit 0;
sub read_processname_from {
  my $pid = shift;
  my $procnamefile = "$dirname/$pid/comm";
  open (FH, $procnamefile) or die "Can't open $procnamefile: $!";
  my $procname = <FH>;
  chomp $procname;
  return $procname;
sub get_swapsize_from {
  my $pid = shift;
  my $sum = 0;
  my $smapsfile = "$dirname/$pid/smaps";
  open (FH, $smapsfile) or die "Can't open $smapsfile: $!";
  while (<FH>){
    next unless /^Swap/;
    $_ =~ m/^Swap:\s+(\d+)\s+kB*$/;
    my $swapsize = $1;
    $sum += $swapsize;
 close (FH);
 return $sum;

It already includes sorting so the extra sort command is no longer needed.

Output sample:

# ./
Processes and their swapsizes
 PID  | SWAP | PROCESSNAME          |
27007 | 0    | migration/2          |
429   | 832  | udevd                |
3169  | 832  | puppet               |
3327  | 1048 | sensord              |
2382  | 1800 | colord               |
Overall Swap used: 24072 KiB

In case you find it interesting, you can also find the script itself at my bitbucket repository:

Feel free to leave comments on how it sucks, but do inform me of your great improvements!

Uw treinreis op 31 januari 2012

Jullie herinneren zich nog de brief die ik eerder deze maand aan de Ombudsdienst van de NMBS richtte. Ook heb ik beloofd jullie op de hoogte te houden.

Bij deze, hun reactie, en inhoudelijk is die niet geheel onverwacht.

Zoals ik in de brief ook vermeldde ben ik me bewust van de opmaak van de regeling, maar is de toepassing van “per jaar” niet geheel duidelijk. Volgens de treinbegeleider kwam het neer op “kalenderjaar”, en wordt het aantal in januari terug op 0 gezet. Aan het loket vermeldde de bediende dat het ging over “een periode van een jaar”.

Aan de bijzondere regelingsprocedure zijn enkele strikte voorwaarden gekoppeld. Bovendien worden slechts twee vergetelheden per jaar kosteloos geregeld.

En zoals je kan zien in deze brief wordt het nog steeds niet duidelijk uitgelegd.

Bij nazicht van onze computerbestanden blijkt dat u reeds meerdere keren gebruik maakte van deze kosteloze regeling. In principe bent u dan ook het bedrag, vermeld op de vaststelling, verschuldigd.

Dit was inderdaad de derde keer in een periode van een jaar, maar de eerste keer in 2012. Verder begrijp ik niet dat men anno 2012 nog deze ouderwetse manier van klantenbehandeling aanhoudt (administratieve kosten). Als men perfect kan nagaan hoeveel keer ik mijn abonnement vergeten ben, in een periode van een jaar, dan weet men ook perfect dat ik een abonnement heb en wel degelijk voor welke periode die geldig is. En dan begrijp ik niet dat ik, toch wel betalende klant, nogmaals extra moet bijbetalen. Zo hou je echt wel een positief imago..
Als dit dan toch wettelijk is vastgelegd, dan zou het geen slecht idee zijn van onze Minister van Openbaar Vervoer om de klantgerichtheid wat meer te stimuleren, en deze belachelijke regeling uit de wet te halen. Dit raakt de reeds betalende klant al even erg als de verplichte anti-piraterij boodschappen en domme reclame bij legaal aangekochte DVDs en cinemabezoek.

En nu de volledige brief zelf:

Geachte Heer

Wij ontvingen uw opmerkingen over de onregelmatigheid die ons personeel vaststelde naar aanleiding van uw treinreis op 31 januari 2012.

Onze bediende maakte een vaststelling van een onregelmatigheid (document “C170″) op omdat u bij de controle geen geldig vervoerbewijs kon voorleggen.

U verklaarde immers dat u uw treinkaart vergeten had.

Om met de trein te reizen moet u steeds in het bezit zijn van een geldig vervoerbewijs, dat u aankoopt voor u instapt. Kon u zich geen vervoerbewijs aanschaffen, dan geldt één regel: verwittig de treinbegeleider spontaan voor u instapt of zo snel mogelijk na het vertrek. Was de biljettenverkoop in uw vertrekstation verzekerd, dan wordt u in de trein een toeslag van 3 EUR aangerekend. Verwittigde u niet dan betaalt u de prijs van de reis en een toeslag van 12,50 EUR. Zo vermijdt u dat een vaststelling van een onregelmatigheid wordt opgemaakt. Betaalt u dit bedrag niet in de trein, dan stelt de treinbegeleider een <<vaststelling van een onregelmatigheid reiziger>> (formulier C170) op, waarop hij de prijs van de reis en de toeslag vermeldt die u in het station moet betalen. Die toeslag bedraagt 60,00 EUR voor een volwassene en 30,00 EUR voor een minderjarige. U beschikt over 14 kalenderdagen om uw dossier te regelen. Daarna wordt de toeslag verhoogd tot 200,00 EUR. Stellen we vast dat uw betaling uitblijft, dan zal de NMBS haar rechten via gerechtelijke weg laten gelden, met alle mogelijke financiële gevolgen…

Onze controlebedienden kunnen in de trein onmogelijk nagaan of de reiziger inderdaad over een geldige treinkaart beschikt. Daarom maken zij in dit geval steeds een vaststellingsformulier. Wanneer de klant zich nadien aanbiedt in het station binnen de veertien dagen met het vaststellingsformulier en zijn geldige kaart voorlegt, wordt het dossier geregeld.

Aan de bijzondere regelingsprocedure zijn enkele strikte voorwaarden gekoppeld. Bovendien worden slechts twee vergetelheden per jaar kosteloos geregeld.

Bij nazicht van onze computerbestanden blijkt dat u reeds meerdere keren gebruik maakte van deze kosteloze regeling. In principe bent u dan ook het bedrag, vermeld op de vaststelling, verschuldigd.

Uit commercieel oogpunt zijn wij uitzonderlijk bereid af te wijken van de wettelijke bepalingen en slechts het minimumbedrag van 7,70 EUR administratiekosten aan te rekenen voor de regeling van het dossier.

We nodigen u bijgevolg uit om binnen de veertien dagen het bedrag van 7,70 EUR over te maken op het rekeningnummer BE … van de NMBS, B-Mo 0613, Hallepoortlaan 40 te 1060 Brussel, met vermelding van het dossierkenmerk. U kan hiertoe bijgevoegd overschrijvingsformulier gebruiken.

Na ontvangst van dit bedrag zullen wij het dossier afsluiten.

Met vriendelijke groeten,