Clean up maildir folders with python

Time to clean up some older mails for certain folders in my maildir directory, I thought. It didn’t need to be something fancy, just that I wanted to put different requirements for days to keep per folder. First I thought I’d go looking in the maildir and look per file for file-timestamps and clean up directly from there, but Python has the ‘mailbox’ module for dealling with mailboxes in mbox/maildir formats.

So after throwing some code together, here’s a basic result.

#!/usr/bin/env python3
# Clean up script for certain old mail folders

import mailbox
import os
import time

# Maildir main folder
mailpath = os.environ.get('HOME') + '/Mail/'

# List of tuples with (Maildir subfolder, number of days to keep)
cleanup_paths = [
    ('0_MAYBE_SPAM', 3),
    ('0_SPAM', 3),
    ('Trash', 3),
    ('Internet.CR-Net_Log', 7),
    ('CR-Net.Promotions', 31),
]

def main():
    now = time.time()
    print("Cleaning up old mails...")
    maildir = mailbox.Maildir(mailpath)
    for (cl_path, keep_days) in cleanup_paths:
        print("*** Processing {} discarding older than {} days".format(cl_path, keep_days))
        keep_time = now - keep_days * 86400
        cleanfolder = maildir.get_folder(cl_path)
        for filename, msg in cleanfolder.items():
            # If epoch timestamp of the msg is smaller than keep_time epoch timestamp
            if msg.get_date() < keep_time:
                print("Filename:", filename, "\nSubject:", msg['subject'], "\nDate", msg['date'], "\n----------------------------------")
                cleanfolder.remove(filename)
    maildir.close()

if __name__ == '__main__':
    main()

You can also find the code at Bitbucket and Github

DRM-Free e-books

Recently I have bought a new e-reader, the Kobo Aura HD. I’m very happy with it. My previous reading device was a 7.7″ Android tablet (Samsung Galaxy 7.7), which is, by itself, an excellent device, but where it fails (just like all other tablets), is when it comes to reading outside in the sunlight. It can not cope with the external light, the glare and reflection of the screen, and does not deliver enough contrast.
So enters the e-paper e-reader..

Having such a new toy also made me start reading again on a higher pace, so I continued with the “A Song of Ice And Fire” series. I have the bundle with the 4 books which I bought through Amazon. Of course, that (like most of their books), came with DRM installed.
After a little hassle to have the DRM stripped, so I could import the book into Calibre, which runs on my Linux PC, it reads perfectly on my new device. But that raised me the following question for my future reading:

What e-book shop offers A Dance of Dragons DRM-free?

Both the Kobo webshop and Amazon have Adobe DRM, which is an extra hassle to import in Calibre.

Shops without DRM have my preference. Shops that “Do it well”TM are O’Reilly and InformIT which offer DRM-free books. O’Reilly even pushes them automatically to Kindle and Google Drive, and both offer plain downloads from their website in various formats (Kindle, PDF, EPUB, .. ).

That’s how it should be, let the consumer choose his reading device. Walled gardens are inconvenient. The music industry has realized that as well. Next should be book and movie/series publishers.

In short. I prefer e-book shops that make my lazy life easier. Get to work, and you have an extra customer.

Playing with KVM virtualisation – part 1.1

Looks like I bumped into a small issue when updating one of the VMs (Debian) I was playing with: Shortage of diskspace on the root volume. Disaster when running dist-upgrade!

# df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/vm2-root    1.5G  1.4G     0 100% /
udev                     10M     0   10M   0% /dev
tmpfs                    25M  224K   25M   1% /run
tmpfs                   5.0M     0  5.0M   0% /run/lock
tmpfs                    49M     0   49M   0% /run/shm
/dev/vda1               228M   58M  158M  27% /boot
/dev/mapper/vm2-home    2.1G  3.6M  2.0G   1% /home

Fortunately, it’s an LVM so it’s not so difficult to extend this, even on-the-fly.

Let’s create our new disk first

virsh# vol-create-as MyVolume vm2-2.img 2G --format raw
Vol vm2-2.img created

virsh # vol-list --details MyVolume
Name                 Path                                  Type   Capacity  Allocation
--------------------------------------------------------------------------------------
Android-x86-4.0.img  /virtualmachines/Android-x86-4.0.img  file   4,00 GiB  498,75 MiB
cvm3.img             /virtualmachines/cvm3.img             file  16,00 GiB   61,81 MiB
vm2-2.img            /virtualmachines/vm2-2.img            file   2,00 GiB    2,00 GiB
vm2.img              /virtualmachines/vm2.img              file   4,00 GiB    1,83 GiB

Now that we have created the volume, let’s attach it to the VM. It is possible while it’s running live.

virsh # list
 Id    Name                           State
----------------------------------------------------
 4     vm2                            running

virsh # attach-disk vm2 /virtualmachines/vm2-2.img vdc
Disk attached successfully

Within the VM it’s automatically detected as well.

vm2# tail /var/log/kern.log
...
Dec 21 11:58:00 vm2 kernel: [ 3978.289939]  vdc: unknown partition table

The new disk is there, but it has not been partitioned yet. So let’s do that

root@vm2:/# fdisk /dev/vdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xc9f8eafe.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): m
Command action
a toggle a bootable flag
b edit bsd disklabel
c toggle the dos compatibility flag
d delete a partition
l list known partition types
m print this menu
n add a new partition
o create a new empty DOS partition table
p print the partition table
q quit without saving changes
s create a new empty Sun disklabel
t change a partition's system id
u change display/entry units
v verify the partition table
w write table to disk and exit
x extra functionality (experts only)

Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-4095999, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-4095999, default 4095999):
Using default value 4095999

Command (m for help): t
Selected partition 1

Hex code (type L to list codes): 8e

8e is the code for Linux LVM

Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Done! Now we need to add it to our LVM. First let’s check the volume group (vm2 here).

# vgdisplay
  --- Volume group ---
  VG Name               vm2
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               3.76 GiB
  PE Size               4.00 MiB
  Total PE              962
  Alloc PE / Size       962 / 3.76 GiB
  Free  PE / Size       0 / 0   
  VG UUID               ksC1ha-y05g-E6gx-JshG-b09N-dw6q-kcF2t7

As you can see, no Free space left (Free PE / Size). We are ready to add our new disk and its freshly created partition

# vgextend vm2 /dev/vdb1
No physical volume label read from /dev/vdb1
Physical volume "/dev/vdb1" successfully created
Volume group "vm2" successfully extended

# vgdisplay
  --- Volume group ---
  VG Name               vm2
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  6
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               5.71 GiB
  PE Size               4.00 MiB
  Total PE              1461
  Alloc PE / Size       1218 / 4.76 GiB
  Free  PE / Size       243 / 972.00 MiB
  VG UUID               ksC1ha-y05g-E6gx-JshG-b09N-dw6q-kcF2t7

Great! The new space has been allocated (VG Size, Free PE / Size). Now we are ready to exptend the root volume

# lvextend -L+1G /dev/vm2/root
Extending logical volume root to 2.47 GiB
Logical volume root successfully resized
# df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/vm2-root    2.5G  1.4G  945M  60% /
udev                     10M     0   10M   0% /dev
tmpfs                    25M  228K   25M   1% /run
tmpfs                   5.0M     0  5.0M   0% /run/lock
tmpfs                    49M     0   49M   0% /run/shm
/dev/vda1               228M   58M  158M  27% /boot
/dev/mapper/vm2-home    2.1G  3.6M  2.0G   1% /home

Just extending the logical volume does not automatically add the free space to the filesystem, that needs to be resized as well.

# resize2fs -p /dev/vm2/root 
resize2fs 1.42.8 (20-Jun-2013)
Filesystem at /dev/vm2/root is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/vm2/root is now 648192 blocks long.

root@vm2:/dev# df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/vm2-root    2.5G  1.4G  945M  60% /
udev                     10M     0   10M   0% /dev
tmpfs                    25M  224K   25M   1% /run
tmpfs                   5.0M     0  5.0M   0% /run/lock
tmpfs                    49M     0   49M   0% /run/shm
/dev/vda1               228M   58M  158M  27% /boot
/dev/mapper/vm2-home    2.1G  3.6M  2.0G   1% /home

Great success! One last look to the volume group stats shows that not all space has been allocated yet.

# vgdisplay
  --- Volume group ---
  VG Name               vm2
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  6
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               5.71 GiB
  PE Size               4.00 MiB
  Total PE              1461
  Alloc PE / Size       1218 / 4.76 GiB
  Free  PE / Size       243 / 972.00 MiB
  VG UUID               ksC1ha-y05g-E6gx-JshG-b09N-dw6q-kcF2t7

In case we need some tweaking to other logical volumes, it can be added accordingly.

Playing with KVM virtualisation

When I attended Loadays 2013 back in the beginning of April, I was introduced to Ansible, an interesting configuration management system. In order to properly play with it I wanted to have a couple of hosts that could be messed with. Virtual machines are perfect for that. For the exercise I opted for KVM. No particular reason, other options are just as fine for this. This is also mainly a beginning reference for myself, so the setup can probably be handled in a much better and efficient way (don’t hesitate to tell me, of course).

For the impatient, the source files as example below can be found in this Bitbucket repository

Prerequisites

Create bridge interface

First of all, in order to have different IP addresses assigned, and you want to have the VMs connect directly on your physical network, you can create a bridge interface br0, which all the VMs can use for networking. In the example scripts below, this way is used.

A good starting point for more information on setting this up on Debian can be found on the Debian Wiki BridgeNetworkConnections page.

LibVirt

For this setup, we’re using libVirt

Webserver to host files

For testing, I installed a simple Apache Webserver which hosted the files in its root directory. YMMV (which is also recommended)

Create virtual machines

Of course, this can all be done through the virt-manager GUI as well, but when working remote, without an X-server or VNC at your disposal, it’s handy to know some command-line as well. And to reduce typing the same thing over and over (although it’s not bad to get the commands in your muscle memory), let’s put it in a small script. (Inspiration was found here)

Creation scripts

Note: The example scripts are for testing purposes, and are meant to generate a small image. Your usage requirements will most probably differ.

#!/bin/sh
#
# A script to install Debian Wheezy on a KVM guest
# using automatic installation from installable distribution
# image on an HTTP server

# Defined MAC range 52:54:00:8E:FF:00 - 52:54:00:8E:FF:FF
# Choose the last number to your own preference
 
if [ $# -ne 1 ]
then
echo "Usage: $0 guest-name"
exit 1
fi

size=2
CPUs=2
RAM=512
MyVMMACRANGE='52:54:00:8e:ff'
VMMAC=${MyVMMACRANGE}:00
domain=home
wwwhost=http://example.com
 
virt-install \
--connect=qemu:///system \
--virt-type kvm \
--name=${1} \
--ram=${RAM} \
--vcpus=${CPUs} \
--disk path=/var/lib/libvirt/images/${1}.img,size=${size} \
-l http://ftp.be.debian.org/debian/dists/wheezy/main/installer-amd64/ \
--vnc \
--os-variant debianwheezy \
--noautoconsole \
--hvm \
--network bridge=br0,mac=${VMMAC} \
--extra-args="auto=true hostname=${1} domain=${domain} url=${wwwhost}/preseed-debian_vm.txt text console=tty1 console=ttyS0,115200"

Call this script like this

# create_vm-debianwheezy.sh vm1

and it will result in a call to libvirt to create a new virtual machine, with 2 CPUs, 512MB RAM and 2 GB image on disk. 1 CPU and 256MB also work fine, you can easily define that in the script (or later in the generated xml config file for the VM, located at /etc/libvirt/qemu/vm1.xml). It also has a console that you can attach to from within virsh.

To see the progress of the installation, type

# virsh console vm1

The -l parameter in the script tells us to connect to the Debian mirror to fetch the online installation image for Wheezy. It also adds some kernel parameters with –extra-args which tells the installer to fetch a pre-configured preseed file from the wwwhost you defined above.

Headless installation files

An example preseed.txt file has been put in the www subdirectory of the repository. It will create a minimal install of Debian Wheezy, with a belgian mirror configured. Its filesystem is only 1 partition without LVM. No accessible root user is configured, but the pre-defined “myuser” has sudo rights for root permissions. Some useful (YMMV) packages are also automatically installed.

CentOS headless installation

I’ve also added VM-creation script for CentOS. There is a corresponding Kickstart file which gives a hardened CentOS image. The main source of inspiration was found in RackerHackers Github repository

VM fun

Listing the VMs

# virsh list --all
 Id    Name                           State
----------------------------------------------------
 1     vm1                            running
 2     vm2                            shut off

Starting the VM

# virsh start vm1
Domain vm1 started

Stopping the VM

# virsh shutdown vm1
Domain vm1 is being shutdown

virsh commands and help

# virsh help 
... 
 Virsh itself (help keyword 'virsh'):
    cd                             change the current directory
    echo                           echo arguments
    exit                           quit this interactive terminal
    help                           print help
    pwd                            print the current directory
    quit                           quit this interactive terminal

Cloning fun

If you want to backup or clone your VM, you can use the virt-clone command.

# virt-clone -o vm1 -n vm2 -m 52:54:00:8E:FF:02 -f /var/lib/libvirt/images/vm2.img

A new VM will be created, exactly the same as the original image, except for the name, MAC address and image-file. Also note that the ssh-host keys will be identical, so they’d need to be regenerated.

The End

With enough tiny clones, you are ready for the other fun stuff: orchestration. Those first steps will probably be dealt with in a future blogpost about Ansible and/or Puppet

And to end, for those who want to go even further in this:

• Self-replicating virtual machine through Ansible provisioning
• Automating Debian Installs with Preseed and Puppet

And most of all, have fun, if you feel something needs to be added or corrected, let me know!

Lost harddisk

As promised earlier, I’d be writing a small blogpost today. And I had warned before it was on a geekier subject than booze.. Although some people might have another opinion about that.

Last week I had lost my primary harddrive which contained my root directory and home directory. Fortunately as various thousands of websites have suggested, I have backups of the most important things. Now of course, you always forget to include some new things in your backup plan, which was the case here, as I didn’t consider enough space just yet for a full image of all my data. Yes, that’ll be my upcoming correction and we all learn from our mistakes now, don’t we?

The main symptom:

When booting the PC, the disk is not recognized by the BIOS anymore. Normally at POST you should get an ID of your disks. Unfortunately nothing appeared, so my heart sank seeing that, as no other signs of breaking were seen nor heard..

Attempts to fix:

• Fiddling the cables:

Of course one of the first things you check are loose cables, wires, .. Nothing seemed to be wrong. Pulling out all other disks, leaving this broken one alone and lonely under my watchful glare.. didn’t seem to do the trick. So let’s try different cables, SATA ports, all to no avail.. The idea that the problem was to be found here was shoved aside rather quickly.

• Take it out and insert it into a bay:

Leaving the idea of cabling errors, in came the docking bay. Thanks to [shameful commercial]Coolblue’s quick orders[/shameful commercial] I managed to get a Sharkoon dual bay the next day already. Hooked it up, and I almost jumped up and down as a small kid (I have a perfect example of that running around here, half of the time), plugged in the disk, connected it to the PC and booted.. That childishly enthusiasm quickly dissipated when the appropriate “dmesg” output simply.. didn’t appear.. nada, nothing, zilch.. Plugging in a different disk did work, so the bay was not at fault, fortunately). Off to the next option then..

• Freeze the bastard:

While searching for other methods, I came across the following site:
Emerging Techs – Put Your Hard Drive in the Freezer to Recover Data
After reading through various enthusiastic successes, I decided to attempt this as well. You never know what miracle crosses your way. So MrDisk was put into the plastic bag, carefully closed and into the freezer for a night. With the same enthusiasm as before I took it out the next day and put it into the bay for testing. The disk spun up again, although much slower than before, but as time passed, and it became warmer, this got better.

Unfortunately, this post doesn’t come with a happy ending.. This didn’t fix my disk either, although many stories seemed to suggest to give it at least a shot. Fortunately there’s still warranty on the disk so it’ll be prepped for RMA.
Fortunately most media, and some code, have been carefully saved already, unfortunately a few things have not been..
But this is a good wake-up call for an important lesson : “Better backups!”

Hello world!

Helloo everyone.. I’ ve entered the blooogging world as well..

Let’s not hope too much from this, or even think that this will even contribute to the world. Venting ideas and thoughts would rather be enough already. For the lonely minds who are mindlessly wandering and clicking around on the web, and stumbling on this site by accident.

(Or you are a friend and are interested in what I have to say/mumbojumbo and want to waste your time reading this, of course).

Enjoy, for what it’s worth and see you sometime soon.

Cheers,

Chris.