Clean up maildir folders with python

Time to clean up some older mails for certain folders in my maildir directory, I thought. It didn’t need to be something fancy, just that I wanted to put different requirements for days to keep per folder. First I thought I’d go looking in the maildir and look per file for file-timestamps and clean up directly from there, but Python has the ‘mailbox’ module for dealling with mailboxes in mbox/maildir formats.

So after throwing some code together, here’s a basic result.

#!/usr/bin/env python3
# Clean up script for certain old mail folders

import mailbox
import os
import time

# Maildir main folder
mailpath = os.environ.get('HOME') + '/Mail/'

# List of tuples with (Maildir subfolder, number of days to keep)
cleanup_paths = [
    ('0_MAYBE_SPAM', 3),
    ('0_SPAM', 3),
    ('Trash', 3),
    ('Internet.CR-Net_Log', 7),
    ('CR-Net.Promotions', 31),
]

def main():
    now = time.time()
    print("Cleaning up old mails...")
    maildir = mailbox.Maildir(mailpath)
    for (cl_path, keep_days) in cleanup_paths:
        print("*** Processing {} discarding older than {} days".format(cl_path, keep_days))
        keep_time = now - keep_days * 86400
        cleanfolder = maildir.get_folder(cl_path)
        for filename, msg in cleanfolder.items():
            # If epoch timestamp of the msg is smaller than keep_time epoch timestamp
            if msg.get_date() < keep_time:
                print("Filename:", filename, "\nSubject:", msg['subject'], "\nDate", msg['date'], "\n----------------------------------")
                cleanfolder.remove(filename)
    maildir.close()

if __name__ == '__main__':
    main()

You can also find the code at Bitbucket and Github

Advertisements

Playing with KVM virtualisation

When I attended Loadays 2013 back in the beginning of April, I was introduced to Ansible, an interesting configuration management system. In order to properly play with it I wanted to have a couple of hosts that could be messed with. Virtual machines are perfect for that. For the exercise I opted for KVM. No particular reason, other options are just as fine for this. This is also mainly a beginning reference for myself, so the setup can probably be handled in a much better and efficient way (don’t hesitate to tell me, of course).

For the impatient, the source files as example below can be found in this Bitbucket repository

Prerequisites

Create bridge interface

First of all, in order to have different IP addresses assigned, and you want to have the VMs connect directly on your physical network, you can create a bridge interface br0, which all the VMs can use for networking. In the example scripts below, this way is used.

A good starting point for more information on setting this up on Debian can be found on the Debian Wiki BridgeNetworkConnections page.

LibVirt

For this setup, we’re using libVirt

Webserver to host files

For testing, I installed a simple Apache Webserver which hosted the files in its root directory. YMMV (which is also recommended)

Create virtual machines

Of course, this can all be done through the virt-manager GUI as well, but when working remote, without an X-server or VNC at your disposal, it’s handy to know some command-line as well. And to reduce typing the same thing over and over (although it’s not bad to get the commands in your muscle memory), let’s put it in a small script. (Inspiration was found here)

Creation scripts

Note: The example scripts are for testing purposes, and are meant to generate a small image. Your usage requirements will most probably differ.

#!/bin/sh
#
# A script to install Debian Wheezy on a KVM guest
# using automatic installation from installable distribution
# image on an HTTP server

# Defined MAC range 52:54:00:8E:FF:00 - 52:54:00:8E:FF:FF
# Choose the last number to your own preference
 
if [ $# -ne 1 ]
then
echo "Usage: $0 guest-name"
exit 1
fi

size=2
CPUs=2
RAM=512
MyVMMACRANGE='52:54:00:8e:ff'
VMMAC=${MyVMMACRANGE}:00
domain=home
wwwhost=http://example.com
 
virt-install \
--connect=qemu:///system \
--virt-type kvm \
--name=${1} \
--ram=${RAM} \
--vcpus=${CPUs} \
--disk path=/var/lib/libvirt/images/${1}.img,size=${size} \
-l http://ftp.be.debian.org/debian/dists/wheezy/main/installer-amd64/ \
--vnc \
--os-variant debianwheezy \
--noautoconsole \
--hvm \
--network bridge=br0,mac=${VMMAC} \
--extra-args="auto=true hostname=${1} domain=${domain} url=${wwwhost}/preseed-debian_vm.txt text console=tty1 console=ttyS0,115200"

Call this script like this

# create_vm-debianwheezy.sh vm1

and it will result in a call to libvirt to create a new virtual machine, with 2 CPUs, 512MB RAM and 2 GB image on disk. 1 CPU and 256MB also work fine, you can easily define that in the script (or later in the generated xml config file for the VM, located at /etc/libvirt/qemu/vm1.xml). It also has a console that you can attach to from within virsh.

To see the progress of the installation, type

# virsh console vm1

The -l parameter in the script tells us to connect to the Debian mirror to fetch the online installation image for Wheezy. It also adds some kernel parameters with –extra-args which tells the installer to fetch a pre-configured preseed file from the wwwhost you defined above.

Headless installation files

An example preseed.txt file has been put in the www subdirectory of the repository. It will create a minimal install of Debian Wheezy, with a belgian mirror configured. Its filesystem is only 1 partition without LVM. No accessible root user is configured, but the pre-defined “myuser” has sudo rights for root permissions. Some useful (YMMV) packages are also automatically installed.

CentOS headless installation

I’ve also added VM-creation script for CentOS. There is a corresponding Kickstart file which gives a hardened CentOS image. The main source of inspiration was found in RackerHackers Github repository

VM fun

Listing the VMs
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 1     vm1                            running
 2     vm2                            shut off
Starting the VM

# virsh start vm1
Domain vm1 started

Stopping the VM

# virsh shutdown vm1
Domain vm1 is being shutdown

virsh commands and help
# virsh help 
... 
 Virsh itself (help keyword 'virsh'):
    cd                             change the current directory
    echo                           echo arguments
    exit                           quit this interactive terminal
    help                           print help
    pwd                            print the current directory
    quit                           quit this interactive terminal

Cloning fun

If you want to backup or clone your VM, you can use the virt-clone command.

# virt-clone -o vm1 -n vm2 -m 52:54:00:8E:FF:02 -f /var/lib/libvirt/images/vm2.img

A new VM will be created, exactly the same as the original image, except for the name, MAC address and image-file. Also note that the ssh-host keys will be identical, so they’d need to be regenerated.

The End

With enough tiny clones, you are ready for the other fun stuff: orchestration. Those first steps will probably be dealt with in a future blogpost about Ansible and/or Puppet

And to end, for those who want to go even further in this:

And most of all, have fun, if you feel something needs to be added or corrected, let me know!