Clean up maildir folders with python

Time to clean up some older mails for certain folders in my maildir directory, I thought. It didn’t need to be something fancy, just that I wanted to put different requirements for days to keep per folder. First I thought I’d go looking in the maildir and look per file for file-timestamps and clean up directly from there, but Python has the ‘mailbox’ module for dealling with mailboxes in mbox/maildir formats.

So after throwing some code together, here’s a basic result.

#!/usr/bin/env python3
# Clean up script for certain old mail folders

import mailbox
import os
import time

# Maildir main folder
mailpath = os.environ.get('HOME') + '/Mail/'

# List of tuples with (Maildir subfolder, number of days to keep)
cleanup_paths = [
    ('0_MAYBE_SPAM', 3),
    ('0_SPAM', 3),
    ('Trash', 3),
    ('Internet.CR-Net_Log', 7),
    ('CR-Net.Promotions', 31),
]

def main():
    now = time.time()
    print("Cleaning up old mails...")
    maildir = mailbox.Maildir(mailpath)
    for (cl_path, keep_days) in cleanup_paths:
        print("*** Processing {} discarding older than {} days".format(cl_path, keep_days))
        keep_time = now - keep_days * 86400
        cleanfolder = maildir.get_folder(cl_path)
        for filename, msg in cleanfolder.items():
            # If epoch timestamp of the msg is smaller than keep_time epoch timestamp
            if msg.get_date() < keep_time:
                print("Filename:", filename, "\nSubject:", msg['subject'], "\nDate", msg['date'], "\n----------------------------------")
                cleanfolder.remove(filename)
    maildir.close()

if __name__ == '__main__':
    main()

You can also find the code at Bitbucket and Github

Playing with KVM virtualisation – part 1.1

Looks like I bumped into a small issue when updating one of the VMs (Debian) I was playing with: Shortage of diskspace on the root volume. Disaster when running dist-upgrade!

# df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/vm2-root    1.5G  1.4G     0 100% /
udev                     10M     0   10M   0% /dev
tmpfs                    25M  224K   25M   1% /run
tmpfs                   5.0M     0  5.0M   0% /run/lock
tmpfs                    49M     0   49M   0% /run/shm
/dev/vda1               228M   58M  158M  27% /boot
/dev/mapper/vm2-home    2.1G  3.6M  2.0G   1% /home

Fortunately, it’s an LVM so it’s not so difficult to extend this, even on-the-fly.

Let’s create our new disk first

virsh# vol-create-as MyVolume vm2-2.img 2G --format raw
Vol vm2-2.img created

virsh # vol-list --details MyVolume
Name                 Path                                  Type   Capacity  Allocation
--------------------------------------------------------------------------------------
Android-x86-4.0.img  /virtualmachines/Android-x86-4.0.img  file   4,00 GiB  498,75 MiB
cvm3.img             /virtualmachines/cvm3.img             file  16,00 GiB   61,81 MiB
vm2-2.img            /virtualmachines/vm2-2.img            file   2,00 GiB    2,00 GiB
vm2.img              /virtualmachines/vm2.img              file   4,00 GiB    1,83 GiB

Now that we have created the volume, let’s attach it to the VM. It is possible while it’s running live.

virsh # list
 Id    Name                           State
----------------------------------------------------
 4     vm2                            running

virsh # attach-disk vm2 /virtualmachines/vm2-2.img vdc
Disk attached successfully

Within the VM it’s automatically detected as well.

vm2# tail /var/log/kern.log
...
Dec 21 11:58:00 vm2 kernel: [ 3978.289939]  vdc: unknown partition table

The new disk is there, but it has not been partitioned yet. So let’s do that

root@vm2:/# fdisk /dev/vdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xc9f8eafe.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): m
Command action
a toggle a bootable flag
b edit bsd disklabel
c toggle the dos compatibility flag
d delete a partition
l list known partition types
m print this menu
n add a new partition
o create a new empty DOS partition table
p print the partition table
q quit without saving changes
s create a new empty Sun disklabel
t change a partition's system id
u change display/entry units
v verify the partition table
w write table to disk and exit
x extra functionality (experts only)

Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-4095999, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-4095999, default 4095999):
Using default value 4095999

Command (m for help): t
Selected partition 1

Hex code (type L to list codes): 8e

8e is the code for Linux LVM

Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Done! Now we need to add it to our LVM. First let’s check the volume group (vm2 here).

# vgdisplay
  --- Volume group ---
  VG Name               vm2
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               3.76 GiB
  PE Size               4.00 MiB
  Total PE              962
  Alloc PE / Size       962 / 3.76 GiB
  Free  PE / Size       0 / 0   
  VG UUID               ksC1ha-y05g-E6gx-JshG-b09N-dw6q-kcF2t7

As you can see, no Free space left (Free PE / Size). We are ready to add our new disk and its freshly created partition

# vgextend vm2 /dev/vdb1
No physical volume label read from /dev/vdb1
Physical volume "/dev/vdb1" successfully created
Volume group "vm2" successfully extended

# vgdisplay
  --- Volume group ---
  VG Name               vm2
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  6
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               5.71 GiB
  PE Size               4.00 MiB
  Total PE              1461
  Alloc PE / Size       1218 / 4.76 GiB
  Free  PE / Size       243 / 972.00 MiB
  VG UUID               ksC1ha-y05g-E6gx-JshG-b09N-dw6q-kcF2t7

Great! The new space has been allocated (VG Size, Free PE / Size). Now we are ready to exptend the root volume

# lvextend -L+1G /dev/vm2/root
Extending logical volume root to 2.47 GiB
Logical volume root successfully resized
# df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/vm2-root    2.5G  1.4G  945M  60% /
udev                     10M     0   10M   0% /dev
tmpfs                    25M  228K   25M   1% /run
tmpfs                   5.0M     0  5.0M   0% /run/lock
tmpfs                    49M     0   49M   0% /run/shm
/dev/vda1               228M   58M  158M  27% /boot
/dev/mapper/vm2-home    2.1G  3.6M  2.0G   1% /home

Just extending the logical volume does not automatically add the free space to the filesystem, that needs to be resized as well.

# resize2fs -p /dev/vm2/root 
resize2fs 1.42.8 (20-Jun-2013)
Filesystem at /dev/vm2/root is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/vm2/root is now 648192 blocks long.

root@vm2:/dev# df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/vm2-root    2.5G  1.4G  945M  60% /
udev                     10M     0   10M   0% /dev
tmpfs                    25M  224K   25M   1% /run
tmpfs                   5.0M     0  5.0M   0% /run/lock
tmpfs                    49M     0   49M   0% /run/shm
/dev/vda1               228M   58M  158M  27% /boot
/dev/mapper/vm2-home    2.1G  3.6M  2.0G   1% /home

Great success! One last look to the volume group stats shows that not all space has been allocated yet.

# vgdisplay
  --- Volume group ---
  VG Name               vm2
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  6
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               5.71 GiB
  PE Size               4.00 MiB
  Total PE              1461
  Alloc PE / Size       1218 / 4.76 GiB
  Free  PE / Size       243 / 972.00 MiB
  VG UUID               ksC1ha-y05g-E6gx-JshG-b09N-dw6q-kcF2t7

In case we need some tweaking to other logical volumes, it can be added accordingly.

Hello world!

Helloo everyone.. I’ ve entered the blooogging world as well..

Let’s not hope too much from this, or even think that this will even contribute to the world. Venting ideas and thoughts would rather be enough already. For the lonely minds who are mindlessly wandering and clicking around on the web, and stumbling on this site by accident.

(Or you are a friend and are interested in what I have to say/mumbojumbo and want to waste your time reading this, of course).

Enjoy, for what it’s worth and see you sometime soon.

Cheers,

Chris.