Dovecot: Pigeonhole sieve-filter refilter delivered email
After adjusting your sieve rules or adding them for the first time on an existing mailbox you will probably want to ‘refilter’ any existing delivered mail so they are moved to the correct IMAP folders. The dovecot wiki suggests to redeliver the mail to yourself to run it though the new filters, the method is outlined here wiki2.dovecot.org/HowTo/RefilterMail. There is another way of doing the same thing which is far less messy, using an inbuilt tool, sieve-filter.
Introducing ‘sieve-filter’
As of dovecot pigeonhole v0.3 a new script, sieve-filter, has been added to do just this. You simply point it at your sieve script and a source mailbox and it will do the rest. Before we look into how to use this script, I am first going to reiterate a section of the man page.
From sieve-filter(1) man page
CAUTION Although this is a very useful tool, it can also be very destructive when used improperly. A small bug in your Sieve script in combination with the wrong command line options could cause it to discard the wrong e-mails. And, even if the source-mailbox is opened in read-only mode to prevent such mishaps, it can still litter other mailboxes with spurious copies of your e-mails if your Sieve script decides to do so. There- fore, users are advised to read this manual carefully and to use the simulation mode first to check what the script will do. And, of course: MAKING A BACKUP IS IMPERATIVE FOR ANY IMPORTANT MAIL!
So please be careful you could really mess up you mailbox. I suggest you setup a test mailbox with some basic sieve rules, send some mail to it and test it without putting your real mailbox at risk.
Setting up the test environment
First things first, SSH to you mail server, check you have pigeonhole v0.3 or newer installed and that the script sieve-filter is available. If you are new to dovecot sieve you can find in the FreeBSD ports tree at mail/dovecot2-pigeonhole. You will, of course, need to be using dovecot v2 or newer also, the port can be found at mail/dovecot2.
The mail server in these examples is using dovecot-2.1.10 and dovecot-pigeonhole-0.3.3. Sieve is configured to store the rules in a text file in the mailbox root, Maildir/sieve/rules.sieve. If you are new to configuring sieve rules I recommend installing roundcube webmail and enable the sieve plugin, this makes managing your rules very easy.
Create a test mailbox
For these examples we will be using a mailbox called sieve@example.com with a username of sieve. The mail directory will be located in /var/mail/example.com/sieve/Maildir. An IMAP directory has been created called ‘sieve-test’ and a simple sieve rule has been added to match a subject of ‘move to folder sieve-test’ and move the email to this IMAP folder.
require ["fileinto"]; # rule:[sieve-test] if header :contains "Subject" "move to folder sieve-test" { fileinto "INBOX.sieve-test"; stop; }
Now the mailbox is configured we will send the following test emails to it.
Email #1
Subject: land in inbox Body: Test email to land in Inbox.
Email #2
Subject: move to folder sieve-test Body: Test email to land in sieve-test folder.
Email #3
Subject: this will be refiltered Body: Test email to be refiltered post delivery.
Email #4
Subject: this is spam Body: Test email spam to be deleted.
After these test emails have been delivered we can see email #1 has landed in the inbox, email #2 matched the sieve rule and was moved the folder ‘sieve-test’. Email #3 and #4 have landed in inbox as expected, no filter rule currently exists.
Adding a new sieve rule
To test refiltering delivered mail we will add a new sieve rule. At this point we will only add the rule to match email #3 with subject ‘this will be refiltered’, we will cover removing the test spam email later on. This is what our sieve rules file now looks like.
require ["fileinto"]; # rule:[sieve-test] if header :contains "Subject" "move to folder sieve-test" { fileinto "INBOX.sieve-test"; stop; } # rule:[refilter] if header :contains "Subject" "this will be refiltered" { fileinto "INBOX.sieve-test"; stop; }
Running sieve-filter for the first time
Now that we have something to refilter in our mailbox we can run sieve-filter against it. By default sieve-filter runs in read only mode, for obvious safety reasons. We will not change this default behaviour to begin with, as we want to make sure it’s going to do what we expect.
First we’ll look at the options we need to give to sieve-filter and break down what they do.
sieve-filter usage output
# sieve-filter Usage: sieve-filter [-c <config-file>] [-C] [-D] [-e] [-m <default-mailbox>] [-P <plugin>] [-q <output-mailbox>] [-Q <mail-command>] [-s <script-file>] [-u <user>] [-v] [-W] [-x <extensions>] <script-file> <source-mailbox> [<discard-action>]
Option: -u
-u user Run the Sieve script for the given user.
This is option specifies which dovecot user’s mailbox you are refiltering, this must match the login name of the IMAP account.
Option: -C
-C Force compilation. By default, the compiled binary is stored on disk. When this binary is found during the next execution of sieve-filter and its modification time is more recent than the script file, it is used and the script is not compiled again. This option forces the script to be compiled, thus ignoring any present binary. Refer to sievec(1) for more information about Sieve compilation.
When sieve-filter is runs it uses a complied version of the sieve rule set, this is done for speed. To ensure the sieve rule set is always recompliled before the refilter we use this option.
Option: -v
-v Produce verbose output during filtering.
This option is self explanatory, we use to make sure we see everything that’s going on. I recommend piping the output to a file, in-case something does go wrong, you will have something to go on.
Option: -e *Warning*
-e Turns on execution mode. By default, the sieve-filter command runs in simulation mode in which it changes nothing, meaning that no mailbox is altered in any way and no actions are per- formed. It only prints what would be done. Using this option, the sieve-filter command becomes active and performs the requested actions.
When we are ready to run sieve-filter for real and actually make changes to our mailbox we will use this option.
Option: -W *Warning*
-W Enables write access to the source-mailbox. This allows (re)mov- ing the messages from the source-mailbox, changing their con- tents, and changing the assigned IMAP flags and keywords.
When using the ‘-e’ option, refiltered mails will be copied to the new destination, but with out this option they wont be removed from the source mailbox. If you don’t use this option you will find yourself with a lot of duplicate email.
Argument: <script-file>
script-file Specifies the Sieve script to (compile and) execute. Note that this tool looks for a pre-compiled binary file with a .svbin extension and with basename and path identical to the specified script. Use the -C option to disable this behavior by forcing the script to be compiled into a new binary.
The path to your sieve rule set.
Argument: <source-mailbox>
source-mailbox Specifies the source mailbox containing the messages that the Sieve filter will act upon. This is the name of a mailbox, as visible to IMAP clients, except in UTF-8 format. The hierarchy separator between a parent and child mailbox is commonly '/' or '.', but this depends on your selected mailbox storage format and namespace configura- tion. The mailbox names may also require a namespace prefix. This mailbox is not modified unless the -W option is specified.
The name of the source mailbox, as the man page extract says, it is the mailbox as it is visable to IMAP clients *Not* the full path to the mail directory on your server. E.g. ‘INBOX’ or ‘INBOX.folder’, *not* ‘/var/mail/example.com/sieve/Maildir/cur’.
Constructing the command and dry run
It’s finally time to run our first refilter test! We will be doing a dry run first, let’s have a look at what this looks like.
# sieve-filter -v -C -u sieve /var/mail/example.com/sieve/Maildir/sieve/rules.sieve 'INBOX' >> Filtering message: ID: <507EBEC5.8060608@example.com> Date: Wed, 17 Oct 2012 15:20:02 +0100 Size: 2702 bytes Subject: land in inbox Performed actions: (none) Implicit keep: * store message in folder: INBOX >> Filtering message: ID: <507EBEC5.8060608@example.com> Date: Wed, 17 Oct 2012 15:20:53 +0100 Size: 1765 bytes Subject: this will be refiltered Performed actions: * store message in folder: INBOX.sieve-test Implicit keep: (none) >> Filtering message: ID: <507EBEDC.4000401@example.com> Date: Wed, 17 Oct 2012 15:21:16 +0100 Size: 1753 bytes Subject: this is spam Performed actions: (none) Implicit keep: * store message in folder: INBOX
As we can see from the output, this would of worked as expected. If we look at the ‘Performed actions’ section for the mail with subject ‘this will be refiltered’ we can see that it would of been moved to INBOX.sieve-test. The other 2 emails, which have no matching rules, have no ‘Performed actions’ and instead have an ‘Implicit keep’ which leaves them in INBOX.
At this point we could safely add the ‘-e’ and ‘-W’ options to actually perform the move.
# sieve-filter -e -W -v -C -u sieve /var/mail/example.com/sieve/Maildir/sieve/rules.sieve 'INBOX' info: filtering: [Wed, 17 Oct 2012 15:20:02 +0100; 2702 bytes] `land in inbox'. info: msgid=<507EBEC5.8060608@example.com>: left message in mailbox 'INBOX'. info: message kept in source mailbox. info: filtering: [Wed, 17 Oct 2012 15:20:53 +0100; 1765 bytes] `this will be refiltered'. info: msgid=<507EBEC5.8060608@example.com>: stored mail into mailbox 'INBOX.sieve-test'. info: message expunged from source mailbox upon successful move. info: filtering: [Wed, 17 Oct 2012 15:21:16 +0100; 1753 bytes] `this is spam'. info: msgid=<507EBEDC.4000401@example.com>: left message in mailbox 'INBOX'. info: message kept in source mailbox.
Check your mailbox, and we can see the mail ‘this will be refiltered’ has been removed from INBOX and moved into INBOX.sieve-test.
Discarding email
Rememeber that lonely test spam email sitting in our inbox? It’s now time to get rid of it. Let’s add a new rule to our sieve rule set to discard it.
require ["fileinto"]; # rule:[sieve-test] if header :contains "Subject" "move to folder sieve-test" { fileinto "INBOX.sieve-test"; stop; } # rule:[refilter] if header :contains "Subject" "this will be refiltered" { fileinto "INBOX.sieve-test"; stop; } # rule:[spam] if header :contains "Subject" "this is spam" { discard; stop; }
Dry run on the mailbox to test the new rule:
# sieve-filter -v -C -u sieve /var/mail/example.com/sieve/Maildir/sieve/rules.sieve 'INBOX' keep >> Filtering message: ID: <507EBEC5.8060608@example.com> Date: Wed, 17 Oct 2012 15:20:02 +0100 Size: 2702 bytes Subject: land in inbox Performed actions: (none) Implicit keep: * store message in folder: INBOX >> Filtering message: ID: <507EBEDC.4000401@example.com> Date: Wed, 17 Oct 2012 15:21:16 +0100 Size: 1753 bytes Subject: this is spam Performed actions: * discard Implicit keep: (none)
The only change we need to make to our previous sieve-filter command is to add a discard action to the end. We have a number of different actions we can apply to discared mails, the default action is ‘keep’, if you don’t specify a discard action ‘keep’ is implicitly implied. When ‘keep’ is used the mail is marked to be descarded but it is not actually removed, as we can see in the last 2 lines of this live run.
# sieve-filter -e -W -v -C -u sieve /var/mail/example.com/sieve/Maildir/sieve/rules.sieve 'INBOX' keep info: filtering: [Wed, 17 Oct 2012 15:20:02 +0100; 2702 bytes] `land in inbox'. info: msgid=<507EBEC5.8060608@example.com>: left message in mailbox 'INBOX'. info: message kept in source mailbox. info: filtering: [Wed, 17 Oct 2012 15:21:16 +0100; 1753 bytes] `this is spam'. info: msgid=<507EBEDC.4000401@example.com>: marked message to be discarded if not explicitly delivered (discard action). info: message left in source mailbox.
So what options do we have when discarding mails, this is what the man page says about it.
Argument: <discard-action>
discard-action Specifies what is done with messages in the source-mailbox that where not kept or otherwise stored by the Sieve script; i.e. those messages that would normally be discarded if the Sieve script were executed at delivery. The discard-action parameter accepts one of the following values: keep (default) Keep discarded messages in source mailbox. move mailbox Move discarded messages to the indicated mailbox. This is for instance useful to move messages to a Trash mailbox. Refer to the explanation of the source-mailbox argument for more information on mailbox naming. delete Flag discarded messages as \DELETED. expunge Expunge discarded messages, meaning that these are removed irreversibly when the tool finishes filtering. When the -W option is not specified, the source-mailbox is immutable and the specified discard-action has no effect. This means that messages are at most copied to a new location. In contrast, when the -W is specified, messages that are success- fully stored somewhere else by the Sieve script are always expunged from the source-mailbox, with the effect that these are thus moved to the new location. This happens irrespective of the specified discard-action. Remember: only discarded messages are affected by the specified discard-action.
So our options are keep, move, delete and expunge. Let’s see some examples.
Discard action: move
In this example we are moving our discared messages to our Trash folder, INBOX.Trash.
# sieve-filter -e -W -v -C -u sieve /var/mail/example.com/sieve/Maildir/sieve/rules.sieve 'INBOX' move INBOX.Trash info: filtering: [Wed, 17 Oct 2012 15:20:02 +0100; 2702 bytes] `land in inbox'. info: msgid=<507EBEC5.8060608@example.com>: left message in mailbox 'INBOX'. info: message kept in source mailbox. info: filtering: [Wed, 17 Oct 2012 15:21:16 +0100; 1753 bytes] `this is spam'. info: msgid=<507EBEDC.4000401@example.com>: marked message to be discarded if not explicitly delivered (discard action). info: message in source mailbox moved to mailbox 'Trash'.
Discard action: delete
In this example we are marking the email as \DELETED.
# sieve-filter -e -W -v -C -u sieve /var/mail/example.com/sieve/Maildir/sieve/rules.sieve 'INBOX' delete info: filtering: [Wed, 17 Oct 2012 15:20:02 +0100; 2702 bytes] `land in inbox'. info: msgid=<507EBEC5.8060608@example.com>: left message in mailbox 'INBOX'. info: message kept in source mailbox. info: filtering: [Wed, 17 Oct 2012 16:06:28 +0100; 1755 bytes] `this is spam'. info: msgid=<507EC974.8090309@example.com>: marked message to be discarded if not explicitly delivered (discard action). info: message flagged as deleted in source mailbox.
To show is has worked, we can telnet on to our IMAP server and look at the flags.
# telnet localhost 143 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN AUTH=LOGIN AUTH=DIGEST-MD5 AUTH=CRAM-MD5] MailServer ready. a login sieve password a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE QUOTA] Logged in a1 select INBOX * FLAGS (\Answered \Flagged \Deleted \Seen \Draft) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted. * 2 EXISTS * 0 RECENT * OK [UIDVALIDITY 1349723219] UIDs valid * OK [UIDNEXT 6] Predicted next UID * OK [NOMODSEQ] No permanent modsequences a1 OK [READ-WRITE] Select completed. a2 fetch 2 full * 2 FETCH (FLAGS (\Deleted \Seen) INTERNALDATE "17-Oct-2012 16:06:35 +0100" RFC822.SIZE 1755 ENVELOPE ("Wed, 17 Oct 2012 16:06:28 +0100" "this is spam" (("Jake Smith" NIL "jake" "example.com")) (("Jake Smith" NIL "jake" "example.com")) (("Jake Smith" NIL "jake" "example.com")) ((NIL NIL "sieve" "example.com")) NIL NIL NIL "<507EC974.8090309@example.com>") BODY ("text" "plain" ("charset" "ISO-8859-1") NIL NIL "7bit" 14 1)) a2 OK Fetch completed. a3 logout * BYE Logging out a3 OK Logout completed. Connection closed by foreign host.
Discard action: expunge
In this example we will irreversibly remove the mail.
# sieve-filter -e -W -v -C -u sieve /var/mail/example.com/sieve/Maildir/sieve/rules.sieve 'INBOX' expunge info: filtering: [Wed, 17 Oct 2012 15:20:02 +0100; 2702 bytes] `land in inbox'. info: msgid=<507EBEC5.8060608@example.com>: left message in mailbox 'INBOX'. info: message kept in source mailbox. info: filtering: [Wed, 17 Oct 2012 16:06:28 +0100; 1755 bytes] `this is spam'. info: msgid=<507EC974.8090309@example.com>: marked message to be discarded if not explicitly delivered (discard action). info: message expunged from source mailbox.
N.B.
On a final note, sieve-filter can run out of memeory when running in live mode with very large mailboxes. The delete from source directory ‘-W’ does not happen until all mail has been copied to the new desination folders, a crash like this will leave you with duplicate email. Be careful!
- Share me!
- Tweet
- Tumblr
- Stumble
- Digg
- Delicious
FreeBSD: Crontab run every second now in HEAD
Last night a new functionality and cron shortcut was added to the FreeBSD crontab in HEAD, “@every_second”.
It does what it says on the tin, runs a cron entry once a second. A feature which I’m sure will be welcomed by many. Currently with the latest commit you can only run once a second, no support for things like every 10 seconds, every 30 seconds, etc but I with any luck this functionality will be added soon.
See sobomax’s commit log
r241576 | sobomax | 2012-10-15 09:21:49 +0100 (Mon, 15 Oct 2012) | 16 lines Changed paths: M /head/usr.sbin/cron/cron/cron.c M /head/usr.sbin/cron/cron/cron.h M /head/usr.sbin/cron/crontab/crontab.5 M /head/usr.sbin/cron/lib/entry.c Add per-second scheduling into the cron(8). Right now it's only available via the new @every_second shortcut. ENOTIME to implement crontab(5) format extensions to allow more flexible scheduling. In order to address some concerns expressed by Terry Lambert while discussing the topic few years ago, about per-second cron possibly causing some bad effects on /etc/crontab by stat()ing it every second instead of every minute now (i.e. atime update), only check that database needs to be reloaded on every 60-th loop run. This should be close enough to the current behaviour. Add "@every_minute" shortcut while I am here. MFC after: 1 month
Thanks to everyone involved! As you can see from the commit log “@every_minute” is now also available.
Example crontab entry
@every_second root echo "hello, world"
My initial tests are working well, if you want to give it a go check out the latest HEAD from svn, build and get testing!
- Share me!
- Tweet
- Tumblr
- Stumble
- Digg
- Delicious
Equation: Broadcast address from CIDR – PHP/MySQL
When working with subnets in CIDR (Classless Inter-Domain Routing) notation, for example 10.0.0.0/8, you may find yourself needing to know the broadcast address. Normally I would use ipcalc, which can be found in net-mgmt/ipcalc within the FreeBSD ports tree. However it is not always conviniant to use a 3rd party program, escpecially when needing to do the conversion inside some code.
So the awnser, a simple equation to get the broadcast address from the network address and cidr.
BROADCAST = NETWORK + (2^(32-CIDR)) - 1
The broadcast address is the network IP, plus, 32 minus the CIDR, to the power of 2, minus 1.
An example of this, find the broadcast address from 192.168.32.64/26
192.168.32.64 + (2^(32-26)) - 1 = 192.168.32.127
So, the broadcast address is 192.168.32.127. There is one problem with this example. How is it I can use an IP address as if it were an integer? Before this equation can be executed we must first convert the IP address into an integer. There are many ways to achieve this, in MySQL you would use the INET_ATON() function, in PHP you would use ip2long(). Lets look at the previous example again, but this time lets use an integer representation of the IP address.
Find the broadcast address from 192.168.32.64/26, where the IP address is first converted to an integer
3232243776 + (2^(32-26)) - 1 = 3232243839
N.B. this conversion was done using a 64 bit CPU. On a 64bit CPU, 192.168.32.64 = 323224377 and on a 32bit CPU, 192.168.32.64 = -1062723457. This make no differance to the equation, it will work on both 32 and 64 bit architectures.
Converting these integers back to an IP address is simple, for MySQL we use INET_NTOA() and for PHP we use long2ip().
Lets check our work
long2ip(3232243776) = 192.168.32.64 // Network long2ip(3232243839) = 192.168.32.127 // Broadcast
Here is an example SQL query to get the broadcast from CIDR
mysql> SELECT INET_NTOA(INET_ATON('192.168.32.64') + (pow(2, (32-26))-1)); +-------------------------------------------------------------+ | INET_NTOA(INET_ATON('192.168.32.64') + (pow(2, (32-26))-1)) | +-------------------------------------------------------------+ | 192.168.32.127 | +-------------------------------------------------------------+ 1 row in set (0.00 sec)
Here is an example PHP function to get the broadcast from CIDR
function cidr2broadcast($network, $cidr) { $broadcast = long2ip(ip2long($network) + pow(2, (32 + $cidr)) - 1); return $broadcast; } // 192.168.0.0/20 $network = "192.168.0.0"; $cidr = 20; print(cidr2broadcast($network, $cidr));
- Share me!
- Tweet
- Tumblr
- Stumble
- Digg
- Delicious
FreeBSD: 102 Tips and Tricks
These tips will look familiar to any one who has games/fortune in their MOTD, because indeed this is where they’re from. These tips are well worth reading, even for an experienced FreeBSD user. For example, how many users know about the “look” command, I for one am enlightened!
FreeBSD Tip: #1
Any user that is a member of the wheel group can use "su -" to simulate a root login. You can add a user to the wheel group by editing /etc/group. -- Konstantinos Konstantinidis <kkonstan at duth.gr>
FreeBSD Tip: #2
By pressing "Scroll Lock" you can use the arrow keys to scroll backward through the console output. Press "Scroll Lock" again to turn it off.
FreeBSD Tip: #3
Can't remember if you've installed a certain port or not? Try "pkg_info -Ix port_name".
FreeBSD Tip: #4
Ever wonder what those numbers after command names were, as in cat(1)? It's the section of the manual the man page is in. "man man" will tell you more. -- David Scheidt <dscheidt at tumbolia.com>
FreeBSD Tip: #5
Forget how to spell a word or a variation of a word? Use look portion_of_word_you_know -- Dru <genesis at istar.ca>
FreeBSD Tip: #6
Forget what directory you are in? Type "pwd". -- Dru <genesis at istar.ca>
FreeBSD Tip: #7
Forget when Easter is? Try "ncal -e". If you need the date for Orthodox Easter, use "ncal -o" instead. -- Dru <genesis at istar.ca>
FreeBSD Tip: #8
FreeBSD is started up by the program 'init'. The first thing init does when starting multiuser mode (ie, starting the computer up for normal use) is to run the shell script /etc/rc. By reading /etc/rc and the /etc/rc.d/ scripts, you can learn a lot about how the system is put together, which again will make you more confident about what happens when you do something with it.
FreeBSD Tip: #9
Handy bash(1) prompt: PS1="\u at \h \w \!$ " -- David Scheidt <dscheidt at tumbolia.com>
FreeBSD Tip: #10
Having trouble using fetch through a firewall? Try setting the environment variable FTP_PASSIVE_MODE to yes, and see fetch(3) for more details.
FreeBSD Tip: #11
If other operating systems have damaged your Master Boot Record, you can reinstall it either with /usr/sbin/sysinstall or with boot0cfg(8). See "man boot0cfg" for details.
FreeBSD Tip: #12
If you accidentally end up inside vi, you can quit it by pressing Escape, colon (:), q (q), bang (!) and pressing return.
FreeBSD Tip: #13
If you are in the C shell and have just installed a new program, you won't be able to run it unless you first type "rehash". -- Dru <genesis at istar.ca>
FreeBSD Tip: #14
If you do not want to get beeps in X11 (X Windows), you can turn them off with xset b off