Avoiding the "huge INBOX of death"

classic Classic list List threaded Threaded
5 messages Options
Raphaël Fournier-S'niehotta Raphaël Fournier-S'niehotta
Reply | Threaded
Open this post in threaded view
|

Avoiding the "huge INBOX of death"

Hello fellow notmuchers,

I am a longtime "classic" mutt user, willing to switch to a notmuch-based
solution (probably neomutt). However, as far as I understand, I will have to
deal with a "huge INBOX of death", as put by Anarcat in a recent blog post [1],
that when all emails remain in a single maildir, which may the become very
large.

I would like to avoid this, for quota reasons on my work imap server first, and
also because I sometimes have to use webmail access, with poor searching
capabilities.

Thus, I think I need to have my emails sorted into maildir folders according to
the notmuch tags. Namely, emails with the "inbox" tag should stay in the INBOX
maildir, those with "archive" should be moved to Archives (all synchronized with
the server). As a bonus, it would be great to have a few more features, like
"mails in Archives older than 6 months are moved to Archives/<year>" (kept
locally and not stored on the server).

As far as I understand, "afew" and its MailMover feature could be useful for my
case, but it does not seem currently maintained (plus, very few mentions of it
on the notmuch archives) and Michael Turquette said it's buggy [2], so I would
like more feedback to be sure before letting it mess with my maildir.

Another tool that could be useful is the "maildir-notmuch-sync" script written
by Ethan Schoonover and adapted by Michael Turquette [3]. However, it seems
dedicated to handling the synchronization of Gmail labels, not a generic imap
server hierarchy.

Perhaps there is another solution to deal with this in a notmuch-based setup, I
would very much like any of your feedback.

[1]: https://anarc.at/blog/2016-05-12-email-setup/
[2]: http://deferred.io/2016/01/17/state-of-the-dotfiles.html
[3]: http://deferred.io/2016/01/18/notmuch-tags-gmail-labels-bidirectional-sync.html

--
Raphaël Fournier-S'niehotta
http://raphael.fournier-sniehotta.fr/apropos

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
David Bremner-2 David Bremner-2
Reply | Threaded
Open this post in threaded view
|

Re: Avoiding the "huge INBOX of death"

Raphaël Fournier-S'niehotta <[hidden email]> writes:

> Hello fellow notmuchers,
>
> I am a longtime "classic" mutt user, willing to switch to a notmuch-based
> solution (probably neomutt). However, as far as I understand, I will have to
> deal with a "huge INBOX of death", as put by Anarcat in a recent blog post [1],
> that when all emails remain in a single maildir, which may the become very
> large.
>

TBH, I don't understand what specific problem anarcat is having, it
sounds like a workflow issue.  I currently have 1074 messages tagged
inbox, out of about 450K messages.

Notmuch doesn't care how you organize your mail files on disk, as long
as they are one message per file, and one tree of messages. So feel free
to store them e.g. by date using procmail if you are worried about
performance, or in some other set of folders with semantic

For performance I'd also say measure before you optimize. I currently
have about 77K messages in one of my maildirs, with no noticable
performance hit.  Of course you want e.g. directory indexing on ext3,
but that is default on anything recent.

> Thus, I think I need to have my emails sorted into maildir folders according to
> the notmuch tags. Namely, emails with the "inbox" tag should stay in the INBOX
> maildir, those with "archive" should be moved to Archives (all synchronized with
> the server). As a bonus, it would be great to have a few more features, like
> "mails in Archives older than 6 months are moved to Archives/<year>" (kept
> locally and not stored on the server).

Generally, you should search directly by folder, rather than trying to
sync tags and folders.

I'm not familiar with afew as a user.

d



_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch

signature.asc (660 bytes) Download Attachment
Vladimir Marek Vladimir Marek
Reply | Threaded
Open this post in threaded view
|

Re: Avoiding the "huge INBOX of death"

In reply to this post by Raphaël Fournier-S'niehotta
I can't speak for everyone, but here's what I am using.

I do have 'messages' directory which contains 'imap' and 'archive'.

messages
 \-imap
 \-archive

The imap directory is being synchronized via offlineimap to the server.

In the offlineimap postsync hook I call a script which does some
bookkeeping on the newly received messages and it also marks old and
already read and not flagged emails by the 'old-tread' tag.

N=notmuch
while : ; do
   AGE=100d # 100 days
   # Find all threads containing old messages
   OLD_MESSAGES=$( $N search --output=threads not tag:archive and date:..$AGE and not tag:unread and not tag:flagged )
   [ -n "$OLD_MESSAGES" ] || break
   # Find threads containing newer messages
   NEW_MESSAGES=$( $N search --output=threads not tag:archive and not '(' date:..$AGE or tag:unread or tag:flagged ')' )
   # remove the threads containing newer messages
   OLD_THREADS=$( /usr/xpg4/bin/fgrep -v -x -f <(echo "$NEW_MESSAGES") < <(echo "$OLD_MESSAGES") )
   [ -n "$OLD_THREADS" ] || break
   $N tag +old-thread -- $OLD_THREADS
   break
done

# If old thread received newer message, unmark it too
$N tag -old-thread -- tag:old-thread and '(' not date:..$AGE or tag:unread or tag:flagged or tag:archive ')'


Once in a while I review the old-thread messages and assign them by hand
'archive' flag. The postsync hook is looking for this flag and moves all
messages to be archived into archive folder. You can list files to be
archived by

$N search --output=files 'path:imap/**' tag:archive

I won't list the commands to move files from messages/imap to
messages/archive since I have a bit more complex scenario. But it should
be straightforward.

On next run offlineimap sees that the messages disappeared from 'imap'
folder and removes them from imap server.

$ notmuch count '*'
734346
$ notmuch count 'path:imap/**'
18419


In my case I am server side filtering the emails into several folders so
that I have at least some order in emails when reading email via phone,
so 'path:imap/**' contains all the messages in whole imap tree.

$ notmuch count 'path:imap/** and tag:inbox and tag:unread'
11

I'm trying to have my virtual INBOX empty most of the time.

I'm interested what tricks others use!
Cheers
--
        Vlad
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Olivier Schwander Olivier Schwander
Reply | Threaded
Open this post in threaded view
|

Re: Avoiding the "huge INBOX of death"

In reply to this post by Raphaël Fournier-S'niehotta
Le 19 Jul 2016 11:42, Raphaël Fournier-S'niehotta a écrit:

> Hello fellow notmuchers,
>
> I am a longtime "classic" mutt user, willing to switch to a notmuch-based
> solution (probably neomutt). However, as far as I understand, I will have to
> deal with a "huge INBOX of death", as put by Anarcat in a recent blog post [1],
> that when all emails remain in a single maildir, which may the become very
> large.
>
> I would like to avoid this, for quota reasons on my work imap server first, and
> also because I sometimes have to use webmail access, with poor searching
> capabilities.
>
> Thus, I think I need to have my emails sorted into maildir folders according to
> the notmuch tags. Namely, emails with the "inbox" tag should stay in the INBOX
> maildir, those with "archive" should be moved to Archives (all synchronized with
> the server). As a bonus, it would be great to have a few more features, like
> "mails in Archives older than 6 months are moved to Archives/<year>" (kept
> locally and not stored on the server).
>
> As far as I understand, "afew" and its MailMover feature could be useful for my
> case, but it does not seem currently maintained (plus, very few mentions of it
> on the notmuch archives) and Michael Turquette said it's buggy [2], so I would
> like more feedback to be sure before letting it mess with my maildir.
>
> Another tool that could be useful is the "maildir-notmuch-sync" script written
> by Ethan Schoonover and adapted by Michael Turquette [3]. However, it seems
> dedicated to handling the synchronization of Gmail labels, not a generic imap
> server hierarchy.
>
> Perhaps there is another solution to deal with this in a notmuch-based setup, I
> would very much like any of your feedback.
>
> [1]: https://anarc.at/blog/2016-05-12-email-setup/
> [2]: http://deferred.io/2016/01/17/state-of-the-dotfiles.html
> [3]: http://deferred.io/2016/01/18/notmuch-tags-gmail-labels-bidirectional-sync.html

You could use fdm [1] to store messages into a folder hierarchy based on
the date in the messages. In the past, I used something like that to
periodically move messages in my inbox to sub-folder with the year:

account "archives" disabled maildirs {
   "${mailpath}INBOX"
}
action "archive" maildir "${mailpath}%[maildir]/%[mail_year]"
match account "archives" {
    match age > 6 months action "archive"
    match all action keep
}

Nowadays, I use a custom script [2] to mirror some notmuch queries to
maildirs (ran periodically in my crontab), such like:

notmuch-imap.sh ~/Maildir/ NOT tag:archive AND NOT tag:killed AND NOT tag:spam AND tag:inbox
notmuch-imap.sh ~/Maildir/.todo NOT tag:archive AND NOT tag:killed AND NOT tag:spam AND tag:todo

Both solutions works server-side, since I control my own mail server but
it may probably used in an offlineimap hook.

Best,

Olivier

[1]: https://github.com/nicm/fdm
[2]: https://chadok.info/~oschwand/poubelle/e53f1d9ac30909fd17ca98f4160375bbc9a6aca5/notmuch-imap.sh.html

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Nicolás Reynolds-2 Nicolás Reynolds-2
Reply | Threaded
Open this post in threaded view
|

Re: Avoiding the "huge INBOX of death"

In reply to this post by Raphaël Fournier-S'niehotta
Raphaël Fournier-S'niehotta <[hidden email]> writes:

> Hello fellow notmuchers,
> Perhaps there is another solution to deal with this in a notmuch-based setup, I
> would very much like any of your feedback.

i made toomuch[^0] to archive email by year-month and it works really
well for me :P

[^0]: https://github.com/fauno/toomuch

--
http://utopia.partidopirata.com.ar/

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch

signature.asc (595 bytes) Download Attachment