How do you synchronize your notmuch tags across multiple machines?

classic Classic list List threaded Threaded
14 messages Options
Dan Čermák Dan Čermák
Reply | Threaded
Open this post in threaded view
|

How do you synchronize your notmuch tags across multiple machines?

Hi list,

first and foremost: thanks to everyone who contributed to notmuch, it's
a truly awesome piece of software.

Unfortunately, I am facing the problem (which I have been postponing to
face for a while) that I'll need to keep my notmuch database in sync
over multiple machines.
Thus my question: how are you achieving that? How well does your
solution work?

(I have found muchsync, but unfortunately very little reports about how
well it works, which isn't necessarily a bad thing. Beside that only
some scripts around notmuch dump.)


Thanks in advance,

Dan

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch

signature.asc (847 bytes) Download Attachment
Ralph Seichter Ralph Seichter
Reply | Threaded
Open this post in threaded view
|

Re: How do you synchronize your notmuch tags across multiple machines?

* Dan Čermák:

> I have found muchsync, but unfortunately very little reports about how
> well it works, which isn't necessarily a bad thing.

Muchsync works well for me, although I only need to sync between two
machines. It is quite fast after the initial synchronisation, and I
did not have any problems yet. I reported a small error in the
documentation, but that should be fixed by now.

-Ralph
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Tom Hirschowitz-2 Tom Hirschowitz-2
Reply | Threaded
Open this post in threaded view
|

Re: How do you synchronize your notmuch tags across multiple machines?


I also use muchsync. Had a few issues with it in the beginning but the
author, David Mazières, was quite efficient in fixing them, and patient
with me being awkwardly incompetent. It's been working like a charm
since then.

It took me some time to figure out a working set up though. My mail is
now fetched first on the same machine everytime, and then synchronised
from other machines through muchsync. This is a bit annoying, and I'd be
curious if anyone had a better way.

Tom

> * Dan Čermák:
>
>> I have found muchsync, but unfortunately very little reports about how
>> well it works, which isn't necessarily a bad thing.
>
> Muchsync works well for me, although I only need to sync between two
> machines. It is quite fast after the initial synchronisation, and I
> did not have any problems yet. I reported a small error in the
> documentation, but that should be fixed by now.
>
> -Ralph
> _______________________________________________
> notmuch mailing list
> [hidden email]
> https://notmuchmail.org/mailman/listinfo/notmuch
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Dan Čermák Dan Čermák
Reply | Threaded
Open this post in threaded view
|

Re: How do you synchronize your notmuch tags across multiple machines?

I have just given muchsync a try and it synchronizes email and tags very
quickly. I am quite impressed by it!

Unfortunately, I have hit exactly the same problem that you describe: I
have to have a single machine that pulls in my email via offlineimap and
then sync to the others via muchsync. That is a little inconvenient, as
I was hoping that I could switch the "master" depending on which machine
I am currently using and not having it up and online at all times.

I am afraid that that's not going to be easy to accomplish, as it
requires muchsync and a maildir synchronization program (in my case
offlineimap) to play together. The problem with offlineimap seems to be
that it expects the maildir filenames to have a different form and does
not recognize those that muchsync created (and it instead tries to clone
all my inboxes again). In case I'll manage to get muchsync to work
without a dedicated master, I'll let the list know.


Cheers,

Dan

Tom Hirschowitz <[hidden email]> writes:

> I also use muchsync. Had a few issues with it in the beginning but the
> author, David Mazières, was quite efficient in fixing them, and patient
> with me being awkwardly incompetent. It's been working like a charm
> since then.
>
> It took me some time to figure out a working set up though. My mail is
> now fetched first on the same machine everytime, and then synchronised
> from other machines through muchsync. This is a bit annoying, and I'd be
> curious if anyone had a better way.
>
> Tom
>
>> * Dan Čermák:
>>
>>> I have found muchsync, but unfortunately very little reports about how
>>> well it works, which isn't necessarily a bad thing.
>>
>> Muchsync works well for me, although I only need to sync between two
>> machines. It is quite fast after the initial synchronisation, and I
>> did not have any problems yet. I reported a small error in the
>> documentation, but that should be fixed by now.
>>
>> -Ralph
>> _______________________________________________
>> notmuch mailing list
>> [hidden email]
>> https://notmuchmail.org/mailman/listinfo/notmuch
> _______________________________________________
> notmuch mailing list
> [hidden email]
> https://notmuchmail.org/mailman/listinfo/notmuch

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch

signature.asc (847 bytes) Download Attachment
Brian Sniffen-2 Brian Sniffen-2
Reply | Threaded
Open this post in threaded view
|

Re: How do you synchronize your notmuch tags across multiple machines?

I had the same problem. I found a solution: dispense with imap. This can be in two ways:

For work, I use getmail (and imap) to read messages from an Exchange server and drop them in a maildir. Muchsync then coordinates those.

For personal mail, I just use muchsync to directly manipulate Dovecot’s maildir.

Both work fine. In the “work” case, changes are not pushed back up to the server.

--
Brian Sniffen

> On Jan 3, 2019, at 5:27 PM, Dan Čermák <[hidden email]> wrote:
>
> I have just given muchsync a try and it synchronizes email and tags very
> quickly. I am quite impressed by it!
>
> Unfortunately, I have hit exactly the same problem that you describe: I
> have to have a single machine that pulls in my email via offlineimap and
> then sync to the others via muchsync. That is a little inconvenient, as
> I was hoping that I could switch the "master" depending on which machine
> I am currently using and not having it up and online at all times.
>
> I am afraid that that's not going to be easy to accomplish, as it
> requires muchsync and a maildir synchronization program (in my case
> offlineimap) to play together. The problem with offlineimap seems to be
> that it expects the maildir filenames to have a different form and does
> not recognize those that muchsync created (and it instead tries to clone
> all my inboxes again). In case I'll manage to get muchsync to work
> without a dedicated master, I'll let the list know.
>
>
> Cheers,
>
> Dan
>
> Tom Hirschowitz <[hidden email]> writes:
>
>> I also use muchsync. Had a few issues with it in the beginning but the
>> author, David Mazières, was quite efficient in fixing them, and patient
>> with me being awkwardly incompetent. It's been working like a charm
>> since then.
>>
>> It took me some time to figure out a working set up though. My mail is
>> now fetched first on the same machine everytime, and then synchronised
>> from other machines through muchsync. This is a bit annoying, and I'd be
>> curious if anyone had a better way.
>>
>> Tom
>>
>>> * Dan Čermák:
>>>
>>>> I have found muchsync, but unfortunately very little reports about how
>>>> well it works, which isn't necessarily a bad thing.
>>>
>>> Muchsync works well for me, although I only need to sync between two
>>> machines. It is quite fast after the initial synchronisation, and I
>>> did not have any problems yet. I reported a small error in the
>>> documentation, but that should be fixed by now.
>>>
>>> -Ralph
>>> _______________________________________________
>>> notmuch mailing list
>>> [hidden email]
>>> https://notmuchmail.org/mailman/listinfo/notmuch
>> _______________________________________________
>> notmuch mailing list
>> [hidden email]
>> https://notmuchmail.org/mailman/listinfo/notmuch
> _______________________________________________
> notmuch mailing list
> [hidden email]
> https://notmuchmail.org/mailman/listinfo/notmuch

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Dan Čermák Dan Čermák
Reply | Threaded
Open this post in threaded view
|

Re: How do you synchronize your notmuch tags across multiple machines?

Could you maybe elaborate in more detail, as I don't know how it would
help me exactly? Are you running a server for syncing your tags?

My problem is the following: I have 3 machines E, B and C of which none
is always up and running. I would like to be able to sync my email on
any of the machines with offlineimap (but always only with one syncing)
and to transfer what's missing to the others, once I switch the
currently used machine.


Cheers,

Dan

Brian Sniffen <[hidden email]> writes:

> I had the same problem. I found a solution: dispense with imap. This can be in two ways:
>
> For work, I use getmail (and imap) to read messages from an Exchange server and drop them in a maildir. Muchsync then coordinates those.
>
> For personal mail, I just use muchsync to directly manipulate Dovecot’s maildir.
>
> Both work fine. In the “work” case, changes are not pushed back up to the server.
>
> --
> Brian Sniffen
>
>> On Jan 3, 2019, at 5:27 PM, Dan Čermák <[hidden email]> wrote:
>>
>> I have just given muchsync a try and it synchronizes email and tags very
>> quickly. I am quite impressed by it!
>>
>> Unfortunately, I have hit exactly the same problem that you describe: I
>> have to have a single machine that pulls in my email via offlineimap and
>> then sync to the others via muchsync. That is a little inconvenient, as
>> I was hoping that I could switch the "master" depending on which machine
>> I am currently using and not having it up and online at all times.
>>
>> I am afraid that that's not going to be easy to accomplish, as it
>> requires muchsync and a maildir synchronization program (in my case
>> offlineimap) to play together. The problem with offlineimap seems to be
>> that it expects the maildir filenames to have a different form and does
>> not recognize those that muchsync created (and it instead tries to clone
>> all my inboxes again). In case I'll manage to get muchsync to work
>> without a dedicated master, I'll let the list know.
>>
>>
>> Cheers,
>>
>> Dan
>>
>> Tom Hirschowitz <[hidden email]> writes:
>>
>>> I also use muchsync. Had a few issues with it in the beginning but the
>>> author, David Mazières, was quite efficient in fixing them, and patient
>>> with me being awkwardly incompetent. It's been working like a charm
>>> since then.
>>>
>>> It took me some time to figure out a working set up though. My mail is
>>> now fetched first on the same machine everytime, and then synchronised
>>> from other machines through muchsync. This is a bit annoying, and I'd be
>>> curious if anyone had a better way.
>>>
>>> Tom
>>>
>>>> * Dan Čermák:
>>>>
>>>>> I have found muchsync, but unfortunately very little reports about how
>>>>> well it works, which isn't necessarily a bad thing.
>>>>
>>>> Muchsync works well for me, although I only need to sync between two
>>>> machines. It is quite fast after the initial synchronisation, and I
>>>> did not have any problems yet. I reported a small error in the
>>>> documentation, but that should be fixed by now.
>>>>
>>>> -Ralph
>>>> _______________________________________________
>>>> notmuch mailing list
>>>> [hidden email]
>>>> https://notmuchmail.org/mailman/listinfo/notmuch
>>> _______________________________________________
>>> notmuch mailing list
>>> [hidden email]
>>> https://notmuchmail.org/mailman/listinfo/notmuch
>> _______________________________________________
>> notmuch mailing list
>> [hidden email]
>> https://notmuchmail.org/mailman/listinfo/notmuch

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch

signature.asc (847 bytes) Download Attachment
David Mazieres David Mazieres
Reply | Threaded
Open this post in threaded view
|

Re: How do you synchronize your notmuch tags across multiple machines?

Dan Čermák <[hidden email]> writes:

> Could you maybe elaborate in more detail, as I don't know how it would
> help me exactly? Are you running a server for syncing your tags?
>
> My problem is the following: I have 3 machines E, B and C of which none
> is always up and running. I would like to be able to sync my email on
> any of the machines with offlineimap (but always only with one syncing)
> and to transfer what's missing to the others, once I switch the
> currently used machine.

The problem is that muchsync doesn't know about imap or offlineimap.
Muchsync will happily ensure that all of your notmuch maildirs and tags
are synchronized.  But if you run offlineimap on E and then sync E to B,
B will already have a copy of all new messages.  If you run offlineimap
on B then, it will download additional copies of the messages already
synchronized over muchsync.

One solution would be to archive all the messages downloaded with
offlineimap as you download them.  But then you have to remember to run
muchsync to upload them to other machines, or you risk losing track of
messages if you download them to a machine that you then immediately
turn off.

I'm open to other ideas if they require some kind of new feature from
muchsync, but I'm worried that this would require delving into the guts
of offlineimap (and then wouldn't work with other solutions like isync).

The ideal solution would be to implement an imap server on top of
libnotmuch.  If we had that, then you could just use offlineimap and
isync through the imap (as opposed to file system) interface, and
everything will just work.  That would also have the benefit of making
notmuch work really well with phones--you could use the gmail app on
your phone and the emacs/vim front-end on your desktop.  It's even
conceivable that such an imap server could use notmuch's indexing to
support the gmail imap extensions:

        https://developers.google.com/gmail/imap/imap-extensions

Unfortunately, implementing an imap server is a bit beyond the scope of
muchsync and not something I have time for right now.

David
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Dan Čermák Dan Čermák
Reply | Threaded
Open this post in threaded view
|

Re: How do you synchronize your notmuch tags across multiple machines?

David Mazieres <[hidden email]> writes:

> Dan Čermák <[hidden email]> writes:
>
>> Could you maybe elaborate in more detail, as I don't know how it would
>> help me exactly? Are you running a server for syncing your tags?
>>
>> My problem is the following: I have 3 machines E, B and C of which none
>> is always up and running. I would like to be able to sync my email on
>> any of the machines with offlineimap (but always only with one syncing)
>> and to transfer what's missing to the others, once I switch the
>> currently used machine.
>
> The problem is that muchsync doesn't know about imap or offlineimap.
> Muchsync will happily ensure that all of your notmuch maildirs and tags
> are synchronized.  But if you run offlineimap on E and then sync E to B,
> B will already have a copy of all new messages.  If you run offlineimap
> on B then, it will download additional copies of the messages already
> synchronized over muchsync.
>
> One solution would be to archive all the messages downloaded with
> offlineimap as you download them.  But then you have to remember to run
> muchsync to upload them to other machines, or you risk losing track of
> messages if you download them to a machine that you then immediately
> turn off.
>
> I'm open to other ideas if they require some kind of new feature from
> muchsync, but I'm worried that this would require delving into the guts
> of offlineimap (and then wouldn't work with other solutions like isync).
>
I am afraid that you are right in terms of delving into
offlineimap/isync/etc.

I have experimented a little with offlineimap + muchsync, but I couldn't
get offlineimap to recognize messages that muchsync downloaded. My guess
is that offlineimap keeps a separate UID database somewhere, that
muchsync would have to write to. Messing with offlineimap's internal
state sounds wrong to me.

> The ideal solution would be to implement an imap server on top of
> libnotmuch.  If we had that, then you could just use offlineimap and
> isync through the imap (as opposed to file system) interface, and
> everything will just work.

Stupid question: how would the tags be synced with your local machines
in this case? Via muchsync or would one keep the tags on the
notmuch-imap server?

> That would also have the benefit of making
> notmuch work really well with phones--you could use the gmail app on
> your phone and the emacs/vim front-end on your desktop.  It's even
> conceivable that such an imap server could use notmuch's indexing to
> support the gmail imap extensions:
>
>         https://developers.google.com/gmail/imap/imap-extensions
>
> Unfortunately, implementing an imap server is a bit beyond the scope of
> muchsync and not something I have time for right now.
>
> David
How about a hacky and not ideal solution: "teach" notmuch to not only
synchronize the read and deleted tags with the maildir, but all tags?
E.g. by injecting a header into the email?
I would guess that this is simpler to implement then an IMAP server, but
it carries its own set of problems (iirc there were discussions about
such an idea on this list but were discarded for reasons that I don't
remember). For instance I don't know whether one could even push tags to
the server, as one would have push the whole message back.

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch

signature.asc (847 bytes) Download Attachment
David Mazieres David Mazieres
Reply | Threaded
Open this post in threaded view
|

Re: How do you synchronize your notmuch tags across multiple machines?

Dan Čermák <[hidden email]> writes:

>> The ideal solution would be to implement an imap server on top of
>> libnotmuch.  If we had that, then you could just use offlineimap and
>> isync through the imap (as opposed to file system) interface, and
>> everything will just work.
>
> Stupid question: how would the tags be synced with your local machines
> in this case? Via muchsync or would one keep the tags on the
> notmuch-imap server?

However you want.  If you run a local imap server and a remote imap
server, then you can just sync between the two imap servers with any
imap synching tool.  But if the two hosts happen to be running imap
servers implemented on top of libnotmuch, then you could use muchsync
which would be faster.

> How about a hacky and not ideal solution: "teach" notmuch to not only
> synchronize the read and deleted tags with the maildir, but all tags?

Unfortunately, there's no standard for how to encode these.  Also,
there's a pretty fundamental design decision in notmuch that it doesn't
edit the mail messages.  Now some imap servers, like dovecot, define
other flags on a per-maildir basis, but it only works up to a certain
number of flags, and you'd have to parse an extra file that tells you
what the mappings are.

David
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Jeff Templon Jeff Templon
Reply | Threaded
Open this post in threaded view
|

Re: How do you synchronize your notmuch tags across multiple machines?

Hi,

I too, am using muchsync to synchronize between two machine, I too have
profited from David's quick responses, and I too deal with the problem
of only being able to "fetch mail" on one machine.

I followed the discussion with interest.

Looking at the isync (mbsync) docs (I can't speak for offlineimap) it
might be easier: the state of the sync seems to be kept in little files
per Maildir subdirectory, called .mbsyncstate and .uivalidity.  If
muchsync were to sync these files as well, then I think it would work.
See the explanation of the .mbsyncstate file in this thread:

https://sourceforge.net/p/isync/mailman/message/34682807/

quote: "mbsync maintains a mapping of remote (col 1) to local (col 2) uids.
when you migrate, you can just duplicate the columns."

He's talking about migrating a mail store from offlineimap to mbsync, I
guess the issues would be the same.

HTH,

JT
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
David Mazieres David Mazieres
Reply | Threaded
Open this post in threaded view
|

Re: How do you synchronize your notmuch tags across multiple machines?

Jeff Templon <[hidden email]> writes:

> quote: "mbsync maintains a mapping of remote (col 1) to local (col 2) uids.
> when you migrate, you can just duplicate the columns."
>
> He's talking about migrating a mail store from offlineimap to mbsync, I
> guess the issues would be the same.

I wonder if the thing to do would be to write an mbsync-specific utility
to convert IMAP uids to notmuch properties and back.  Then maybe I could
extend muchsync to sync properties (which it currently ignores because I
wrote muchsync before properties existed, and still don't really know
what properties are good for).

A problem with syncing properties, though, is that I don't know how to
do conflict resolution correctly 100% of the time.  One of the key
insights behind muchsync is that with file names and binary flags, I
think I figured out a sane way of resolving all conflicts without
involving the user (namely take the max number of links in each
directory with special treatment for new/ and cur/, then logically and
the muchsync.and_tags, and logically or the other tags).

With properties, it's not clear how to sync them generically.  Moreover,
some properties like session-key probably should not be synced, since
you probably don't want to upload the equivalent of plaintext main
messages to your mail server.

David
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
jpellegrini jpellegrini
Reply | Threaded
Open this post in threaded view
|

Re: How do you synchronize your notmuch tags across multiple machines?

In reply to this post by Brian Sniffen-2
Perhaps a bit late to reply to this, but I believe this can be useful:

I managed to sync one machine which runs postfix and receives my email, and
other three (home desktop, work desktop and notebook).

- the server runs Postfix and Dovecot
- muchsync is configured to synchronize local maildirs in the clients with
the same maildir which is used by postfix and dovecot
- the only synchronization I saw as necessary was to not let 2 different
muchsync clients, or the dovecot server, to access the maildir at the same
time (postfix never removes or edits messages, it only drops them in the
maildir). since dovecot /does/ provide a maildir locking program, this was
actually easy to do:

  the clients run this:




 in  the server, the program that muchsync runs (the $remote_program above)
is



- also, since both myuser, dovecote and postfix will write to the same
directory, I had to configure dovecot and postfix to use my uid and gid (I
actually have these in a postfixadmin setup, so they're in adatabase).

So far it works fine -- because I don't need to use IMAP all the time. I
mostly use notmuch-aware clientes in the desktops and notebook, and only use
IMAP from the phone.

Hope this is useful.
J.




--
Sent from: http://notmuch.198994.n3.nabble.com/
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Ioan-Adrian Ratiu Ioan-Adrian Ratiu
Reply | Threaded
Open this post in threaded view
|

Re: How do you synchronize your notmuch tags across multiple machines?

In reply to this post by Dan Čermák
My solution to this problem is to have only one notmuch instance
run on a remote VPS and connect to it via "thin" notmuch-emacs
front-ends, very similar to [1]. It has been working very well for
the last few years.

[1] https://notmuchmail.org/remoteusage/

On Mon, 17 Dec 2018, Dan Čermák <[hidden email]> wrote:

> Hi list,
>
> first and foremost: thanks to everyone who contributed to notmuch, it's
> a truly awesome piece of software.
>
> Unfortunately, I am facing the problem (which I have been postponing to
> face for a while) that I'll need to keep my notmuch database in sync
> over multiple machines.
> Thus my question: how are you achieving that? How well does your
> solution work?
>
> (I have found muchsync, but unfortunately very little reports about how
> well it works, which isn't necessarily a bad thing. Beside that only
> some scripts around notmuch dump.)
>
>
> Thanks in advance,
>
> Dan
> _______________________________________________
> notmuch mailing list
> [hidden email]
> https://notmuchmail.org/mailman/listinfo/notmuch
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Dan Čermák Dan Čermák
Reply | Threaded
Open this post in threaded view
|

Re: How do you synchronize your notmuch tags across multiple machines?

In reply to this post by jpellegrini
Hi,

thanks for sharing your setup!

I have two points/questions:
1. Since your server runs dovecot & postfix, do you use this as your
   primary email server and forward all your email to it or do you use
   some other method to keep it synchronized with your email accounts?

2. I think the sync command was dropped from your email (see the quoted
   part below).


Cheers,

Dan

jpellegrini <[hidden email]> writes:

> Perhaps a bit late to reply to this, but I believe this can be useful:
>
> I managed to sync one machine which runs postfix and receives my email, and
> other three (home desktop, work desktop and notebook).
>
> - the server runs Postfix and Dovecot
> - muchsync is configured to synchronize local maildirs in the clients with
> the same maildir which is used by postfix and dovecot
> - the only synchronization I saw as necessary was to not let 2 different
> muchsync clients, or the dovecot server, to access the maildir at the same
> time (postfix never removes or edits messages, it only drops them in the
> maildir). since dovecot /does/ provide a maildir locking program, this was
> actually easy to do:
>
>   the clients run this:
>
>
>
>
>  in  the server, the program that muchsync runs (the $remote_program above)
> is
>
>
>
> - also, since both myuser, dovecote and postfix will write to the same
> directory, I had to configure dovecot and postfix to use my uid and gid (I
> actually have these in a postfixadmin setup, so they're in adatabase).
>
> So far it works fine -- because I don't need to use IMAP all the time. I
> mostly use notmuch-aware clientes in the desktops and notebook, and only use
> IMAP from the phone.
>
> Hope this is useful.
> J.
>
>
>
>
> --
> Sent from: http://notmuch.198994.n3.nabble.com/
> _______________________________________________
> notmuch mailing list
> [hidden email]
> https://notmuchmail.org/mailman/listinfo/notmuch

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch

signature.asc (847 bytes) Download Attachment