Quantcast

[gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

classic Classic list List threaded Threaded
10 messages Options
Gaute Hope Gaute Hope
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

Hi,

  'gmailieer' (or 'gmi') is a small program that can pull email and labels
(and changes to labels) from your GMail account and store them locally in a
maildir with the labels synchronized with a notmuch database. The
changes to tags in the notmuch database may be pushed back remotely to
your GMail account.

The initial fetch of all emails takes some time, but synchronizing
labels and tags, and checking for new messages, is usually done in 1-2
seconds.

It requires the most recent notmuch, the python googleapi bindings and
tqdm.

Disclaimer:

  This is still experimental, but it does not have access to delete
  e-mail on your account - only fetch and change labels, so damage
  should be limited.



Instructions and source code can be found here:

  https://github.com/gauteh/gmailieer




Regards, Gaute

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch

attachment0 (849 bytes) Download Attachment
David Bremner-2 David Bremner-2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

Gaute Hope <[hidden email]> writes:

> Hi,
>
>   'gmailieer' (or 'gmi') is a small program that can pull email and labels
> (and changes to labels) from your GMail account and store them locally in a
> maildir with the labels synchronized with a notmuch database. The
> changes to tags in the notmuch database may be pushed back remotely to
> your GMail account.

Hi Gaute;

Thanks for writing this. It seems like the right approach for people who
want to sync gmail labels with notmuch.

d
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Rafael Avila de Espindola Rafael Avila de Espindola
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

In reply to this post by Gaute Hope

That looks really interesting. What is the license? :-)

Cheers,
Rafael

Gaute Hope <[hidden email]> writes:

> Hi,
>
>   'gmailieer' (or 'gmi') is a small program that can pull email and labels
> (and changes to labels) from your GMail account and store them locally in a
> maildir with the labels synchronized with a notmuch database. The
> changes to tags in the notmuch database may be pushed back remotely to
> your GMail account.
>
> The initial fetch of all emails takes some time, but synchronizing
> labels and tags, and checking for new messages, is usually done in 1-2
> seconds.
>
> It requires the most recent notmuch, the python googleapi bindings and
> tqdm.
>
> Disclaimer:
>
>   This is still experimental, but it does not have access to delete
>   e-mail on your account - only fetch and change labels, so damage
>   should be limited.
>
>
>
> Instructions and source code can be found here:
>
>   https://github.com/gauteh/gmailieer
>
>
>
>
> Regards, Gaute
> _______________________________________________
> notmuch mailing list
> [hidden email]
> https://notmuchmail.org/mailman/listinfo/notmuch
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Gaute Hope Gaute Hope
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

Rafael Avila de Espindola writes on mars 14, 2017 16:51:
>
> That looks really interesting. What is the license? :-)

Ay, GPL v3+ it is!

- gaute

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Rafael Avila de Espindola Rafael Avila de Espindola
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

I currently have a mbsync + notmuch setup and I am just about to give
this a try.

One question is what the notmuch setup should look like. I assume I
should create a new config file pointing to the directory where the
emails will be written to. Do I have to run notmuch setup to create the
db?

Cheers,
Rafael

Gaute Hope <[hidden email]> writes:

> Rafael Avila de Espindola writes on mars 14, 2017 16:51:
>>
>> That looks really interesting. What is the license? :-)
>
> Ay, GPL v3+ it is!
>
> - gaute
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Gaute Hope Gaute Hope
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

Rafael Avila de Espindola writes on mars 27, 2017 4:32:
> I currently have a mbsync + notmuch setup and I am just about to give
> this a try.
>
> One question is what the notmuch setup should look like. I assume I
> should create a new config file pointing to the directory where the
> emails will be written to. Do I have to run notmuch setup to create the
> db?
>

Yes, it is a good idea to just back up your entire notmuch db and set up
a new one to test gmailieer with.

gmailieer respects $NOTMUCH_CONFIG and reads it to figure out which
'new' tags to add (no need to use inbox or unread here, that will only
create confusion - I only use 'new' to do tagging).

Assuming you want to set up a separate notmuch database to test you do
the following:

first:
    get a client secret API key for gmailieer [0], you only need this
  during setup or if gmailieer needs to be re-authenticated. so store it
  somewhere safe, say: ~/.secret/client_secret.json. ideally this should
  be unnecessary [1].

# setting up a separate notmuch db
$ mkdir ~/test_mail/  # our new mail root
$ cd ~/test_mail
$ export NOTMUCH_CONFIG=$(pwd)/.notmuch-config
$ mkdir mail          # new mail database root
$ notmuch setup
  * specify ~/test_mail/mail as mail database
  * specify only 'new' as new tag
$ notmuch new         # create db
$ cd mail

NOTE: As long as you are using a non-standard notmuch config location make
sure that environment variable is set when using gmailieer.

# setting up gmailieer
$ mkdir gmail
  * this will be the gmailieer local repository.
  * it should be a sub-dir of the root mail db.
  * you can keep other mail (from other accounts) outside this
    gmailieer local repository.
$ cd gmail
$ gmi auth -c ~/.secret/client_secret.json
  * authenticate gmailieer, choose the correct account, say:
    [hidden email]
$ gmi init -a [hidden email] # gmailieer should now be set up
$ gmi pull # does the initial pull, fetching all you messages - this
           # could take some time.

When you want to synchronize your account you do:
$ gmi sync    # which effectively does gmi push; gmi pull

this will also update the notmuch database so it should not be necessary
to run 'notmuch new' afterwards.

Regards, Gaute

[0] https://console.developers.google.com/flows/enableapi?apiid=gmail
[1] https://github.com/gauteh/gmailieer/issues/1

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Rafael Avila de Espindola Rafael Avila de Espindola
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

> Assuming you want to set up a separate notmuch database to test you do
> the following:
>
> first:
>    get a client secret API key for gmailieer [0], you only need this
>  during setup or if gmailieer needs to be re-authenticated. so store it
>  somewhere safe, say: ~/.secret/client_secret.json. ideally this should
>  be unnecessary [1].
>
> # setting up a separate notmuch db
> $ mkdir ~/test_mail/  # our new mail root
> $ cd ~/test_mail
> $ export NOTMUCH_CONFIG=$(pwd)/.notmuch-config
> $ mkdir mail          # new mail database root
> $ notmuch setup
>  * specify ~/test_mail/mail as mail database
>  * specify only 'new' as new tag
> $ notmuch new         # create db
> $ cd mail
>
> NOTE: As long as you are using a non-standard notmuch config location make
> sure that environment variable is set when using gmailieer.
>
> # setting up gmailieer
> $ mkdir gmail
>  * this will be the gmailieer local repository.
>  * it should be a sub-dir of the root mail db.
>  * you can keep other mail (from other accounts) outside this
>    gmailieer local repository.
> $ cd gmail
> $ gmi auth -c ~/.secret/client_secret.json
>  * authenticate gmailieer, choose the correct account, say:
>    [hidden email]
> $ gmi init -a [hidden email] # gmailieer should now be set up
> $ gmi pull # does the initial pull, fetching all you messages - this
>           # could take some time.

It is currently running. Thank you so much.

The only things I noticed so far:

* running auth before init fails, but running init -c ... -a ... works
* I had to name the auth file client_secret.json or "gmi pull" would
fail trying to read it.

Thanks again for writing this. It seems a much better way to sync with
gmail. My previous setup was to use mbsync. It is really fast for
fetching new emails (--pull-new), but with imap the only way to get
flag changes (including read/unread) seems to be to ask for the flag
of every message!

Cheers,
Rafael
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Gaute Hope Gaute Hope
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

Rafael EspĂ­ndola writes on mars 27, 2017 15:23:

>> $ gmi pull # does the initial pull, fetching all you messages - this
>>           # could take some time.
>
> It is currently running. Thank you so much.
>
> The only things I noticed so far:
>
> * running auth before init fails, but running init -c ... -a ... works
> * I had to name the auth file client_secret.json or "gmi pull" would
> fail trying to read it.


Right, that should be the other way around. I'll update the README.

There are two secret files. One is the API client_secret which can be
specified with the '-c' option (also for pull), this one is only needed
initially and can otherwise be stored somewhere else:

When authenticating (either running 'auth' or some operation requiring
authentication: pull/push) a local access token is retrieved, this is
stored in: .credentials.gmailieer.json in the local mail repository.

> gmail. My previous setup was to use mbsync. It is really fast for
> fetching new emails (--pull-new), but with imap the only way to get
> flag changes (including read/unread) seems to be to ask for the flag
> of every message!

Thanks!

- gaute
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Rafael Avila de Espindola Rafael Avila de Espindola
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

In reply to this post by Gaute Hope
After a few issues with the initial sync this is working perfectly.

Thanks a lot, it is a big improvement over mbsync.

Cheers,
Rafael

Gaute Hope <[hidden email]> writes:

> Hi,
>
>   'gmailieer' (or 'gmi') is a small program that can pull email and labels
> (and changes to labels) from your GMail account and store them locally in a
> maildir with the labels synchronized with a notmuch database. The
> changes to tags in the notmuch database may be pushed back remotely to
> your GMail account.
>
> The initial fetch of all emails takes some time, but synchronizing
> labels and tags, and checking for new messages, is usually done in 1-2
> seconds.
>
> It requires the most recent notmuch, the python googleapi bindings and
> tqdm.
>
> Disclaimer:
>
>   This is still experimental, but it does not have access to delete
>   e-mail on your account - only fetch and change labels, so damage
>   should be limited.
>
>
>
> Instructions and source code can be found here:
>
>   https://github.com/gauteh/gmailieer
>
>
>
>
> Regards, Gaute
> _______________________________________________
> notmuch mailing list
> [hidden email]
> https://notmuchmail.org/mailman/listinfo/notmuch
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Gaute Hope Gaute Hope
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [gmailieer] fast fetch and two-way tag synchronization between notmuch and GMail

Rafael Avila de Espindola writes on april 3, 2017 16:35:
> After a few issues with the initial sync this is working perfectly.
>
> Thanks a lot, it is a big improvement over mbsync.
>

Great, thanks for the patches! This made inital sync much more robust.

I have a non-user-api-key version working [0], but I need to figure out if
it is safe first [1].

- Gaute

[0] https://github.com/gauteh/gmailieer/pull/9
[1] http://stackoverflow.com/questions/43173367/is-it-safe-to-distribute-client-id-and-client-secret-for-google-api-for-an-insta

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch

attachment0 (849 bytes) Download Attachment
Loading...