web interface to notmuch

classic Classic list List threaded Threaded
27 messages Options
12
Matthew Lear Matthew Lear
Reply | Threaded
Open this post in threaded view
|

Re: web interface to notmuch


On Tue, Oct 31, 2017 at 7:21 PM, Brian Sniffen <[hidden email]> wrote:

> I'm no Python expert, but from a quick google it would seem like the cause
> of such an exception is related to not using utf-8.

Neat.  So to get there, this has to be a text/html part.  It has to have
been decoded, either with the declared content type or with ascii.  If a
\u201c (left double quote) showed up, it didn't get decoded as
ascii---and indeed, it looks like the content-type specifies latin-1.
But now when we try to encode back, using the same latin-1, it fails?
That's really neat.

> Brian - do you think something needs modifying in nmweb.py to cater for
> this type of thing, or is this somehow related my own mailstore (not sure
> why that would be as my messages haven't been modified).

Lots of mail has busted encoding.  I've done some defensive work against
that---look at decodeAnyway and shed a tear for purity---but clearly not
enough.  Can you send me a message that causes the problem?

I'll need to fix up the text in the message because it's confidential. That should be easy enough to do.
I'll send it to you once I've done that.

One other thing - it looks like accessing attachments should work, but I've seen messages in my local set up here which have attachments shown, but I've not been able to retrieve them.
Not sure what would cause that. Also some messages which are tagged as having attachments, don't have them shown by nmweb.

FWIW this link (https://nmweb.evenmere.org/show/CACMMjMLecmXopb8AATjE3UuCnNLOO%2B5Nmev5X8K-UostDEUdrQ%40mail.gmail.com) has the tag attachment applied to the message, but there is no attachment shown.  And another (https://nmweb.evenmere.org/show/87d31artti.fsf%40inf-8657.int-evry.fr).

Maybe text/plain only emails are the ones which aren't problematic w.r.t. having their attachments shown?
Cheers,
--  Matt



_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Matthew Lear Matthew Lear
Reply | Threaded
Open this post in threaded view
|

Re: web interface to notmuch

Comparing with the Haskell based notmuch-web, while slightly slower to render a browser page with the same search terms as nmweb, I can view the email which causes nmweb to throw the encoding exception just fine in it. I guess something in that implementation is able to handle encoding differently.
Regards,
  Matt


On Tue, 31 Oct 2017, 21:32 Matthew Lear, <[hidden email]> wrote:
On Tue, Oct 31, 2017 at 7:21 PM, Brian Sniffen <[hidden email]> wrote:

> I'm no Python expert, but from a quick google it would seem like the cause
> of such an exception is related to not using utf-8.

Neat.  So to get there, this has to be a text/html part.  It has to have
been decoded, either with the declared content type or with ascii.  If a
\u201c (left double quote) showed up, it didn't get decoded as
ascii---and indeed, it looks like the content-type specifies latin-1.
But now when we try to encode back, using the same latin-1, it fails?
That's really neat.

> Brian - do you think something needs modifying in nmweb.py to cater for
> this type of thing, or is this somehow related my own mailstore (not sure
> why that would be as my messages haven't been modified).

Lots of mail has busted encoding.  I've done some defensive work against
that---look at decodeAnyway and shed a tear for purity---but clearly not
enough.  Can you send me a message that causes the problem?

I'll need to fix up the text in the message because it's confidential. That should be easy enough to do.
I'll send it to you once I've done that.

One other thing - it looks like accessing attachments should work, but I've seen messages in my local set up here which have attachments shown, but I've not been able to retrieve them.
Not sure what would cause that. Also some messages which are tagged as having attachments, don't have them shown by nmweb.

FWIW this link (https://nmweb.evenmere.org/show/CACMMjMLecmXopb8AATjE3UuCnNLOO%2B5Nmev5X8K-UostDEUdrQ%40mail.gmail.com) has the tag attachment applied to the message, but there is no attachment shown.  And another (https://nmweb.evenmere.org/show/87d31artti.fsf%40inf-8657.int-evry.fr).

Maybe text/plain only emails are the ones which aren't problematic w.r.t. having their attachments shown?
Cheers,
--  Matt



_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Brian Sniffen-2 Brian Sniffen-2
Reply | Threaded
Open this post in threaded view
|

Re: web interface to notmuch

In reply to this post by Matthew Lear
On Oct 31, 2017, at 5:32 PM, Matthew Lear <[hidden email]> wrote:


FWIW this link (https://nmweb.evenmere.org/show/CACMMjMLecmXopb8AATjE3UuCnNLOO%2B5Nmev5X8K-UostDEUdrQ%40mail.gmail.com) has the tag attachment applied to the message, but there is no attachment shown.  And another (https://nmweb.evenmere.org/show/87d31artti.fsf%40inf-8657.int-evry.fr).

Funny thing, the text of those attachments was being emitted by the app server... but since an iframe tag used in showing text/html parts *wasn’t* being closed, that didn’t matter.

I’ll push some fixes to encoding to github later today.

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Matthew Lear Matthew Lear
Reply | Threaded
Open this post in threaded view
|

Re: web interface to notmuch


On Wed, 1 Nov 2017, 14:38 Brian Sniffen, <[hidden email]> wrote:
On Oct 31, 2017, at 5:32 PM, Matthew Lear <[hidden email]> wrote 

I’ll push some fixes to encoding to github later today.

I've just tested with e3ba123 and there is a significant improvement. Also, attachments can be accessed from html too. And it's even faster than before! Sweet :-) I've still got a couple of mails which don't throw exceptions but do have issues when rendered in the browser (encoding of the body generates unreadable characters). I'll try to send you some examples tomorrow.


_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Brian Sniffen-2 Brian Sniffen-2
Reply | Threaded
Open this post in threaded view
|

Re: web interface to notmuch

In reply to this post by Daniel Kahn Gillmor
> Assuming that you had a sanitize_this_html_part() function available to
> you, do you think it would be possible to make this safe?  Have you
> considered proposing it for inclusion in contrib upstream?

Okay, https://github.com/briansniffen/notmuch/tree/nmweb is now rebased
onto the notmuchmail.org head as of this morning.  All of the changes
are under contrib/notmuch-web.

I haven't done this before, so: exactly how would you like this proposed
for upstream inclusion?  It looks from
https://notmuchmail.org/contributing/ like you'd like documentation,
tests, NEWS, and then `git send-email`.  Is that right?  

Do you want this crunched into one commit, "write an e-mail client"?

Thanks,
Brian
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Daniel Kahn Gillmor Daniel Kahn Gillmor
Reply | Threaded
Open this post in threaded view
|

Re: web interface to notmuch

Hi Brian--

On Wed 2017-12-06 10:00:19 -0500, Brian Sniffen wrote:
> Okay, https://github.com/briansniffen/notmuch/tree/nmweb is now rebased
> onto the notmuchmail.org head as of this morning.  All of the changes
> are under contrib/notmuch-web.

thanks for doing this!  

traditionally, we've encouraged code contributions to come in in
separate patches (not one big lump), with each commit clearly justifying
their changes.  And yes, tests would make it much more likely that this
sweet contribution doesn't bitrot!

one thing i note is you've included a copy of jquery directly.  i think
we want to avoid having retrieved or generated code in the repository.
if there's an external dependency, the repo should point to it but it
shouldn't include it.  notmuch doesn't include gmime, but it states that
it has gmime as a dependency (with specific versions), and lets the
builder/maintainer retrieve the dependency themselves.  This makes
licensing, source vetting, and maintenance easier by ensuring that it
all stays loosely coupled.  Presumably you aren't signing up for
ensuring that any security vulnerabilities in jquery get pushed into the
notmuch repo too :)

Maybe you could do a similar explicitly-stated dependency for jquery
instead of including it directly?  debian ships jquery as libjs-jquery,
for example (though you could also use an explicit fetch from the web
for systems that aren't running an OS that provides a packaged form of
jquery).

other folks might have other preferences, i hope they'll chime in.

         --dkg

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch

signature.asc (847 bytes) Download Attachment
David Bremner-2 David Bremner-2
Reply | Threaded
Open this post in threaded view
|

Re: web interface to notmuch

In reply to this post by Brian Sniffen-2
Brian Sniffen <[hidden email]> writes:

>> Assuming that you had a sanitize_this_html_part() function available to
>> you, do you think it would be possible to make this safe?  Have you
>> considered proposing it for inclusion in contrib upstream?
>
> Okay, https://github.com/briansniffen/notmuch/tree/nmweb is now rebased
> onto the notmuchmail.org head as of this morning.  All of the changes
> are under contrib/notmuch-web.
>

Maybe a minor point, but there is at least one existing "notmuch-web" [1].

I'm not sure if John Lenz (in copy) considers the project a going
concern, but it might be worth thinking about search engine collision.

> I haven't done this before, so: exactly how would you like this proposed
> for upstream inclusion?  It looks from
> https://notmuchmail.org/contributing/ like you'd like documentation,
> tests, NEWS, and then `git send-email`.  Is that right?  
>
> Do you want this crunched into one commit, "write an e-mail client"?

I think Daniel covered this well, but let me reinforce that we want to
review things in meaningful, not too large chunks.

[1]: https://bitbucket.org/wuzzeb/notmuch-web
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
12