Bug: fatal error with notmuch new, second run starts indexing all over again

classic Classic list List threaded Threaded
18 messages Options
Gregor Zattler Gregor Zattler
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Bug: fatal error with notmuch new, second run starts indexing all over again

Dear notmuch developers,

notmuch new fails while indexing my emails and when executed a
second time it obviously starts over again.  This is on a
up-to-date debian stetch with notmuch from git:

0 (master) grfz@len:~/Mail$ notmuch new
Found 940740 total files (that's not much mail).
Error: A Xapian exception occurred. Halting processing.
Processed 659979 total files in 1h 43m 20s (106 files/sec.).
Added 584654 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
1 (master) grfz@len:~/Mail$ notmuch new
Error: A Xapian exception occurred. Halting processing.
Processed 660023 total files in 1h 44m 45s (105 files/sec.).
Added 584698 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
1 (master) grfz@len:~/Mail$ notmuch --version
notmuch 0.24.2+100~gab02265

The resutling xapian databases are not as large as I would
expect (an older xapian database from another config is 11 G
big):

0 (master) grfz@len:~/Mail$ du -hs .
28G     .
0 (master) grfz@len:~/Mail$ du -hs .notmuch
7,1G    .notmuch

It's now not possible to search with notmuch:
0 (master) grfz@len:~/Mail$ notmuch search date:year..
0 (master) grfz@len:~/Mail$

Any ideas how to proceed in order to get a working notmuch setup?


Ciao, Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
David Bremner-2 David Bremner-2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

Gregor Zattler <[hidden email]> writes:

> Dear notmuch developers,
>
> notmuch new fails while indexing my emails and when executed a
> second time it obviously starts over again.  This is on a
> up-to-date debian stetch with notmuch from git:
> Any ideas how to proceed in order to get a working notmuch setup?
>

> Ciao, Gregor

Some things to try

1. Check disk space
2. run xapian-fsck
3. run "notmuch new" under gdb to try to get a backtrace
4. use "git bisect" to try to find what commit introduced this problem

d



_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch

signature.asc (671 bytes) Download Attachment
Gregor Zattler Gregor Zattler
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

Hi David, notmuch developers,
* David Bremner <[hidden email]> [03. Jul. 2017]:
> Gregor Zattler <[hidden email]> writes:
>> notmuch new fails while indexing my emails and when executed a
>> second time it obviously starts over again.  This is on a
>> up-to-date debian stetch with notmuch from git:
>> Any ideas how to proceed in order to get a working notmuch setup?

> 1. Check disk space

Sorry, forgot to mention: there is enough free space on RAM (8
GB) and SSD (40 G free).

> 2. run xapian-fsck

xapian-check does not find any errors, only spelling and synonym
are "Lazily created, and not yet used."

> 3. run "notmuch new" under gdb to try to get a backtrace

I have no clue about gdb, there was no backtrace:

0 (master) grfz@len:~$ gdb --args notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from notmuch...done.
(gdb) run
Starting program: /usr/local/bin/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".
Error: A Xapian exception occurred. Halting processing.
Processed 662357 total files in 1h 52m 51s (97 files/sec.).
Added 587023 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
[Inferior 1 (process 4509) exited with code 01]
(gdb) bt
No stack.
(gdb)


> 4. use "git bisect" to try to find what commit introduced this problem

I'll do so in the next few days.

Thanks, Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
David Bremner-2 David Bremner-2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

Gregor Zattler <[hidden email]> writes:

>
> I have no clue about gdb, there was no backtrace:
>
...
> Error: A Xapian exception occurred. Halting processing.
> Processed 662357 total files in 1h 52m 51s (97 files/sec.).
> Added 587023 new messages to the database.
> Note: A fatal error was encountered: A Xapian exception occurred
> [Inferior 1 (process 4509) exited with code 01]
> (gdb) bt
> No stack.
> (gdb)

Try setting a break in _notmuch_database_log (b _notmuch_database_log)
and run "bt" at that break point. It might (or might not) be worth
continuing after the first breakpoint to inspect other errors.

d
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Tomi Ollila-2 Tomi Ollila-2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Bug: fatal error with notmuch new, second run starts indexing all over again


You could also try notmuch new --verbose

--- and then look into the last file it was indexing when crash occurred...


Tomi

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Gregor Zattler Gregor Zattler
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

In reply to this post by David Bremner-2
Hi David, notmuch developers,
* David Bremner <[hidden email]> [03. Jul. 2017]:
> Try setting a break in _notmuch_database_log (b _notmuch_database_log)
> and run "bt" at that break point. It might (or might not) be worth
> continuing after the first breakpoint to inspect other errors.

I did so with newest notmuch:

0 (master *) grfz@len:~/tmp$ notmuch --version
notmuch 0.24.2+84~g1ec6344
0 (master *) grfz@len:~/tmp$

0 (master=) grfz@len:~/src/notmuch$ gdb --args /home/grfz/src/notmuch/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/grfz/src/notmuch/notmuch...done.
(gdb) b _notmuch_database_log
Breakpoint 1 at 0x1f6e0: file lib/database.cc, line 426.
(gdb) run
Starting program: /home/grfz/src/notmuch/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 941498 total files (that's not much mail).
q
Breakpoint 1, _notmuch_database_log (notmuch=0x555555825dd0, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
426     {
(gdb) bt
#0  _notmuch_database_log (notmuch=0x555555825dd0, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
#1  0x0000555555577f58 in notmuch_database_add_message (notmuch=notmuch@entry=0x555555825dd0, filename=filename@entry=0x55557b204400 "/home/grfz/Mail/~ml/[hidden email]/cur/1488666048.R6065357166258349555.len:2,",
    message_ret=message_ret@entry=0x7fffffffd048) at lib/database.cc:2597
#2  0x000055555556802f in add_file (state=0x7fffffffd540, filename=0x55557b204400 "/home/grfz/Mail/~ml/[hidden email]/cur/1488666048.R6065357166258349555.len:2,", notmuch=0x555555825dd0) at notmuch-new.c:264
#3  add_files (notmuch=notmuch@entry=0x555555825dd0, path=path@entry=0x555574a15b80 "/home/grfz/Mail/~ml/[hidden email]/cur", state=state@entry=0x7fffffffd540) at notmuch-new.c:599
#4  0x0000555555567b44 in add_files (notmuch=notmuch@entry=0x555555825dd0, path=path@entry=0x55558520f1b0 "/home/grfz/Mail/~ml/[hidden email]", state=state@entry=0x7fffffffd540) at notmuch-new.c:483
#5  0x0000555555567b44 in add_files (notmuch=notmuch@entry=0x555555825dd0, path=path@entry=0x5555638993b0 "/home/grfz/Mail/~ml", state=state@entry=0x7fffffffd540) at notmuch-new.c:483
#6  0x0000555555567b44 in add_files (notmuch=0x555555825dd0, path=path@entry=0x5555557cf710 "/home/grfz/Mail", state=state@entry=0x7fffffffd540) at notmuch-new.c:483
#7  0x00005555555689ed in notmuch_new_command (config=0x5555557ce1d0, argc=<optimized out>, argv=<optimized out>) at notmuch-new.c:1099
#8  0x0000555555561a27 in main (argc=<optimized out>, argv=0x7fffffffda68) at notmuch.c:456
(gdb) continue
Continuing.
Error: A Xapian exception occurred. Halting processing.
Processed 660118 total files in 8h 59m 43s (20 files/sec.).
Added 584749 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
[Inferior 1 (process 14593) exited with code 01]
(gdb) continue
The program is not being run.
(gdb)



Since the gdb output shows a filename I looked at its contents:
There is a second "Content-Length:" -Header which does not make
sense but otherwise should not be a problem.  

Since its an email from an open mailing list I attach it to
this very email.

I then configured notmuch to scan the whole mail folder
/home/grfz/Mail/~ml/[hidden email]:

0 (master=) grfz@len:~/src/notmuch$ gdb --args /home/grfz/src/notmuch/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/grfz/src/notmuch/notmuch...done.
(gdb) b _notmuch_database_log
Breakpoint 1 at 0x1f6e0: file lib/database.cc, line 426.
(gdb) run
Starting program: /home/grfz/src/notmuch/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 114424 total files (that's not much mail).

Breakpoint 1, _notmuch_database_log (notmuch=0x5555557d2780, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
426     {
(gdb) bt
#0  _notmuch_database_log (notmuch=0x5555557d2780, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
#1  0x0000555555577f58 in notmuch_database_add_message (notmuch=notmuch@entry=0x5555557d2780, filename=filename@entry=0x555558118ba0 "/home/grfz/Mail/~ml/[hidden email]/cur/1488666090.R8081408151026683680.len:2,",
    message_ret=message_ret@entry=0x7fffffffd2c8) at lib/database.cc:2597
#2  0x000055555556802f in add_file (state=0x7fffffffd540, filename=0x555558118ba0 "/home/grfz/Mail/~ml/[hidden email]/cur/1488666090.R8081408151026683680.len:2,", notmuch=0x5555557d2780) at notmuch-new.c:264
#3  add_files (notmuch=notmuch@entry=0x5555557d2780, path=path@entry=0x5555557d25f0 "/home/grfz/Mail/~ml/[hidden email]/cur", state=state@entry=0x7fffffffd540) at notmuch-new.c:599
#4  0x0000555555567b44 in add_files (notmuch=0x5555557d2780, path=path@entry=0x5555557d4f90 "/home/grfz/Mail/~ml/[hidden email]", state=state@entry=0x7fffffffd540) at notmuch-new.c:483
#5  0x00005555555689ed in notmuch_new_command (config=0x5555557ce1d0, argc=<optimized out>, argv=<optimized out>) at notmuch-new.c:1099
#6  0x0000555555561a27 in main (argc=<optimized out>, argv=0x7fffffffda68) at notmuch.c:456
(gdb) continue
Continuing.
Error: A Xapian exception occurred. Halting processing.
Processed 69364 total files in 24m 51s (46 files/sec.).
Added 69350 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
[Inferior 1 (process 23253) exited with code 01]
(gdb) continue
The program is not being run.
(gdb)


--> This time it's another file with another email but at closer
inspection it shows it also has two "Content-Length:" -headers.  For
the sake of consistency I attach the file to this very email.


I then created a test maildir folder with the first offending email as
the very only email and configured notmuch to only scan this test
maildir folder: *no problem*.  I the put the second offending email in
the test maildir folder and tried again: *no problem*



I then rescanned the maildir folder with emacs-org-mode mailing list
emails with notmuch from debian stretch:
0 (master=) grfz@len:~/src/notmuch$ rm -rf /home/grfz/Mail/~ml/[hidden email]/.notmuch/
0 (master=) grfz@len:~/src/notmuch$ gdb --args /usr/bin/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/notmuch...(no debugging symbols
found)...done.
(gdb) b _notmuch_database_log
Function "_notmuch_database_log" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (_notmuch_database_log) pending.
(gdb) run
Starting program: /usr/bin/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 114424 total files (that's not much mail).
Error: A Xapian exception occurred. Halting processing.
Processed 69364 total files in 13m 45s (84 files/sec.).
Added 69350 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
[Inferior 1 (process 24459) exited with code 01]
(gdb) bt
No stack.
(gdb) continue
The program is not being run.
(gdb)




Since notmuch from debian strech also produced a xapian exeption I
assume bisecting will be of no help.  I'll try to bisect but the next
week or so I don't have much free time.  I also will run memtest and
smartmontools.


Thanks for your help, Gregor

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch

1488666048.R6065357166258349555.len:2, (4K) Download Attachment
1488666090.R8081408151026683680.len:2, (5K) Download Attachment
David Bremner-2 David Bremner-2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

Gregor Zattler <[hidden email]> writes:

>
> Since its an email from an open mailing list I attach it to
> this very email.
>

It might or might not be related, but I noticed that those two messages
get merged for me into a thread 1734 messages long of org-mode messages
(I have some old org messages, but I think I not the two you mention).

I'm not sure yet if it's a bug in notmuch, or something strange about
mail from the org-mode list.

d

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Gregor Zattler Gregor Zattler
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

Hi David,
* David Bremner <[hidden email]> [09. Jul. 2017]:

> Gregor Zattler <[hidden email]> writes:
>> Since its an email from an open mailing list I attach it to
>> this very email.
>>
>
> It might or might not be related, but I noticed that those two messages
> get merged for me into a thread 1734 messages long of org-mode messages
> (I have some old org messages, but I think I not the two you mention).
>
> I'm not sure yet if it's a bug in notmuch, or something strange about
> mail from the org-mode list.

This *might* be related with some other problem I reported
regarding false threading of messages, as discussed in the thread
containing amongst others id:[hidden email]

Short version: Some of the messages on this mailinglist had very
weired References: headers mot probably causing notmuch to
misbehave while threading the messages.  But then there was no
xapian exeption involved.


Otherwise I did an extensive memory test: no prblem and an
smartctl -t long and -t offline test: no problems.  Remains to do
a bisecting but I have no free time atm.  Will report back later.

Ciao, Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
David Bremner-2 David Bremner-2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

Gregor Zattler <[hidden email]> writes:

> Hi David,
> * David Bremner <[hidden email]> [09. Jul. 2017]:
>> Gregor Zattler <[hidden email]> writes:
>>> Since its an email from an open mailing list I attach it to
>>> this very email.
>>>
>>
>> It might or might not be related, but I noticed that those two messages
>> get merged for me into a thread 1734 messages long of org-mode messages
>> (I have some old org messages, but I think I not the two you mention).
>>
>> I'm not sure yet if it's a bug in notmuch, or something strange about
>> mail from the org-mode list.
>
> This *might* be related with some other problem I reported
> regarding false threading of messages, as discussed in the thread
> containing amongst others id:[hidden email]
>
> Short version: Some of the messages on this mailinglist had very
> weired References: headers mot probably causing notmuch to
> misbehave while threading the messages.  But then there was no
> xapian exeption involved.
>

Right. I looks like there are indeed many message-ids in the resulting
database that look like valid email addresses. So that problem persists,
despite an attempted fix discussed in that thread.

I was wondering if the exception could result from overflowing some
internal limit due to threads many thousands of messages long. I will be
hard to know until I can replicate the exception. Perhaps a more
complete org-mode list archive would do it.

d
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Gregor Zattler Gregor Zattler
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

Hi David, notmuch developers,
* David Bremner <[hidden email]> [09. Jul. 2017]:

> Gregor Zattler <[hidden email]> writes:
>> Short version: Some of the messages on this mailinglist had very
>> weired References: headers mot probably causing notmuch to
>> misbehave while threading the messages.  But then there was no
>> xapian exeption involved.
>>
>
> Right. I looks like there are indeed many message-ids in the resulting
> database that look like valid email addresses. So that problem persists,
> despite an attempted fix discussed in that thread.
>
> I was wondering if the exception could result from overflowing some
> internal limit due to threads many thousands of messages long. I will be
> hard to know until I can replicate the exception. Perhaps a more
> complete org-mode list archive would do it.

Quite possible, when I do
notmuch show  --entire-thread=true --format=mbox path:Mail/~ml/[hidden email]/**   date 64 bit 32 > /tmp/emo.mbox
and open the resulting mbox with mutt, it shows 10199 messages.

There is fun to have regarding message ids:

0 grfz@len:/tmp$ grep -c "^Message-ID:[[:space:]]" eom.mbox
8758
0 grfz@len:/tmp$ grep -ci "^Message-ID:[[:space:]]" eom.mbox
10174

I then split the mbox with mutt in a maildir folder.  This were
10199 individual files.  I then extracted the message ids via
formail from this files and piped this through sort -u.  These
were 10105 message ids.

The maildir with the [hidden email] emails contained 114563
individual files.  I removed every file somewhere containing one
of those message ids.  Now there were only 102164 individual files
in the maildir.

And after that I indexed this cleaned up maildir.  If somehow
this message threading problem affects the indexing it should now
index the files without xapian exeption:


This is with newest notmuch:
0 grfz@len:/tmp$ /home/grfz/src/notmuch/notmuch --version
notmuch 0.24.2+112~g37d1fa5


0 grfz@len:/tmp$ gdb --args /home/grfz/src/notmuch/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/grfz/src/notmuch/notmuch...done.
(gdb) b _notmuch_database_log
Breakpoint 1 at 0x1f6e0: file lib/database.cc, line 426.
(gdb) run
Starting program: /home/grfz/src/notmuch/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 102164 total files (that's not much mail).
Processed 102164 total files in 5m 58s (284 files/sec.).
Added 102151 new messages to the database.
[Inferior 1 (process 26339) exited normally]
(gdb)


While indexing the original emacs-org mode mailing list maildir
with it's 114563 files with the very same binary results in a
xapian exeption:


0 grfz@len:/tmp$ gdb --args /home/grfz/src/notmuch/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/grfz/src/notmuch/notmuch...done.
(gdb) b _notmuch_database_log
Breakpoint 1 at 0x1f6e0: file lib/database.cc, line 426.
(gdb) run
Starting program: /home/grfz/src/notmuch/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 114563 total files (that's not much mail).

Breakpoint 1, _notmuch_database_log (notmuch=0x5555557d2780, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
426     {
(gdb) bt
#0  _notmuch_database_log (notmuch=0x5555557d2780, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
#1  0x0000555555577f58 in notmuch_database_add_message (notmuch=notmuch@entry=0x5555557d2780, filename=filename@entry=0x555557ae2070 "/home/grfz/Mail/~ml/[hidden email]/cur/1488666090.R8081408151026683680.len:2,",
    message_ret=message_ret@entry=0x7fffffffd2c8) at lib/database.cc:2597
#2  0x000055555556802f in add_file (state=0x7fffffffd540, filename=0x555557ae2070 "/home/grfz/Mail/~ml/[hidden email]/cur/1488666090.R8081408151026683680.len:2,", notmuch=0x5555557d2780) at notmuch-new.c:264
#3  add_files (notmuch=notmuch@entry=0x5555557d2780, path=path@entry=0x5555557d25f0 "/home/grfz/Mail/~ml/[hidden email]/cur", state=state@entry=0x7fffffffd540) at notmuch-new.c:599
#4  0x0000555555567b44 in add_files (notmuch=0x5555557d2780, path=path@entry=0x5555557d4f90 "/home/grfz/Mail/~ml/[hidden email]", state=state@entry=0x7fffffffd540) at notmuch-new.c:483
#5  0x00005555555689ed in notmuch_new_command (config=0x5555557ce1d0, argc=<optimized out>, argv=<optimized out>) at notmuch-new.c:1099
#6  0x0000555555561a27 in main (argc=<optimized out>, argv=0x7fffffffda68) at notmuch.c:456
(gdb) continue
Continuing.
Error: A Xapian exception occurred. Halting processing.
Processed 69364 total files in 25m 59s (44 files/sec.).
Added 69350 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
[Inferior 1 (process 27880) exited with code 01]
(gdb) continue
The program is not being run.
(gdb)

The referenced file is the second one I attached in this thread.




So your/Davids intuition was right.  The problem has to do with
this threading problem.


I also did extensive memtest and every kind of smartctl test
possible, to be sure this is no hardware problem.



I then downloaded parts of the archive, merged and sliced it and
now I have a sample of 25001 total files (that's not much mail),
on which notmuch new produces a xapian exeption:

0 (master *) grfz@len:~$ gdb --args /home/grfz/src/notmuch/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/grfz/src/notmuch/notmuch...done.
(gdb) b _notmuch_database_log
Breakpoint 1 at 0x1f6e0: file lib/database.cc, line 426.
(gdb) run
Starting program: /home/grfz/src/notmuch/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 25001 total files (that's not much mail).

Breakpoint 1, _notmuch_database_log (notmuch=0x5555557d27f0, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
426     {
(gdb) bt
#0  _notmuch_database_log (notmuch=0x5555557d27f0, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
#1  0x0000555555577f58 in notmuch_database_add_message (notmuch=notmuch@entry=0x5555557d27f0, filename=filename@entry=0x555558dc96c0 "/tmp/reduced-sample/cur/1499897912.R1843131171398763530.len:2,",
                                                                                              message_ret=message_ret@entry=0x7fffffffd2e8) at lib/database.cc:2597
#2  0x000055555556802f in add_file (state=0x7fffffffd560, filename=0x555558dc96c0 "/tmp/reduced-sample/cur/1499897912.R1843131171398763530.len:2,", notmuch=0x5555557d27f0) at notmuch-new.c:264
#3  add_files (notmuch=notmuch@entry=0x5555557d27f0, path=path@entry=0x5555557e7950 "/tmp/reduced-sample/cur", state=state@entry=0x7fffffffd560) at notmuch-new.c:599
#4  0x0000555555567b44 in add_files (notmuch=0x5555557d27f0, path=path@entry=0x5555557cf710 "/tmp/reduced-sample", state=state@entry=0x7fffffffd560) at notmuch-new.c:483
#5  0x00005555555689ed in notmuch_new_command (config=0x5555557ce1d0, argc=<optimized out>, argv=<optimized out>) at notmuch-new.c:1099
#6  0x0000555555561a27 in main (argc=<optimized out>, argv=0x7fffffffda88) at notmuch.c:456
(gdb)


As a tar.xz it weighs 28 MB.  I could provide this for download
if someone is interested.


Ciao, Gregor

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
David Bremner-2 David Bremner-2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

Gregor Zattler <[hidden email]> writes:

> Hi David, notmuch developers,
> * David Bremner <[hidden email]> [09. Jul. 2017]:
>> Gregor Zattler <[hidden email]> writes:
>>
>> I was wondering if the exception could result from overflowing some
>> internal limit due to threads many thousands of messages long. I will be
>> hard to know until I can replicate the exception. Perhaps a more
>> complete org-mode list archive would do it.
>
> Quite possible, when I do
> notmuch show  --entire-thread=true --format=mbox path:Mail/~ml/[hidden email]/**   date 64 bit 32 > /tmp/emo.mbox
> and open the resulting mbox with mutt, it shows 10199 messages.

Are those literal search terms "date 64 bit 32"?

> As a tar.xz it weighs 28 MB.  I could provide this for download
> if someone is interested.

Sure, send me a download URL.
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Gregor Zattler Gregor Zattler
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

Hi David, notmuch developers,
* David Bremner <[hidden email]> [13. Jul. 2017]:

>> * David Bremner <[hidden email]> [09. Jul. 2017]:
>>> Gregor Zattler <[hidden email]> writes:
>>> I was wondering if the exception could result from overflowing some
>>> internal limit due to threads many thousands of messages long. I will be
>>> hard to know until I can replicate the exception. Perhaps a more
>>> complete org-mode list archive would do it.
>>
>> Quite possible, when I do
>> notmuch show  --entire-thread=true --format=mbox path:Mail/~ml/[hidden email]/**   date 64 bit 32 > /tmp/emo.mbox
>> and open the resulting mbox with mutt, it shows 10199 messages.
>
> Are those literal search terms "date 64 bit 32"?

yes.

>> As a tar.xz it weighs 28 MB.  I could provide this for download
>> if someone is interested.

I'll do so in a private email.  


Ciao, Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
David Bremner-2 David Bremner-2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

In reply to this post by Gregor Zattler
Gregor Zattler <[hidden email]> writes:

>
> I then downloaded parts of the archive, merged and sliced it and
> now I have a sample of 25001 total files (that's not much mail),
> on which notmuch new produces a xapian exeption:
>

> As a tar.xz it weighs 28 MB.  I could provide this for download
> if someone is interested.

For me this corpus does not produce an exception (although it does have
one thread with 5067 messages).

I'm running Debian testing, so there could be slightly different
versions of xapian and gmime. What versions of libxapian-dev and
libgmime-2.6-dev have you built notmuch against?

d
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Gregor Zattler Gregor Zattler
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

Hi David,
* David Bremner <[hidden email]> [13. Jul. 2017]:

> Gregor Zattler <[hidden email]> writes:
>> I then downloaded parts of the archive, merged and sliced it and
>> now I have a sample of 25001 total files (that's not much mail),
>> on which notmuch new produces a xapian exeption:
>>
>
>> As a tar.xz it weighs 28 MB.  I could provide this for download
>> if someone is interested.
>
> For me this corpus does not produce an exception (although it does have
> one thread with 5067 messages).

I cannot view this ATM because I cannot index these emails.  When
viewd with mutt there are no such gargantuan threads.

> I'm running Debian testing, so there could be slightly different
> versions of xapian and gmime. What versions of libxapian-dev and
> libgmime-2.6-dev have you built notmuch against?

Mhm.  I'm running debian stretch.  The versions are:

0 grfz@len:/tmp$ apt-cache policy libxapian-dev libgmime-2.6-dev
libxapian-dev:
Installed: 1.4.3-2
Candidate: 1.4.3-2
Version table:
*** 1.4.3-2 500
500 http://deb.debian.org/debian stretch/main amd64 Packages
100 /var/lib/dpkg/status
libgmime-2.6-dev:
Installed: 2.6.22+dfsg2-1
Candidate: 2.6.22+dfsg2-1
Version table:
*** 2.6.22+dfsg2-1 500
500 http://deb.debian.org/debian stretch/main amd64 Packages
100 /var/lib/dpkg/status


BTW:


0 grfz@len:/tmp$ apt-cache policy gcc
gcc:
Installed: 4:6.3.0-4
Candidate: 4:6.3.0-4
Version table:
*** 4:6.3.0-4 500
500 http://deb.debian.org/debian stretch/main amd64 Packages
100 /var/lib/dpkg/status
0 grfz@len:/tmp$ gcc --version
gcc (Debian 6.3.0-18) 6.3.0 20170516



       

HTH, Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
David Bremner-2 David Bremner-2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

Gregor Zattler <[hidden email]> writes:

> Hi David,
> * David Bremner <[hidden email]> [13. Jul. 2017]:
>> Gregor Zattler <[hidden email]> writes:
>>> I then downloaded parts of the archive, merged and sliced it and
>>> now I have a sample of 25001 total files (that's not much mail),
>>> on which notmuch new produces a xapian exeption:
>>>
>>
>>> As a tar.xz it weighs 28 MB.  I could provide this for download
>>> if someone is interested.
>>
>> For me this corpus does not produce an exception (although it does have
>> one thread with 5067 messages).
>
> I cannot view this ATM because I cannot index these emails.  When
> viewd with mutt there are no such gargantuan threads.
>
>> I'm running Debian testing, so there could be slightly different
>> versions of xapian and gmime. What versions of libxapian-dev and
>> libgmime-2.6-dev have you built notmuch against?
>
> Mhm.  I'm running debian stretch.  The versions are:
>

I tried on a debian stretch with the same library versions, and I still
can't duplicate the crash.
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Gregor Zattler Gregor Zattler
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

Hi David,
* David Bremner <[hidden email]> [13. Jul. 2017]:
>> * David Bremner <[hidden email]> [13. Jul. 2017]:
>>> Gregor Zattler <[hidden email]> writes:
>>>> I then downloaded parts of the archive, merged and sliced it and
>>>> now I have a sample of 25001 total files (that's not much mail),
>>>> on which notmuch new produces a xapian exeption:
[...]
>>> For me this corpus does not produce an exception (although it does have
>>> one thread with 5067 messages).

>>> I'm running Debian testing, so there could be slightly different
>>> versions of xapian and gmime. What versions of libxapian-dev and
>>> libgmime-2.6-dev have you built notmuch against?
[...]
> I tried on a debian stretch with the same library versions, and I still
> can't duplicate the crash.

This is strange.  I rerun the self compiled notmuch new on this
corpus and it did not throw an exeption, nore did the debian
stretch binary.  I have no idea why this is different with the
first run.

On my way reducing the corpus I had an intermediate corpus of
~43000 files on which notmuch new produced a xapian exeption.  I
kept this corpus but did not include the test results in my email
because the later, smaller corpus seemed more interesting.

I redid the test on this mid-sized corpus and I got an
exeption (again):

0 (master *) grfz@len:~$ gdb --args /home/grfz/src/notmuch/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/grfz/src/notmuch/notmuch...done.
(gdb) b _notmuch_database_log
Breakpoint 1 at 0x1f6e0: file lib/database.cc, line 426.
(gdb) run
Starting program: /home/grfz/src/notmuch/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 43255 total files (that's not much mail).

Breakpoint 1, _notmuch_database_log (notmuch=0x5555557d2820, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
426     {
(gdb) bt
#0  _notmuch_database_log (notmuch=0x5555557d2820, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
#1  0x0000555555577f58 in notmuch_database_add_message (notmuch=notmuch@entry=0x5555557d2820, filename=filename@entry=0x555556fa4870 "/tmp/reduced-sample.xapian-exeption/new/1499897795.R10126604875560291557.len",
                                                                                              message_ret=message_ret@entry=0x7fffffffd2e8) at lib/database.cc:2597
#2  0x000055555556802f in add_file (state=0x7fffffffd560, filename=0x555556fa4870 "/tmp/reduced-sample.xapian-exeption/new/1499897795.R10126604875560291557.len", notmuch=0x5555557d2820) at notmuch-new.c:264
#3  add_files (notmuch=notmuch@entry=0x5555557d2820, path=path@entry=0x5555557d26e0 "/tmp/reduced-sample.xapian-exeption/new", state=state@entry=0x7fffffffd560) at notmuch-new.c:599
#4  0x0000555555567b44 in add_files (notmuch=0x5555557d2820, path=path@entry=0x5555557cf730 "/tmp/reduced-sample.xapian-exeption", state=state@entry=0x7fffffffd560) at notmuch-new.c:483
#5  0x00005555555689ed in notmuch_new_command (config=0x5555557ce1d0, argc=<optimized out>, argv=<optimized out>) at notmuch-new.c:1099
#6  0x0000555555561a27 in main (argc=<optimized out>, argv=0x7fffffffda88) at notmuch.c:456
(gdb) continue
Continuing.
Error: A Xapian exception occurred. Halting processing.
Processed 27611 total files in 21m 2s (21 files/sec.).
Added 27481 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
[Inferior 1 (process 6752) exited with code 01]
(gdb)



I redid the test with this medium sized corpus with the notmuch
binary from debian stretch, the xapian exeption hits after the
same number of scanned/added files:

0 (master *) grfz@len:~$ gdb --args /usr/bin/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/notmuch...(no debugging symbols found)...done.
(gdb) b _notmuch_database_log
Function "_notmuch_database_log" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (_notmuch_database_log) pending.
(gdb) run
Starting program: /usr/bin/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 43255 total files (that's not much mail).
Error: A Xapian exception occurred. Halting processing.
Processed 27611 total files in 10m 21s (44 files/sec.).
Added 27481 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
[Inferior 1 (process 10408) exited with code 01]
(gdb) bt
No stack.
(gdb) continue
The program is not being run.
(gdb) quite
Undefined command: "quite".  Try "help".
(gdb) quit



In case your' not exhausted, the mid-sized corpus is here:
https://giku.de/reduced-sample-midsized.xapian-exeption.tar.xz

Regards, Gregor

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
David Bremner-2 David Bremner-2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

Gregor Zattler <[hidden email]> writes:

>
> On my way reducing the corpus I had an intermediate corpus of
> ~43000 files on which notmuch new produced a xapian exeption.  I
> kept this corpus but did not include the test results in my email
> because the later, smaller corpus seemed more interesting.
>
> I redid the test on this mid-sized corpus and I got an
> exeption (again):
[snip]
> Error: A Xapian exception occurred. Halting processing.
> Processed 27611 total files in 21m 2s (21 files/sec.).
> Added 27481 new messages to the database.
> Note: A fatal error was encountered: A Xapian exception occurred
> [Inferior 1 (process 6752) exited with code 01]
> (gdb)

I can finally replicate the crash. I do crash in a different place
(after processing 25184 files), so it suggests a certain non-determinism
or dependence on the host configuration.

Anyway, now that I have a test case, I'll try and figure out what the
underlying Xapian exeception is.

d
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Gregor Zattler Gregor Zattler
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Bug: fatal error with notmuch new, second run starts indexing all over again

Hi David,
* David Bremner <[hidden email]> [14. Jul. 2017]:

> Gregor Zattler <[hidden email]> writes:
>> I redid the test on this mid-sized corpus and I got an
>> exeption (again):
> [snip]
>> Error: A Xapian exception occurred. Halting processing.
>> Processed 27611 total files in 21m 2s (21 files/sec.).
>> Added 27481 new messages to the database.
>> Note: A fatal error was encountered: A Xapian exception occurred
>> [Inferior 1 (process 6752) exited with code 01]
>> (gdb)
>
> I can finally replicate the crash. I do crash in a different place
> (after processing 25184 files), so it suggests a certain non-determinism
> or dependence on the host configuration.

I would assume the first difference is in the file system:
Perhaps the files are accessed in a different order!?

> Anyway, now that I have a test case, I'll try and figure out what the
> underlying Xapian exeception is.


Thanks.

Ciao, Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Loading...