Inconsistent query results

classic Classic list List threaded Threaded
5 messages Options
Kirill A. Shutemov Kirill A. Shutemov
Reply | Threaded
Open this post in threaded view
|

Inconsistent query results

Hello,

I found that on particular queries notmuch return different results if run
the query few times. Re-initialing the query or db doesn't help.

I've attached test case along with corpus of messages.

Unpack the archive and run `make' there. It will initialize the notmuch
database for the corpus, build and run the test-case.

--
 Kirill A. Shutemov

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch

test.tar.xz (228K) Download Attachment
David Bremner-2 David Bremner-2
Reply | Threaded
Open this post in threaded view
|

Re: Inconsistent query results

"Kirill A. Shutemov" <[hidden email]> writes:

> Hello,
>
> I found that on particular queries notmuch return different results if run
> the query few times. Re-initialing the query or db doesn't help.
>
> I've attached test case along with corpus of messages.
>
> Unpack the archive and run `make' there. It will initialize the notmuch
> database for the corpus, build and run the test-case.

Thanks for the report. I don't yet understand where the bug is, but I
think it's safe to say it's not in your code. I made a somewhat simpler
test case that displays the same problem (at the end).

I'm also fairly sure this is different than the exclude related bug I
recently fixed in notmuch, since running your test under
"NOTMUCH_DEBUG_QUERY=yes ./test" shows the same xapian query is used
both times.

One thing I noticed is that if run both your test case and mine under
valgrind, I get a report of some uninitialized memory. The reports are
similar in both cases, here is part of the report from my test case

==11180== Conditional jump or move depends on uninitialised value(s)
==11180==    at 0x5F5D0B1: OrPostList::check(unsigned int, double, bool&) (orpostlist.cc:198)
==11180==    by 0x5F4E171: check_helper (multiandpostlist.h:97)
==11180==    by 0x5F4E171: MultiAndPostList::find_next_match(double) (multiandpostlist.cc:217)
==11180==    by 0x5F44C3F: skip_to_handling_prune (branchpostlist.h:98)
==11180==    by 0x5F44C3F: AndNotPostList::advance_to_next_match(double, Xapian::PostingIterator::Internal*) (andnotpostlist.cc:50)
==11180==    by 0x5F4E317: next_helper (multiandpostlist.h:76)
==11180==    by 0x5F4E317: MultiAndPostList::next(double) (multiandpostlist.cc:238)
==11180==    by 0x5F4FACC: next_handling_prune (branchpostlist.h:85)
==11180==    by 0x5F4FACC: MultiMatch::get_mset(unsigned int, unsigned int, unsigned int, Xapian::MSet&, Xapian::Weight::Internal&, Xapian::MatchDecider const*, Xapian::KeyMaker const*) (multimatch.cc:570)
==11180==    by 0x5E485CE: Xapian::Enquire::Internal::get_mset(unsigned int, unsigned int, unsigned int, Xapian::RSet const*, Xapian::MatchDecider const*) const (omenquire.cc:581)
==11180==    by 0x5E48913: Xapian::Enquire::get_mset(unsigned int, unsigned int, unsigned int, Xapian::RSet const*, Xapian::MatchDecider const*) const (omenquire.cc:939)
==11180==    by 0x4E52FB8: _notmuch_query_count_documents (query.cc:679)
==11180==    by 0x108A99: doit (test.c:17)
==11180==    by 0x108B0C: main (test.c:28)
==11180==
count1: 2, count2: 2

Notice that under valgrind the counts match, which strongly suggests
that whatever is going on here is related to a memory error.

In case someone on the xapian list wants to play with this, you can grab
Kirill's test corpus and driver from

wget http://notmuchmail.org/pipermail/notmuch/attachments/20170308/fa83965a/attachment-0001.xz
tar Jxvf attachment-0001.xz


#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <notmuch.h>

const char *query = "tag:unread to:[hidden email] -(tag:unread (shutemov or tag:followed))";
//const char *query = "tag:unread to:[hidden email] -(shutemov or tag:followed)";

int main(int argc, char **argv)
{
    notmuch_database_t *db;
    notmuch_query_t *q;
    notmuch_threads_t *threads;
    int count1 = 0, count2 = 0;

    notmuch_database_open(DB_PATH, 0, &db);
    q = notmuch_query_create(db, query);

    for (notmuch_query_search_threads_st(q, &threads);
            notmuch_threads_valid(threads);
            notmuch_threads_move_to_next(threads)) {
        count1++;
    }

    notmuch_threads_destroy(threads);
    notmuch_query_destroy(q);
    notmuch_database_close(db);

    notmuch_database_open(DB_PATH, 0, &db);
    q = notmuch_query_create(db, query);

    for (notmuch_query_search_threads_st(q, &threads);
            notmuch_threads_valid(threads);
            notmuch_threads_move_to_next(threads)) {
        count2++;
    }

    notmuch_threads_destroy(threads);
    notmuch_query_destroy(q);
    notmuch_database_close(db);

    printf("count1: %d, count2: %d\n", count1, count2);
    if (count1 != count2)
        printf("WTF?\n");
    return 0;
}

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
David Bremner-2 David Bremner-2
Reply | Threaded
Open this post in threaded view
|

Re: Inconsistent query results

David Bremner <[hidden email]> writes:

>
> Thanks for the report. I don't yet understand where the bug is, but I
> think it's safe to say it's not in your code. I made a somewhat simpler
> test case that displays the same problem (at the end).
>

Oops, I cleverly wiped out my modified test case test-unpacking the
attachment. Luckily I still had it in a buffer. Here it is


#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <assert.h>
#include <notmuch.h>

const char *query = "tag:unread to:[hidden email] -(tag:unread (shutemov or tag:followed))";
//const char *query = "tag:unread to:[hidden email] -(shutemov or tag:followed)";

static void
doit(unsigned int *out) {
  notmuch_database_t *db;
  notmuch_query_t *q;

  assert (NOTMUCH_STATUS_SUCCESS == notmuch_database_open(DB_PATH, 0, &db));
  assert (q = notmuch_query_create(db, query));
  assert (NOTMUCH_STATUS_SUCCESS == notmuch_query_count_messages_st (q, out));
  notmuch_query_destroy(q);
  notmuch_database_close(db);
  notmuch_database_destroy(db);
}

 
int main(int argc, char **argv)
{
  unsigned int count1 = 0, count2 = 0;

    doit(&count1);
    doit(&count2);
    printf("count1: %d, count2: %d\n", count1, count2);

    return (count1 != count2);
}

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Olly Betts Olly Betts
Reply | Threaded
Open this post in threaded view
|

Re: Inconsistent query results

In reply to this post by David Bremner-2
On Wed, Mar 08, 2017 at 10:32:56PM -0400, David Bremner wrote:
> "Kirill A. Shutemov" <[hidden email]> writes:
> > I found that on particular queries notmuch return different results if run
> > the query few times. Re-initialing the query or db doesn't help.
>
> Thanks for the report. I don't yet understand where the bug is, but I
> think it's safe to say it's not in your code. I made a somewhat simpler
> test case that displays the same problem (at the end).

It's a bug in Xapian - I've committed a fix to master (commit
fa12a83957e97349aa6e2a6c0896faf210dfe4b4) which I'll backport for 1.4.4 and
1.2.25 (it also affects 1.2.x).

To trigger it you need an AND operator which is unweighted (e.g. being on the
right side of AND_NOT here) with a subquery which uses the passed max weight
value (the obvious case is another operator, OR in this case).

(David already confirmed on IRC that the fix applied solves this for him.)

Cheers,
    Olly
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Kirill A. Shutemov Kirill A. Shutemov
Reply | Threaded
Open this post in threaded view
|

Re: Inconsistent query results

On Fri, Mar 10, 2017 at 02:56:03AM +0000, Olly Betts wrote:

> On Wed, Mar 08, 2017 at 10:32:56PM -0400, David Bremner wrote:
> > "Kirill A. Shutemov" <[hidden email]> writes:
> > > I found that on particular queries notmuch return different results if run
> > > the query few times. Re-initialing the query or db doesn't help.
> >
> > Thanks for the report. I don't yet understand where the bug is, but I
> > think it's safe to say it's not in your code. I made a somewhat simpler
> > test case that displays the same problem (at the end).
>
> It's a bug in Xapian - I've committed a fix to master (commit
> fa12a83957e97349aa6e2a6c0896faf210dfe4b4) which I'll backport for 1.4.4 and
> 1.2.25 (it also affects 1.2.x).
>
> To trigger it you need an AND operator which is unweighted (e.g. being on the
> right side of AND_NOT here) with a subquery which uses the passed max weight
> value (the obvious case is another operator, OR in this case).
>
> (David already confirmed on IRC that the fix applied solves this for him.)

Verified. Thanks!

--
 Kirill A. Shutemov
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch