experimenting with pytest tests

classic Classic list List threaded Threaded
10 messages Options
Floris Bruynooghe Floris Bruynooghe
Reply | Threaded
Open this post in threaded view
|

experimenting with pytest tests

Hi,

From another conversation on this list I've dug up my earlier attempts
at integrating a pytest run into the notmuch testing suite.  I'd be
grateful for some guidance on whether this is the right way to go about
things.

Here's the diff of configure:

diff --git a/configure b/configure
index c5e2ffed..8aa57b83 100755
--- a/configure
+++ b/configure
@@ -567,6 +567,28 @@ if [ $have_python -eq 0 ]; then
     errors=$((errors + 1))
 fi
 
+check_python() {
+    local bin=$1
+    local var=$(echo $bin | tr -d '.')
+    printf "Checking for $bin (with: pytest)... "
+    if command -v $bin > /dev/null; then
+        if $bin -c 'import pytest' >/dev/null 2>&1; then
+            eval have_$var=1
+            eval $var=$bin
+            printf "Yes.\n"
+        else
+            printf "No (skipping $bin tests).\n"
+        fi
+    else
+        printf "No (skipping $bin tests).\n"
+    fi
+}
+
+check_python python2.7
+check_python python3.5
+check_python python3.6
+check_python pypy3.5
+
 printf "Checking for valgrind development files... "
 if pkg-config --exists valgrind; then
     printf "Yes.\n"
@@ -1209,6 +1231,10 @@ NOTMUCH_HAVE_MAN=$((have_sphinx))
 
 # Name of python interpreter
 NOTMUCH_PYTHON=${python}
+NOTMUCH_PYTHON27=${python27-}
+NOTMUCH_PYTHON35=${python35-}
+NOTMUCH_PYTHON36=${python36-}
+NOTMUCH_PYPY35=${pypy35-}


And I was then also trying to introduce a test/T391-pytest.sh file.
What I'm trying to do in this file, but it doesn't currently work, is to
have one test, with 4 subtests where each subtest is a pytest run with a
particular python version.  But if the NOTMUCH_PYTHON27 (etc) is not
found in sh.config then the subtest should be skipped.  I'm not really
familiar enough with test-lib.sh to do this correctly without a bunch of
more looking into it, but maybe someone else knows how to do this?
Anyway, here my failing attempt:

#!/usr/bin/env bash
test_description="python unittests"
. ./test-lib.sh || exit 1


test_require_external_prereq "${NOTMUCH_PYTHON27}" && {
    test_begin_subtest "${NOTMUCH_PYTHON27}"
    (
        cd "$NOTMUCH_SRCDIR/bindings/python"
        PYTHONPATH=".${PYTHONPATH:+:$PYTHONPATH}" \
        $NOTMUCH_PYTHON27 -m pytest
    )
    test_expect_equal $? 0
}


test_require_external_prereq ${NOTMUCH_PYTHON35} && {
    test_begin_subtest "${NOTMUCH_PYTHON35}"
    (
        cd "$NOTMUCH_SRCDIR/bindings/python"
        PYTHONPATH=".${PYTHONPATH:+:$PYTHONPATH}" \
        $NOTMUCH_PYTHON35 -m pytest
    )
    test_expect_equal $? 0
}


test_done


Any tips on whether this is the right direction?

Many thanks,
Floris
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Tomi Ollila-2 Tomi Ollila-2
Reply | Threaded
Open this post in threaded view
|

Re: experimenting with pytest tests

On Sat, Apr 07 2018, Floris Bruynooghe wrote:

> Hi,
>
> From another conversation on this list I've dug up my earlier attempts
> at integrating a pytest run into the notmuch testing suite.  I'd be
> grateful for some guidance on whether this is the right way to go about
> things.

good stuff, some comments inline

>
> Here's the diff of configure:
>
> diff --git a/configure b/configure
> index c5e2ffed..8aa57b83 100755
> --- a/configure
> +++ b/configure
> @@ -567,6 +567,28 @@ if [ $have_python -eq 0 ]; then
>      errors=$((errors + 1))
>  fi
>  
> +check_python() {

perhaps check_pytest()

> +    local bin=$1
> +    local var=$(echo $bin | tr -d '.')

I'm not so happy of the above, but good enough in this context,
alternarives are somewhat overkill

(e.g. calling like `check_python -v python27 python2.7`
 or even `check_python python2 7` or  `check_python python 2 7` or
 `check_python python 2 . 7` ;)

> +    printf "Checking for $bin (with: pytest)... "
> +    if command -v $bin > /dev/null; then
> +        if $bin -c 'import pytest' >/dev/null 2>&1; then
> +            eval have_$var=1
> +            eval $var=$bin
> +            printf "Yes.\n"
> +        else
> +            printf "No (skipping $bin tests).\n"
> +        fi
> +    else
> +        printf "No (skipping $bin tests).\n"
> +    fi
> +}
> +
> +check_python python2.7

perhaps check_python python3.4 -- that is what EPEL for RHEL/centos7
provides (hmm, it seems python 3.6 is coming to EPEL, see
http://www.nic.funet.fi/pub/mirrors/fedora.redhat.com/pub/epel/7/x86_64/Packages/p/
(or any other mirror), anyway that looks a bit incomplete so far)

> +check_python python3.5
> +check_python python3.6
> +check_python pypy3.5
> +
>  printf "Checking for valgrind development files... "
>  if pkg-config --exists valgrind; then
>      printf "Yes.\n"
> @@ -1209,6 +1231,10 @@ NOTMUCH_HAVE_MAN=$((have_sphinx))
>  
>  # Name of python interpreter
>  NOTMUCH_PYTHON=${python}
> +NOTMUCH_PYTHON27=${python27-}
> +NOTMUCH_PYTHON35=${python35-}
> +NOTMUCH_PYTHON36=${python36-}
> +NOTMUCH_PYPY35=${pypy35-}

perhaps NOTMUCH_PYTEST_PYPY35=${pytest_pypy35-} ...

> And I was then also trying to introduce a test/T391-pytest.sh file.
> What I'm trying to do in this file, but it doesn't currently work, is to
> have one test, with 4 subtests where each subtest is a pytest run with a
> particular python version.  But if the NOTMUCH_PYTHON27 (etc) is not
> found in sh.config then the subtest should be skipped.  I'm not really
> familiar enough with test-lib.sh to do this correctly without a bunch of
> more looking into it, but maybe someone else knows how to do this?
> Anyway, here my failing attempt:

We could start just with if test -n "$NOTMUCH_PYTEST_PYTHON27"
and iterate from that...


> #!/usr/bin/env bash
> test_description="python unittests"
> . ./test-lib.sh || exit 1
>
>
> test_require_external_prereq "${NOTMUCH_PYTHON27}" && {
>     test_begin_subtest "${NOTMUCH_PYTHON27}"
>     (
>         cd "$NOTMUCH_SRCDIR/bindings/python"
>         PYTHONPATH=".${PYTHONPATH:+:$PYTHONPATH}" \
> $NOTMUCH_PYTHON27 -m pytest
>     )
>     test_expect_equal $? 0
> }

The above could look like...

run_pytest() {
     test_begin_subtest "$1"

     PYTHONPATH="$NOTMUCH_SRCDIR/bindings/python${PYTHONPATH:+:$PYTHONPATH}" \
  "$1" -m pytest $NOTMUCH_SRCDIR/bindings/python

     test_expect_equal $? 0
}

if test -n "$NOTMUCH_PYTEST_PYTHON27"; then
     run_pytest "$NOTMUCH_PYTEST_PYTHON27"
fi

(dropped short-circuit &&, since if we ever added `set -e` the
 unhandled || -case would make execution stop there)

(we might want to add export PYTHONDONTWRITEBYTECODE=donot so
$NOTMUCH_SRCDIR/bindings/python is not polluted with *.pyc files,
and we might want to think how to handle all the output files
pytest spits out...)

if the whole test script is for pytest run, PYTHONPATH could
also be put into environment variable "global" to this
particular script.

> test_require_external_prereq "${NOTMUCH_PYTHON27}" && {
>     test_begin_subtest "${NOTMUCH_PYTHON27}"
>     (
>         cd "$NOTMUCH_SRCDIR/bindings/python"
>         PYTHONPATH=".${PYTHONPATH:+:$PYTHONPATH}" \
> $NOTMUCH_PYTHON27 -m pytest
>     )
>     test_expect_equal $? 0
> }

> test_require_external_prereq ${NOTMUCH_PYTHON35} && {
>     test_begin_subtest "${NOTMUCH_PYTHON35}"
>     (
>         cd "$NOTMUCH_SRCDIR/bindings/python"
>         PYTHONPATH=".${PYTHONPATH:+:$PYTHONPATH}" \
> $NOTMUCH_PYTHON35 -m pytest
>     )
>     test_expect_equal $? 0
> }
>
>
> test_done
>
>
> Any tips on whether this is the right direction?
>
> Many thanks,
> Floris
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
David Bremner-2 David Bremner-2
Reply | Threaded
Open this post in threaded view
|

remix of pytest runner

I didn't love the idea of an evergrowing list of variables for
versions of python, since to add or delete a version we'd have to
update several places, even with the use of functions.  This version
uses one variable, which because configure is supposed to be fairly
portable is a space seperated list of names of python binaries to run.
It also replaces inlines functions into the bodies of loops, not for
efficiency but just to avoid some (perceived?) expansion subtleties.

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
David Bremner-2 David Bremner-2
Reply | Threaded
Open this post in threaded view
|

[PATCH 1/2] configure: detect which versions of python can run pytest

Based on a patch from Florian [1].

[1]: id:[hidden email]
---
 configure | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/configure b/configure
index b177b141..c14e6f8b 100755
--- a/configure
+++ b/configure
@@ -571,6 +571,24 @@ if [ $have_python -eq 0 ]; then
     errors=$((errors + 1))
 fi
 
+pytest_pythons=""
+if [ $have_python -eq 1 ]; then
+    for bin in python2.7 python3.5 python3.6 pypy3.5; do
+        printf "Checking for pytest ($bin)... "
+        if command -v $bin > /dev/null; then
+            if $bin -c 'import pytest' >/dev/null 2>&1; then
+                pytest_pythons="${pytest_pythons} $bin"
+                printf "Yes.\n"
+            else
+                printf "No (skipping $bin tests).\n"
+            fi
+        else
+            printf "No (skipping $bin tests).\n"
+        fi
+    done
+    pytest_pythons="${pytest_pythons# }"
+fi
+
 printf "Checking for valgrind development files... "
 if pkg-config --exists valgrind; then
     printf "Yes.\n"
@@ -808,7 +826,6 @@ else
 fi
 rm -f compat/have_canonicalize_file_name
 
-
 printf "Checking for getline... "
 if ${CC} -o compat/have_getline "$srcdir"/compat/have_getline.c > /dev/null 2>&1
 then
@@ -1233,6 +1250,7 @@ NOTMUCH_HAVE_MAN=$((have_sphinx))
 
 # Name of python interpreter
 NOTMUCH_PYTHON=${python}
+NOTMUCH_PYTEST_PYTHONS='${pytest_pythons}'
 
 # Are the ruby development files (and ruby) available? If not skip
 # building/testing ruby bindings.
--
2.16.3

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
David Bremner-2 David Bremner-2
Reply | Threaded
Open this post in threaded view
|

[PATCH 2/2] test: pytest runner for the test suite

In reply to this post by David Bremner-2
The 'test_subtest_known_broken' should be removed when there are
actual tests to run.

Based on a function from Tomi [1]

[1]: id:[hidden email]
---
 test/T391-pytest.sh | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
 create mode 100755 test/T391-pytest.sh

diff --git a/test/T391-pytest.sh b/test/T391-pytest.sh
new file mode 100755
index 00000000..9ac7aabe
--- /dev/null
+++ b/test/T391-pytest.sh
@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+test_description="python bindings (pytest)"
+. $(dirname "$0")/test-lib.sh || exit 1
+
+test_require_external_prereq ${NOTMUCH_PYTHON}
+
+for bin in ${NOTMUCH_PYTEST_PYTHONS}; do
+    test_begin_subtest "pytest ($bin)"
+      test_subtest_known_broken
+       PYTHONPATH="$NOTMUCH_SRCDIR/bindings/python${PYTHONPATH:+:$PYTHONPATH}" \
+                 test_expect_success "$bin -m pytest $NOTMUCH_SRCDIR/bindings/python"
+done
+
+test_done
--
2.16.3

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Tomi Ollila-2 Tomi Ollila-2
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 1/2] configure: detect which versions of python can run pytest

In reply to this post by David Bremner-2
On Sat, Apr 07 2018, David Bremner wrote:

> Based on a patch from Florian [1].
>
> [1]: id:[hidden email]
> ---
>  configure | 20 +++++++++++++++++++-
>  1 file changed, 19 insertions(+), 1 deletion(-)
>
> diff --git a/configure b/configure
> index b177b141..c14e6f8b 100755
> --- a/configure
> +++ b/configure

...

>  # Name of python interpreter
>  NOTMUCH_PYTHON=${python}
> +NOTMUCH_PYTEST_PYTHONS='${pytest_pythons}'

NOTMUCH_PYTEST_PYTHONS="${pytest_pythons}"

_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Tomi Ollila-2 Tomi Ollila-2
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 1/2] configure: detect which versions of python can run pytest

On Sun, Apr 08 2018, Tomi Ollila wrote:

> On Sat, Apr 07 2018, David Bremner wrote:
>
>> Based on a patch from Florian [1].
>>
>> [1]: id:[hidden email]
>> ---
>>  configure | 20 +++++++++++++++++++-
>>  1 file changed, 19 insertions(+), 1 deletion(-)
>>
>> diff --git a/configure b/configure
>> index b177b141..c14e6f8b 100755
>> --- a/configure
>> +++ b/configure
>
> ...
>
>>  # Name of python interpreter
>>  NOTMUCH_PYTHON=${python}
>> +NOTMUCH_PYTEST_PYTHONS='${pytest_pythons}'
>
> NOTMUCH_PYTEST_PYTHONS="${pytest_pythons}"

Argh, forget this >;/

Tomi
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Floris Bruynooghe Floris Bruynooghe
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 2/2] test: pytest runner for the test suite

In reply to this post by David Bremner-2
This series looks good to me, would be great to have!  Do you want to
commit them this or should I just incorporate it and submit together
with tests once actual tests exist.  You could always commit with a ``def
test_dummy(): assert True`` or something if you like.

Thanks!
Floris


On Sat 07 Apr 2018 at 18:39 -0300, David Bremner wrote:

> The 'test_subtest_known_broken' should be removed when there are
> actual tests to run.
>
> Based on a function from Tomi [1]
>
> [1]: id:[hidden email]
> ---
>  test/T391-pytest.sh | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
>  create mode 100755 test/T391-pytest.sh
>
> diff --git a/test/T391-pytest.sh b/test/T391-pytest.sh
> new file mode 100755
> index 00000000..9ac7aabe
> --- /dev/null
> +++ b/test/T391-pytest.sh
> @@ -0,0 +1,14 @@
> +#!/usr/bin/env bash
> +test_description="python bindings (pytest)"
> +. $(dirname "$0")/test-lib.sh || exit 1
> +
> +test_require_external_prereq ${NOTMUCH_PYTHON}
> +
> +for bin in ${NOTMUCH_PYTEST_PYTHONS}; do
> +    test_begin_subtest "pytest ($bin)"
> +      test_subtest_known_broken
> +       PYTHONPATH="$NOTMUCH_SRCDIR/bindings/python${PYTHONPATH:+:$PYTHONPATH}" \
> +                 test_expect_success "$bin -m pytest $NOTMUCH_SRCDIR/bindings/python"
> +done
> +
> +test_done
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Tomi Ollila-2 Tomi Ollila-2
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 2/2] test: pytest runner for the test suite

On Sun, Apr 08 2018, Floris Bruynooghe wrote:

> This series looks good to me, would be great to have!  Do you want to
> commit them this or should I just incorporate it and submit together
> with tests once actual tests exist.  You could always commit with a ``def
> test_dummy(): assert True`` or something if you like.

I'd say incorporate and submit, have some real tests, remove that
*known_broken* line and so on...

Tomi


>
> Thanks!
> Floris
>
>
> On Sat 07 Apr 2018 at 18:39 -0300, David Bremner wrote:
>
>> The 'test_subtest_known_broken' should be removed when there are
>> actual tests to run.
>>
>> Based on a function from Tomi [1]
>>
>> [1]: id:[hidden email]
>> ---
>>  test/T391-pytest.sh | 14 ++++++++++++++
>>  1 file changed, 14 insertions(+)
>>  create mode 100755 test/T391-pytest.sh
>>
>> diff --git a/test/T391-pytest.sh b/test/T391-pytest.sh
>> new file mode 100755
>> index 00000000..9ac7aabe
>> --- /dev/null
>> +++ b/test/T391-pytest.sh
>> @@ -0,0 +1,14 @@
>> +#!/usr/bin/env bash
>> +test_description="python bindings (pytest)"
>> +. $(dirname "$0")/test-lib.sh || exit 1
>> +
>> +test_require_external_prereq ${NOTMUCH_PYTHON}
>> +
>> +for bin in ${NOTMUCH_PYTEST_PYTHONS}; do
>> +    test_begin_subtest "pytest ($bin)"
>> +      test_subtest_known_broken
>> +       PYTHONPATH="$NOTMUCH_SRCDIR/bindings/python${PYTHONPATH:+:$PYTHONPATH}" \
>> +                 test_expect_success "$bin -m pytest $NOTMUCH_SRCDIR/bindings/python"
>> +done
>> +
>> +test_done
> _______________________________________________
> notmuch mailing list
> [hidden email]
> https://notmuchmail.org/mailman/listinfo/notmuch
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch
Floris Bruynooghe Floris Bruynooghe
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 2/2] test: pytest runner for the test suite

In reply to this post by Floris Bruynooghe
On Sun 08 Apr 2018 at 19:14 -0300, David Bremner wrote:

> Floris Bruynooghe <[hidden email]> writes:
>
>> This series looks good to me, would be great to have!  Do you want to
>> commit them this or should I just incorporate it and submit together
>> with tests once actual tests exist.  You could always commit with a ``def
>> test_dummy(): assert True`` or something if you like.
>>
>
> For now, why don't you just incorporate them. Maybe you'll discover some
> issues with them as you work up some real tests.

Sure, that works.

> BTW, it seems like a
> reasonable plan to get a set of unit tests for the existing bindings
> first to help with migration. Is that what you had in mind?

Yes that's what I had in mind.  To safely swap out ctypes for cffi
adding tests to ensure the existing API behaviour remains is a must.

Which reminds me that it's probably worth calling this out explicitly
again: this works implies that cffi will become an external dependency
[0] for the existing bindings.  Just want to make sure this doesn't
become an issue further down the line.

Cheers,
Floris

[0] Technically only on cpython, it's bundled with pypy.
_______________________________________________
notmuch mailing list
[hidden email]
https://notmuchmail.org/mailman/listinfo/notmuch