Using valgrind as part of a daily auto-test script?

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Using valgrind as part of a daily auto-test script?

Jeremy Friesner
Hi guys,

We're planning on enhancing our daily-autobuild process to include some daily
automated testing also, and it would be cool if we could have the unit tests
run under valgrind so that any valgrind-detectable errors show up in the
daily auto-test results.

In order to implement this, I think we would need to somehow tell valgrind the
following:

- If any valgrind-errors are detected, immediately exit with an error code
- If, at program-exit time, any memory leaks are detected, exit with an error
code

Is there any way to tell valgrind to behave like that?  If not, could a flag
of some sort be added to implement this?  Currently it appears that errors
messages are printed to stderr;  I suppose our build script could be made to
parse the stderr output but that would make things more complicated than I
would like to have them.

Thanks,
Jeremy Friesner
Level Control Systems


-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.  How far can you shotput
a projector? How fast can you ride your desk chair down the office luge track?
If you want to score the big prize, get to know the little guy.  
Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
_______________________________________________
Valgrind-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/valgrind-users
Reply | Threaded
Open this post in threaded view
|

Re: Using valgrind as part of a daily auto-test script?

Tom Hughes-2
In message <[hidden email]>
          Jeremy Friesner <[hidden email]> wrote:

> In order to implement this, I think we would need to somehow tell valgrind
> the  following:
>
> - If any valgrind-errors are detected, immediately exit with an error code
> - If, at program-exit time, any memory leaks are detected, exit with an error
> code
>
> Is there any way to tell valgrind to behave like that?  If not, could a
> flag  of some sort be added to implement this?  Currently it appears that
> errors  messages are printed to stderr;  I suppose our build script could
> be made to  parse the stderr output but that would make things more
> complicated than I  would like to have them.

The exit status returned by valgrind is the exit status of the program
being run by valgrind - if valgrind were to replace that with it's own
exit status then you wouldn't know what the original exit status of the
program was.

I think most people doing this just grep the valgrind output for non-zero
error counts.

Tom

--
Tom Hughes ([hidden email])
http://www.compton.nu/


-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.  How far can you shotput
a projector? How fast can you ride your desk chair down the office luge track?
If you want to score the big prize, get to know the little guy.  
Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
_______________________________________________
Valgrind-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/valgrind-users
Reply | Threaded
Open this post in threaded view
|

Re: Using valgrind as part of a daily auto-test script?

jpweber
In reply to this post by Jeremy Friesner
What about questionable leaks?  I'd assert that a little 'more complicated'
in the script can buy you a lot more useful information.

If you don't actually ask to attach gdb, print suppressions, or somesuch,
I think it reliably just runs to completion.  On our project, a python
unit test script runs a slew of test programs without and with valgrind.
Python re.compile calls then extract the error count and various lost
bytes and prints a pretty table.

http://www.freeelectron.org/svn/fe/branch/20050511/test/bld.py

Executable Tests:              norm  sg ex grnd er leak  indir poss  reach
supp
xDL_Loader.exe ................[PASS] 0 0 [PASS] 0 0000  0000  0000  0000
0188
xUnitTest.exe .................[PASS] 0 0 [PASS] 0 0000  0000  0000  0000
0039
xException.exe ................[PASS] 0 0 [PASS] 0 0000  0000  0000  0008
0167
xPlatform.exe .................[PASS] 0 0 [PASS] 0 0000  0000  0000  0000
0039
xList.exe .....................[PASS] 0 0 [PASS] 0 0000  0000  0000  0000
0041
[...]

-- Jason Weber

On Mon, June 13, 2005 3:38 pm, Jeremy Friesner said:

> Hi guys,
>
> We're planning on enhancing our daily-autobuild process to include some
> daily
> automated testing also, and it would be cool if we could have the unit
> tests
> run under valgrind so that any valgrind-detectable errors show up in the
> daily auto-test results.
>
> In order to implement this, I think we would need to somehow tell valgrind
> the
> following:
>
> - If any valgrind-errors are detected, immediately exit with an error code
> - If, at program-exit time, any memory leaks are detected, exit with an
> error
> code
>
> Is there any way to tell valgrind to behave like that?  If not, could a
> flag
> of some sort be added to implement this?  Currently it appears that errors
> messages are printed to stderr;  I suppose our build script could be made
> to
> parse the stderr output but that would make things more complicated than I
> would like to have them.
>
> Thanks,
> Jeremy Friesner
> Level Control Systems
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: NEC IT Guy Games.  How far can you
> shotput
> a projector? How fast can you ride your desk chair down the office luge
> track?
> If you want to score the big prize, get to know the little guy.
> Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
> _______________________________________________
> Valgrind-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/valgrind-users
>




-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.  How far can you shotput
a projector? How fast can you ride your desk chair down the office luge track?
If you want to score the big prize, get to know the little guy.  
Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
_______________________________________________
Valgrind-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/valgrind-users
Reply | Threaded
Open this post in threaded view
|

Re: Using valgrind as part of a daily auto-test script?

njn (Bugzilla)-2
In reply to this post by Tom Hughes-2
On Tue, 14 Jun 2005, Tom Hughes wrote:

>> In order to implement this, I think we would need to somehow tell valgrind
>> the  following:
>>
>> - If any valgrind-errors are detected, immediately exit with an error code
>> - If, at program-exit time, any memory leaks are detected, exit with an error
>> code
>>
>> Is there any way to tell valgrind to behave like that?  If not, could a
>> flag  of some sort be added to implement this?  Currently it appears that
>> errors  messages are printed to stderr;  I suppose our build script could
>> be made to  parse the stderr output but that would make things more
>> complicated than I  would like to have them.
>
> The exit status returned by valgrind is the exit status of the program
> being run by valgrind - if valgrind were to replace that with it's own
> exit status then you wouldn't know what the original exit status of the
> program was.
>
> I think most people doing this just grep the valgrind output for non-zero
> error counts.

Better is to run with -q and use the client requests like
VALGRIND_COUNT_ERRORS and VALGRIND_COUNT_LEAKS within the tests.  Others
have certainly done this -- they may have more advice.

N


-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.  How far can you shotput
a projector? How fast can you ride your desk chair down the office luge track?
If you want to score the big prize, get to know the little guy.  
Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
_______________________________________________
Valgrind-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/valgrind-users
Reply | Threaded
Open this post in threaded view
|

Re: Using valgrind as part of a daily auto-test script?

Mathieu Malaterre
In reply to this post by Jeremy Friesner
Jeremy,

    This is already done with cmake/ctest. Results from both valgrind or  
purify are processed and sent to Dart. For example you can have a look  
at:

http://public.kitware.com/dashboard.php?name=cmake

        Down in the page there is a table: [Dynamic Analysis]

http://www.cmake.org/Testing/Sites/smallfry.kitwareout/Linux-gcc- 
snapshot/20050613-0557-Experimental/DynamicAnalysis.html

My 2 cents
Mathieu

On Jun 13, 2005, at 6:38 PM, Jeremy Friesner wrote:

> Hi guys,
>
> We're planning on enhancing our daily-autobuild process to include  
> some daily
> automated testing also, and it would be cool if we could have the unit  
> tests
> run under valgrind so that any valgrind-detectable errors show up in  
> the
> daily auto-test results.
>
> In order to implement this, I think we would need to somehow tell  
> valgrind the
> following:
>
> - If any valgrind-errors are detected, immediately exit with an error  
> code
> - If, at program-exit time, any memory leaks are detected, exit with  
> an error
> code
>
> Is there any way to tell valgrind to behave like that?  If not, could  
> a flag
> of some sort be added to implement this?  Currently it appears that  
> errors
> messages are printed to stderr;  I suppose our build script could be  
> made to
> parse the stderr output but that would make things more complicated  
> than I
> would like to have them.
>
> Thanks,
> Jeremy Friesner
> Level Control Systems
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: NEC IT Guy Games.  How far can you  
> shotput
> a projector? How fast can you ride your desk chair down the office  
> luge track?
> If you want to score the big prize, get to know the little guy.
> Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
> _______________________________________________
> Valgrind-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/valgrind-users
>



-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.  How far can you shotput
a projector? How fast can you ride your desk chair down the office luge track?
If you want to score the big prize, get to know the little guy.  
Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
_______________________________________________
Valgrind-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/valgrind-users
Reply | Threaded
Open this post in threaded view
|

Re: Using valgrind as part of a daily auto-test script?

Jeremy Friesner
In reply to this post by Jeremy Friesner
Tom Hughes wrote:

>The exit status returned by valgrind is the exit status of the program
>being run by valgrind - if valgrind were to replace that with it's own
>exit status then you wouldn't know what the original exit status of the
>program was.

Right -- but I wouldn't care, either, because if valgrind reported an error,
then the original exit status of the program is irrelevant -- the test
mechanism has detected an error, no matter what exit value the original
program returned.

It would be important that valgrind not replace the original error code in the
case where no valgrind errors were detected, of course.

> I think most people doing this just grep the valgrind output for non-zero
> error counts.

I suppose that can be done, if necessary.

Jeremy


-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.  How far can you shotput
a projector? How fast can you ride your desk chair down the office luge track?
If you want to score the big prize, get to know the little guy.  
Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
_______________________________________________
Valgrind-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/valgrind-users
Reply | Threaded
Open this post in threaded view
|

Re: Using valgrind as part of a daily auto-test script?

Tom Hughes-2
In message <[hidden email]>
          Jeremy Friesner <[hidden email]> wrote:

> Tom Hughes wrote:
>
> >The exit status returned by valgrind is the exit status of the program
> >being run by valgrind - if valgrind were to replace that with it's own
> >exit status then you wouldn't know what the original exit status of the
> >program was.
>
> Right -- but I wouldn't care, either, because if valgrind reported an error,
> then the original exit status of the program is irrelevant -- the test
> mechanism has detected an error, no matter what exit value the original
> program returned.
>
> It would be important that valgrind not replace the original error code in
> the  case where no valgrind errors were detected, of course.

That's not possible as you would have no way of knowing where the
exit status came from - if valgrind exits with 1 when errors are
found and preserves the code in other cases how you would you know
whether an exit code of 1 came from your program or valgrind?

Tom

--
Tom Hughes ([hidden email])
http://www.compton.nu/


-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.  How far can you shotput
a projector? How fast can you ride your desk chair down the office luge track?
If you want to score the big prize, get to know the little guy.  
Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
_______________________________________________
Valgrind-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/valgrind-users
Reply | Threaded
Open this post in threaded view
|

Re: Using valgrind as part of a daily auto-test script?

Stefan Sauer-4
In reply to this post by Jeremy Friesner
Hi Jeremy,

if you want have a look at
http://www.buzztard.org/index.php/Testing
http://www.buzztard.org/files/guadec2005_advanced_unit_testing.pdf
in there we presented some ideas of unit test integration with valgrind. This
list was quite helpful to get it working (thanks again).
I'd like to here back from you, if you find better ways.

Stefan

> Hi guys,
>
> We're planning on enhancing our daily-autobuild process to include some daily
> automated testing also, and it would be cool if we could have the unit tests
> run under valgrind so that any valgrind-detectable errors show up in the
> daily auto-test results.
>
> In order to implement this, I think we would need to somehow tell valgrind the
> following:
>
> - If any valgrind-errors are detected, immediately exit with an error code
> - If, at program-exit time, any memory leaks are detected, exit with an error
> code
>
> Is there any way to tell valgrind to behave like that?  If not, could a flag
> of some sort be added to implement this?  Currently it appears that errors
> messages are printed to stderr;  I suppose our build script could be made to
> parse the stderr output but that would make things more complicated than I
> would like to have them.
>
> Thanks,
> Jeremy Friesner
> Level Control Systems
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: NEC IT Guy Games.  How far can you shotput
> a projector? How fast can you ride your desk chair down the office luge track?
> If you want to score the big prize, get to know the little guy.  
> Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
> _______________________________________________
> Valgrind-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/valgrind-users



-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.  How far can you shotput
a projector? How fast can you ride your desk chair down the office luge track?
If you want to score the big prize, get to know the little guy.  
Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
_______________________________________________
Valgrind-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/valgrind-users
Reply | Threaded
Open this post in threaded view
|

Re: Using valgrind as part of a daily auto-test script?

Daniel Veillard
On Tue, Jun 14, 2005 at 11:50:45AM +0200, Stefan Kost wrote:

> Hi Jeremy,
>
> if you want have a look at
> http://www.buzztard.org/index.php/Testing
> http://www.buzztard.org/files/guadec2005_advanced_unit_testing.pdf
> in there we presented some ideas of unit test integration with valgrind.
> This list was quite helpful to get it working (thanks again).
> I'd like to here back from you, if you find better ways.
>
> Stefan
>
> >Hi guys,
> >
> >We're planning on enhancing our daily-autobuild process to include some
> >daily automated testing also, and it would be cool if we could have the
> >unit tests run under valgrind so that any valgrind-detectable errors show
> >up in the daily auto-test results.

  For the record I have "make valgrind" targets which is just a
"make tests" but running under the control of valgrind, it's precious.
The only big problem I have is that a fair amount of my regression
tests are driven by python scripts and valgrind doesn't grok the
garbage collector of recent python versions.

Daniel

--
Daniel Veillard      | Red Hat Desktop team http://redhat.com/
[hidden email]  | libxml GNOME XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/


-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.  How far can you shotput
a projector? How fast can you ride your desk chair down the office luge track?
If you want to score the big prize, get to know the little guy.  
Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
_______________________________________________
Valgrind-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/valgrind-users
Reply | Threaded
Open this post in threaded view
|

Re: Using valgrind as part of a daily auto-test script?

Bob Rossi
In reply to this post by njn (Bugzilla)-2
On Mon, Jun 13, 2005 at 06:14:53PM -0500, Nicholas Nethercote wrote:

> On Tue, 14 Jun 2005, Tom Hughes wrote:
>
> >>In order to implement this, I think we would need to somehow tell valgrind
> >>the  following:
> >>
> >>- If any valgrind-errors are detected, immediately exit with an error code
> >>- If, at program-exit time, any memory leaks are detected, exit with an
> >>error
> >>code
> >>
> >>Is there any way to tell valgrind to behave like that?  If not, could a
> >>flag  of some sort be added to implement this?  Currently it appears that
> >>errors  messages are printed to stderr;  I suppose our build script could
> >>be made to  parse the stderr output but that would make things more
> >>complicated than I  would like to have them.
> >
> >The exit status returned by valgrind is the exit status of the program
> >being run by valgrind - if valgrind were to replace that with it's own
> >exit status then you wouldn't know what the original exit status of the
> >program was.
> >
> >I think most people doing this just grep the valgrind output for non-zero
> >error counts.
>
> Better is to run with -q and use the client requests like
> VALGRIND_COUNT_ERRORS and VALGRIND_COUNT_LEAKS within the tests.  Others
> have certainly done this -- they may have more advice.

Yes, this is the approach we take and it works wonderfully.

Bob Rossi


-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.  How far can you shotput
a projector? How fast can you ride your desk chair down the office luge track?
If you want to score the big prize, get to know the little guy.  
Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
_______________________________________________
Valgrind-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/valgrind-users
Reply | Threaded
Open this post in threaded view
|

Re: Using valgrind as part of a daily auto-test script?

njn (Bugzilla)-2
In reply to this post by Daniel Veillard
On Tue, 14 Jun 2005, Daniel Veillard wrote:

>  For the record I have "make valgrind" targets which is just a
> "make tests" but running under the control of valgrind, it's precious.
> The only big problem I have is that a fair amount of my regression
> tests are driven by python scripts and valgrind doesn't grok the
> garbage collector of recent python versions.

Why would you run the python scripts under Valgrind?  One possibility is
to use a C wrapper that looks (very roughly) like this:

   #include "valgrind.h"
   int main(argc, argv)
   {
     int n = VALGRIND_COUNT_ERRORS(...)

     ... execute C/C++ program specified on command line ...

     int n2 = VALGRIND_COUNT_ERRORS(...)

     return n2 - n;
   }

If the exit code is non-zero, you have some errors in the test. You could
invoke this from your Python script for each test.  I'm sure there are
other ways of achieving this.

Nick


-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.  How far can you shotput
a projector? How fast can you ride your desk chair down the office luge track?
If you want to score the big prize, get to know the little guy.  
Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
_______________________________________________
Valgrind-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/valgrind-users
Reply | Threaded
Open this post in threaded view
|

Re: Using valgrind as part of a daily auto-test script?

Daniel Veillard
On Tue, Jun 14, 2005 at 08:21:07AM -0500, Nicholas Nethercote wrote:
> On Tue, 14 Jun 2005, Daniel Veillard wrote:
>
> > For the record I have "make valgrind" targets which is just a
> >"make tests" but running under the control of valgrind, it's precious.
> >The only big problem I have is that a fair amount of my regression
> >tests are driven by python scripts and valgrind doesn't grok the
> >garbage collector of recent python versions.
>
> Why would you run the python scripts under Valgrind?

  Because the testing is done from the python bindings, that high
level scripting is needed to make advanced tests, and that doing them
in shell is less potable and harder than from python. One instance of
a python test for example go through 20,000+ individual tests from
a big testsuite.

> to use a C wrapper that looks (very roughly) like this:
>
>   #include "valgrind.h"
>   int main(argc, argv)
>   {
>     int n = VALGRIND_COUNT_ERRORS(...)
>
>     ... execute C/C++ program specified on command line ...
>
>     int n2 = VALGRIND_COUNT_ERRORS(...)
>
>     return n2 - n;
>   }

  No that would not work. Part of the tests are simple that I can use
directly a C based command (xmllint/xsltproc) from shell then that's
fine, but there are tests which are way too complex for that approach.
  It used to work from Python, it doesn't anymore with recent python
versions. The only option would be to rewrite all this Python code in
C, it's not trivial, takes quite a bit of time, and makes maintainance
heavy. Apparently there is no simple solution to this Python problem
short pf recompiling Python with a special option (urgh...).

  Another problem I'm facing is for gamin. It seems valgrind 2.x doesn't
handle the dnotify kernel API, basically if run under valgrind the gamin
server (gam_server) never get the exception raised by the kernel to alert
the user code of changes on the monitored directories. Usually this will
be carried by signal 33 or 34 (Real Time I think).

  Except that I'm a very happy valgrind user :-)

Daniel

--
Daniel Veillard      | Red Hat Desktop team http://redhat.com/
[hidden email]  | libxml GNOME XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/


-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games.  How far can you shotput
a projector? How fast can you ride your desk chair down the office luge track?
If you want to score the big prize, get to know the little guy.  
Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
_______________________________________________
Valgrind-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/valgrind-users