Mrjabaa's Blog

2nd part of the project – testing at the function level (part 2)

Thanks to my mentor I have a list of functions that Hugin is using. It allows me to focus on the most important part of the project – writing the tests. And here is the list if someone needs it:
CheckParams
DisposeAlignInfo
distSquared
erect_albersequalareaconic
erect_mercator
erect_pano
erect_rect
erect_sinusoidal
erect_sphere_tp
erect_stereographic
erect_transmercator
execute_stack_new
GetFullPath
getROI
horiz
inv_radial
mercator_erect
merge
myfree
operator&lt
pano_erect
panoProjectionFeaturesQuery
panoProjectionFormatCount
pano_sphere_tp
ParseScript
persp_rect
persp_sphere
PrintError
PT_setErrorFcn
PT_setInfoDlgFcn
PT_setProgressFcn
queryFOVLimits
radial
rect_erect
rect_sphere_tp
resize
rotate_erect
RunLMOptimizer
SetAdjustDefaults
SetCoordDefaults
SetCorrectDefaults
SetCorrectionRadius
SetGlobalPtr
SetImageDefaults
SetInvMakeParams
SetMakeParams
SetMatrix
SetOptDefaults
SetStitchDefaults
sinusoidal_erect
sphere_tp_erect
sphere_tp_pano
sphere_tp_rect
stereographic_erect
StringtoFullPath
transmercator_erect
vert
WriteResults

I also resigned from using Boost Test Library simply because ctest is already handling the most of BTL tasks (it also needs to install Boost library etc.). Not every element (function) from the list above is needed to be tested, but much of them is very important. So we concentrated on two essencial areas: math/projection and optimization. A list of tested functions (they are already added into svn and integrated with cmake/ctest):

– Optimizer related functions:
* CheckParams
* distSquared
* RunLMOptimizer (this is the main function)

– Math/projection related functions:
* transmercator_erect
* erect_transmercator
* stereographic_erect
* erect_stereographic
* sinusoidal_erect
* erect_sinusoidal
* rect_erect
* erect_rect
* pano_erect
* erect_pano
* mercator_erect
* erect_mercator
* sphere_tp_erect
* erect_sphere_tp
* sphere_tp_pano
* pano_sphere_tp
* sphere_tp_rect
* rect_sphere_tp



2nd part of the project – testing at the function level

I must say it was very hard for me at the beginning to determine what functions are more important and what are less but over time I started to recognize that. I will use the Boost Test Library for unit testing, for organizing tests CTest form CMake would be better because the first part of this project was based on it. Why Boost Test Library?

• it is rather mature and stable
• it has many information and examples of its usage
• it has all the options needed for automatic testing
• it has many additional options which might be used in future tests development
• almost everyone have boost library already in their computer, so no additional installations are needed

more info about BTL : http://www.boost.org/doc/libs/1_34_0/libs/test/doc/index.html

Now some more about the tests. Here is a list I made myself for marking the priority of the testing files in libpano (the more ‘+’ the more it is important to test):

• adjust.c                +++
• bmp.c                    +
• ColourBrightness.c+++
• correct.c            ++
• dump.c                +
• fftn.c                +
• file.c                ++
• filter.c                ++++
• fourier.c            +
• hdrfile.c            +
• javastub.c            +
• jpeg.c                +
• jpegicc.c            +
• lmdif.c                +
• math.c                ++
• metadata.c            +++
• morpher.c            +
• multilayer.c        +
• optimize.c            +++
• pan.c                    +
• PaniniGeneral.c    +
• parser.c                +
• perspect.c            +
• png.c                    +
• ppm.c                    +
• PTcommon.c            ++++
• PTDialogs.c            +
• pteditor.c            +
• ptfeather.c            +++
• ptpicker.c            ++
• ptstitch.c            +++
• queryfeature.c        ++
• remap.c                ++
• resample.c            ++
• rgbe.c                +
• seamer_.c            +
• seamer.c                +
• shell_mac.c            +
• sys_ansi.c            +
• sys_common.c        +
• sys_mac.c            +
• sys_win.c            +
• sys_X11.c            +
• tiff.c                +++
• Triangulate.c        +
• ZComb.c                +

That is quite a big list but it is almost impossible to test every function from this files fully. So it is important to choice wisely. And now a bit smaller list with functions I will test in the next days (or already started testing):

• void ColourBrightness(  fullPath *fullPathImages, fullPath *outFullPathImages, int counterImages, int indexReferenceImage, int parm3,int createCurvesType)
• void CorrectImageColourBrigthness(Image *image, magnolia_struct *magnolia, int parm3)
• void filter( TrformStr *TrPtr, flfn func, flfn16 func16, void* params, int color)
• int panoCroppingMain(int argc,char *argv[], int operation, char *version, char *usage, char *defaultPrefix)
• int CropImage(Image *im, PTRect *r)
• int readJPEG ( Image *im, fullPath *sfile )
• int writeJPEG ( Image *im, fullPath *sfile,     int quality, int progressive )
• int readTIFF ( Image *im, fullPath *sfile )
• int writeTIFF ( Image *im, fullPath *fname)
• int writeHDR ( Image *im, fullPath *sfile )
• int readHDR ( Image *im, fullPath *sfile )
• int blendImages ( fullPath *f0,  fullPath *f1, fullPath *result, double s )
• int panoTiffDisplayInfo(char *fileName)
• int panoCreatePanorama(fullPath ptrImageFileNames[], int counterImageFiles, fullPath * panoFileName, fullPath * scriptFileName)
• void  RunLMOptimizer( OptInfo    *o)
• int panoFlattenTIFF(fullPath *fullPathImages, int counterImageFiles, fullPath *outputFileName, int removeOriginals);
• int panoPSDCreate(fullPath * fullPathImages, int numberImages, fullPath * outputFileName, pano_flattening_parms *flatteningParms)

This list isn’t full and might be changed over time.

Finishing 1st part of the project

Checking and a lot of debugging resulted in some code changes.So a lot of a new reference files were added. Also, right now testing framework gained new ability to test the PToptimizer. I also struggled with a new found weird bug (TIFFOpen), this bug is not deterministic and if come out it just causing that one of the image isn’t created, and so the whole test fails. Some additional small changes might be needed in reference files (especially for parser). I also started to determining what functionality (what functions) need to be tested in function level testing framework and started designing this network (however it is still on very early phase). I will show some results of this issue soon.

Shortly main changes in the project:

• perl script for PToptimizer testing
• added reference files for PTroller, PTcrop, PTuncrop, PTtiff2psd and PToptimizer
• removed and/or added new and/or modified reference files for most of the tools
• some additional changes in test generation
• other changes like comments etc.

PTroller, PTcrop, PTuncrop and PTtiff2psd tests

This week I spent on creating new features to the project. Mainly I created some code to test some tools such as:

• PTroller – right now it creates tiff files itself (similar to ‘simple’ tests used for each of input files), but it is possible to use as input files tiffs that are added to the project as reference
• PTcrop – tests are constructed as in PTroller, but it is also possible to crop and then uncrop given image and check if the image is the same (comparing to the image which was taken to the processing)
• PTuncrop – as above
• PTtiff2psd – for comparing I used a new function (psdCmp) based on identify command -line tool from ImageMagick (more info about using this tool: http://www.imagemagick.org/Usage/basics/#identify and http://www.imagemagick.org/Usage/files/#info – use with -verbose to get more details about the given file).  I intended to check every layer, but when I was using ImageMagick I get results similar to this: http://www.rubblewebs.co.uk/imagemagick/psd.php, so right now only 1st method is used.

other changes made in the project:

• fixed problems with parameter testing (new reference files needed)
• small changes in compare.pl (mainly adjustments to cmake)
• changes in cmake files (for adding tests)
• other smaller changes in other files

Right now I’m working on extend this tools testing capabilities, creating and planing tests for other tools (especially for PToptimizer and PTmasker).

Last week report

Thanks to Daniel M. German, the project gain new ability to dump the input data structures. Till now the main problem in adding new tests cases was the space they take. Now this problem is almost gone. Of course this method isn’t testing everything but in many many cases it is almost everything that is substantial in test case. The main ‘old’ tests must remain to test the most important elements of the project, new method should be essential part of testing that gives more information about problems/bugs that may be found (because much more test cases can be done that way). Now the main problem is the time that tests take, but this is a result of number of them – e.g. combination of projection, formats and interpolators gives about 20x10x23 = 4600 in worst case (when all options are checked) – but there is a solution: not every test must be executed, we can divide tests into two groups first that tests most important elements of the project and second that contains predefined sets of tests cases that tests specific part of the project. This could be done by customizing cmake (ctest), more info: http://www.cmake.org/Wiki/CMake_Testing_With_CTest#Running_Individual_Tests

most important changes in the project during last week:
in parser.pl
– new simple tool for testing with parser – parser.pl (basic version)
– added possibility for generating combined tests
– combine all parameters functions into one
– reference files will be extended if result won’t be found
– improved documentation and added some comments
– code improvements
– other less important changes
in other files
– added 64 new tests into cmake (16 are removed due to time they take)
– added reference files for new tests

Currently I’m working at adjusting existing tests with cmake and making some new tests to other tools.

A report that I should post week ago

Here is a little summary of the project based on doc/stitch.txt file.
p – line options
w various widths are used during tests
h various heights are used during tests
f almost all projection formats are tested
P optional parameters for some projections – only defaults are used
v horizontal field of view of panorama – tested
n most important Panorama file formats are tested (only for f1 as input)
u not tested (default:10)
k not tested
b not tested
d not tested

o – line options
f f0 f1 f2 f3 are tested
v horizontal field of view of image
y various yaw angle are given in input images
p various pitch angle are given in input images
r various roll angle are given in input images
a,b,c lens correction coefficients are tested (for f0 and f2)
d,e initial lens offset are tested (for f0 and f2)
g,t initial lens shear are tested (for f0 and f2)
Tx,Ty,Tz,Ts not tested
+buf, -buf obsolete, not tested
S Selection, f2 has input images with S option, (no separate test for
this option)
C Crop (no separate test for this option)
m,mx,my,s obsolete, not tested

m – line options
g gamma, not tested
i interpolators, all are tested (separate from cmake)
f Invoke Faster Transformation, not tested

Changes that I made to the project in this two weeks:
– preparation for parser testing (I made some functions in code more modular, so
it would be easier to use and combine them)
– improved documentation and added some comments
– removed interpolators test from cmake testing process
– parameters tests don’t need anymore separate files (now they are generated
automatically)
– added functionality to add custom tests
– code improvements
– other less important changes

Project status update.

Main changes:
-removed old unused references
-added multiple changes in test generating script such as testing formats,
testing interpolators and antialising, simple testing, testing parameters
-added new references
-added additional logging procedures
-integrated testing process with cmake
-created runTest.pl for replace similar script written in python
-changes in comparing methods
-many other small changes
-and of course removed the dependency of enblend and changed loaction of the
exexcutables
Currently I’m working at cropping tests, code improvements and making more
comments and documentation to the project.

1st report

The things I done during this week:
+ created reference and tests for a circular fisheye lens
+ created reference and tests for a rectilinear lens
+ made changes in gen.pl (E0 R0 problem is now resolved,
unnecessary lines are now erased, and some other changes)
+ made changes in resizer.pl and resizer2.pl
+ mirror changes in panoAutomateTest.py
+ created simple README.TXT file
+ created new however still primitive tool for comparing images
“compare.pl” (it includes some different methods of comparison)
+ made some changes for a fullframe fisheye lens tests
+ made some changes for a equirectangular lens tests
+ created a new branch and add all files to the SVN repository
+ gathering information about adding regression tests using cmake

Plans for the next week(s):
– add additional parameters for some output projections in gen.pl
– add tests for other parameters
– add tests for cropping
– improve compare.pl
– log tool for tests
– set a rules that decide if the test is passed
– gather more information about adding regression tests using cmake

Problems:
– so far reference images took already much space, but I have some
ideas to reduce it, however all of these ideas have flaws, the main
three ideas are:
1) use only one image for each format in a test – the result of enblend
use so the only results of enblend would be tested
2) use one format of image (for example only *jpg or *.tif) in a test and
ignore small difference between test and reference files
3) combining 1) and 2)

Hello world!

Blog still in update process. This is a test post.