Stereo Classification and Performance Evaluation

cvlab
|SAD|TAD|SSD|original|

Classification and performance evaluation of different aggregation costs for stereo matching


Preliminary experimental results

This table presents the results obtained by the evaluated algorithms on the Middlebury Dataset [11]. For all the algorithms, the local cost measure adopted is that originally proposed by the authors in their work. For those works not addressing a specific measure, the Sum of Absolute Differences (SAD) was used (for additional results concerning different measures, please refer to the sub-menu on top of this page).
The table allows to interactively explore the results:

All the tuned parameter values for each algorithm which were used to produce the experimental results shown in this table are available here.

NOTE: all variants of algorithm Multiple Windows were implemented withouth the use of incremental schemes (Box-Filtering, Integral Images, ..). Hence the reported processing times concerning that algorithm are higher than those achievable by means of any of such techniques.


Algorithm Rank
Accuracy		 Tsukuba
nonocc		 Tsukuba
disc		 Venus
nonocc		 Venus
disc		 Teddy
nonocc		 Teddy
disc		 Cones
nonocc		 Cones
disc		 Rank
Time		 Time Teddy
(hh:mm:ss)		 Avg.
Rank
Locally Consistent + Fast Bilateral 39_3 [16] 1.00 1.77 1 5.92 1 0.27 1 1.77 1 9.30 1 17.90 1 4.75 1 10.50 1 12 00:00:37 6.50
Fast Bilateral 39_3 [15] 3.13 2.95 4 8.69 4 1.15 3 6.64 4 10.70 4 20.80 2 5.23 2 11.40 2 11 00:00:28 7.06
Segment support [10] 3.25 2.15 2 7.22 2 1.38 5 6.27 3 10.50 2 21.20 3 5.83 5 11.80 4 19 00:39:30 11.13
Locally Consistent + Fixed Window [16] 3.50 3.07 5 9.63 6 0.66 2 5.11 2 10.60 3 21.80 4 5.30 3 11.60 3 8 00:00:15 5.75
Fast Bilateral 45_5 [15] 5.75 3.34 7 9.99 7 2.11 6 6.72 5 11.50 5 21.80 4 6.81 6 13.80 6 7 00:00:14 6.38
Segmentation based [5] 6.75 2.25 3 8.87 5 1.37 4 9.40 7 12.70 7 24.80 7 11.10 11 20.10 10 13 00:05:14 9.88
Adaptive weight [14] 6.88 4.66 10 8.25 3 4.61 11 13.30 10 12.70 7 22.40 5 5.50 4 11.90 5 18 00:20:35 12.44
Fast Bilateral 49_7 [15] 7.00 3.99 8 12.30 8 3.01 8 8.42 6 12.30 6 23.00 6 7.50 7 15.10 7 5 00:00:09 6.00
Fast Bilateral 45_9 [15] 8.75 4.60 9 13.70 10 5.42 12 10.60 8 13.90 8 24.80 7 9.47 8 17.70 8 4 00:00:05 6.38
Variable Windows [12] 11.13 3.12 6 12.40 9 2.42 7 13.30 9 17.70 11 25.50 8 21.20 22 27.30 17 10 00:00:26 10.56
Reliability [8] 11.13 5.08 11 17.90 13 3.92 10 13.90 12 18.90 13 29.90 9 11.30 12 18.30 9 14 00:13:39 12.56
Multiple windows* (25W) [7] 14.50 7.57 14 22.70 16 3.91 9 21.10 14 20.90 17 33.20 13 13.70 18 26.90 15 6 00:00:13 10.25
Multiple windows (9W) [7] 14.88 7.60 15 25.70 18 7.02 16 33.00 20 16.00 9 36.90 17 10.60 9 26.90 15 3 00:00:04 8.94
Multiple windows (25W) [7] 15.13 7.28 12 25.90 20 6.18 14 29.00 17 18.00 12 35.60 16 11.80 14 27.10 16 7 00:00:14 11.06
Gradient guided [6] 15.25 7.41 13 16.20 12 12.90 22 32.30 19 20.10 15 32.80 12 13.50 17 24.90 12 9 00:00:16 12.13
Multiple windows* (9W) [7] 15.63 9.18 17 22.60 15 6.23 15 28.10 16 21.40 18 34.50 14 13.20 16 26.70 14 3 00:00:04 9.31
Recursive adaptive [3] 16.38 9.66 20 29.80 25 5.94 13 29.80 18 20.10 14 34.60 15 11.70 13 25.30 13 17 00:20:20 16.69
Shiftable windows [11] 16.75 9.58 18 14.40 11 9.66 19 16.50 13 23.60 20 31.20 10 24.40 23 33.60 20 4 00:00:05 10.38
Multiple windows (5W) [7] 16.88 7.62 16 27.20 23 7.55 17 37.20 21 17.40 10 39.70 19 11.00 10 27.80 19 2 00:00:02 9.44
Multiple adaptive [4] 17.00 11.70 21 27.30 24 11.90 21 13.70 11 20.40 16 31.80 11 15.80 19 25.30 13 22 02:08:17 19.50
Multiple windows* (5W) [7] 18.25 9.61 19 25.10 17 9.36 18 38.30 22 22.20 19 38.00 18 12.10 15 27.50 18 2 00:00:02 10.13
Max connected [2] 21.00 11.80 22 26.40 21 42.50 26 50.90 25 34.50 23 41.00 20 17.70 20 22.70 11 21 01:59:09 21.00
Fixed window 21.13 9.58 18 27.10 22 10.60 20 42.50 24 25.10 21 42.40 22 19.70 21 36.00 21 1 < 1 s 11.06
Oriented rod* [9] 22.25 18.60 25 31.10 26 20.30 23 26.60 15 30.70 22 41.80 21 37.80 24 47.30 22 16 00:17:19 19.13
Oriented rod [9] 22.50 14.20 23 25.80 19 21.90 24 29.80 18 37.50 24 48.60 23 48.50 25 55.50 24 15 00:17:00 18.75
Radial adaptive [13] 23.00 14.80 24 21.80 14 22.40 25 40.40 23 49.60 25 50.10 24 50.20 26 53.60 23 20 01:06:21 21.50

References

[1] F. Tombari, S. Mattoccia, L. Di Stefano, E. Addimanda, “Classification and evaluation of cost aggregation methods for stereo correspondence", IEEE International Conference on Computer Vision and Pattern Recognition (CVPR 2008), 2008.
[2] Y. Boykov, O. Veksler, and R. Zabih. A variable window approach to early vision. IEEE Trans. PAMI, 20(12):1283–1294, 1998.
[3] S. Chan, Y. Wong, and J. Daniel. Dense stereo correspondence based on recursive adaptive size multi-windowing. In Proc. Image and Vision Computing New Zealand (IVCNZ’03), volume 1, pages 256–260, 2003.
[4] C. Demoulin and M. Van Droogenbroeck. A method based on multiple adaptive windows to improve the determination of disparity maps. In Proc. IEEE Workshop on Circuit, Systems and Signal Processing, pages 615–618, 2005.
[5] M. Gerrits and P. Bekaert. Local Stereo Matching with Segmentation-based Outlier Rejection. In Proc. Canadian Conf. on Computer and Robot Vision (CRV 2006), pages 66-66, 2006.
[6] M. Gong and R. Yang. Image-gradient-guided real-time stereo on graphics hardware. In Proc. Int. Conf. 3D Digital Imaging and Modeling (3DIM), pages 548–555, 2005.
[7] H. Hirschmuller, P. Innocent, and J. Garibaldi. Real-time correlation-based stereo vision with reduced border errors. Int. Journ. of Computer Vision, 47:1–3, 2002.
[8] S. Kang, R. Szeliski, and J. Chai. Handling occlusions in dense multi-view stereo. In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR 2001), pages 103–110, 2001.
[9] J. Kim, K. Lee, B. Choi, and S. Lee. A dense stereo matching using two-pass dynamic programming with generalized ground control points. In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR 2005), pages 1075–1082, 2005.
[10] S. Mattoccia, F. Tombari, and L. Di Stefano. Stereo vision enabling precise border localization within a scanline optimization framework. In Proc. Asian Conf. on Computer Vision (ACCV 2007), pages 517–527, 2007.
[11] D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. Jour. Computer Vision, 47(1/2/3):7–42, 2002.
[12] O. Veksler. Fast variable window for stereo correspondence using integral images. In Proc. Conf. on Computer Vision and Pattern Recognition (CVPR 2003), pages 556–561, 2003.
[13] Y. Xu, D. Wang, T. Feng, and H. Shum. Stereo computation using radial adaptive windows. In Proc. Int. Conf. on Pattern Recognition (ICPR 2002), volume 3, pages 595– 598, 2002.
[14] K. Yoon and I. Kweon. Adaptive support-weight approach for correspondence search. IEEE Trans. PAMI, 28(4):650–656, 2006.
[15] S. Mattoccia, S. Giardino and A. Gambini. Accurate and efficient cost aggregation strategy for stereo correspondence based on approximated joint bilateral filtering. In Asian Conference on Computer Vision (ACCV 2009), 2009.
[16] S. Mattoccia. A locally global approach to stereo correspondence. IEEE Workshop on 3D Digital Imaging and Modeling (3DIM 2009), 2009.