December 2012

We have completed the design of our embedded (stereo and mono) camera with highly efficient FPGA onboard processing. In stereo mode, the whole processing pipeline fits into entry level FPGA devices without additional hardware requirements delivering accurate and dense depth map in real-time. The imaging sensors, connected to the FPGA board with a standard interface,  provide color and monochrome images up to 60 fps.
 The embedded camera has software API for:

    - Windows 32 and 64 bit 
    - Linux 32 and 64 bit
    - Linux ARM 
    - Mac  
    - Android 

Further details and videos will be available soon.

If you are interested in this project for your applications feel free to contact me:

The research activity on stereo reported below is quite outdated. For an updated overview of my research activity on stereo follow this link.

Moreover, if you are interested in stereo vision you might find interesting this seminar on "Stereo vision: algorithms and applications".



This page provides experimental results and applications concerned with the Single Matching Phase (SMP) stereo algorithm [1]. Although several approaches for computing very accurate depth maps have been recently proposed (see for example [8], [9]) most of these are not currently suitable for real-time applications (see [11] for a performace evaluation of cost aggregation strategies proposed for stereo matching). Conversely, SMP is a fast and reliable algorithm for computing dense stereo correspondence in real-time. The SMP algorithm uses the uniqueness constraint as one of the main cues for detecting unreliable measurements. In [1] we provide, on a large set of standard stereo pairs with ground truth (namely "Tsukuba", "Map", "Sawtooth", "Venus", "Barn1", "Barn2", "Bull" and "Poster" available at the Scharstein and Szeliski's web site [4] and used in their paper [5]), the result of a quantitative comparison between the SMP approach and a known algorithm [3] based on bidirectional matching (BM). Bidirectional matching is also often referred to as left-right consistency check or left-right constraint. We also provide, in [1] and [2], experimental results concerned with rectified stereo sequences acquired in our laboratory with a digital stereo camera and preliminary results concerned with a 3D Tracking application and a 3D People Counting application. The SMP algorithm has been implemented in C exploiting the SIMD parallell capabilities (e.g. MMX and SSE technologies) available in recent Intel, AMD (and many others) microprocessors. A detailed description of the SIMD mapping of the SMP algorithm is available in [2]. A more recent approach concerned with (near) real-time stereo matching algorithm  was proposed in [12] (experimental results here, evaluation on the Middlebury dataset here).

November 2010: The SMP algorithm has been implemented on a Texas Instrument DaVinci DSP (300 MHz CPU + 600 MHz DSP) by Anouar Manders at SenseIT. This implementation runs at 5/6 fps with 640x480 stereo pairs, 15x15 windows, disparity range of 64 pixels and 1/8 subpixel disparity interpolation (detection of unreliable disparities is not implemented yet).



If you are interested in the SMP algorithm or in its applications feel free to contact me at: 





In this page are provided detailed experimental results and videos concerned with 3D Tracking, 3D People Counting and 3D Change/Intrusion Detection applications (described in [10]) that rely on the SMP algorithm [1] for real-time dense depth measurements.

This section provides experimental results obtained with SMP [1] and BM [3] on a standard set of stereo pairs (namely "Tsukuba", "Map", "Sawtooth", "Venus", "Barn1", "Barn2", "Bull" and "Poster") with available ground truth. The stereo pairs and the ground truth are available at the Scharstein and Szeliski's [4] web site. Disparity values are encoded with 256 gray levels, with brighter levels representing points closer to the camera and unmatched points represented in white.

For example; on a Pentium III Processor at 800 MHz, with 800x600 stereo pairs and a disparity range of 16 our algorithm runs at 5.56 fps while BM at 6.96. With this image size and a disparity range of 80 our SMP algorithm is nearly twice faster than BM (i.e. 2.89 fps for SMP and 1.51 for BM).

The MEGA-D stereo head uses a IEEE 1394 firewire interface and has a fixed baseline of  about 9 cm. The original stereo pairs were rectified using the intrinsic and extrinsic camera parameters estimated with the functions provided by the MATLAB Camera Calibration Toolbox available here.  Image size is 640x480 and the rectified sequences were processed using a 15x15 correlation window, a disparity search range of 64 pixels and a subpixel accuracy of 1/8 .

In this section we show experimental results obtained with SMP [1] on two stereo sequences acquired in our laboratory and referred to as "Outdoor" and "Indoor". We are currently using these sequences within a research activity aimed at developing a real-time 3D People Tracking application. The tracking approach is based on first merging the disparity maps extracted by SMP algorithm with the information provided by a change-detection algorithm in order to build a suitable plan-view representation [6] and [7] that enables us to track, in real-time, moving objects in the 3D space.

At this link you can find information concerned with the stereo sequence "Cortile". We provide: the rectified stereo sequences (320x240 and 640x480), the disparity maps (for five settings 1, 2, ,4 , 8, 16 of the subpixel parameter) computed by the SMP stereo algorithm and the parameters for obtaining the 3D depth measurements.  Disparity maps are encoded with RGB images (saved with OpenCV) as described in the README file.

• The original stereo sequence in ".iss" format is available here
  
  
• The left images of the original sequence in ".bmp" format are available here
  
  
• The right images of the original sequence in ".bmp" format are available here
  
  
• The rectified left images of the original sequence in ".bmp" format are available here
  
  
• The rectified right images of the original sequence in ".bmp" format are available here
  
  
• The disparity maps (size 640x480) in ".bmp" format obtained processing the rectified stereo sequence with the SMP algorithm:
  
       - disparity range 64, subpixel 1/8, window size 15 x 15 and threshold set to 0: here
  
       - disparity range 64, subpixel 1/8, window size 15 x 15 and threshold set to 1: here
  
       - disparity range 64, subpixel 1/8, window size 15 x 15 and threshold set to 2: here
  
       - disparity range 64, subpixel 1/8, window size 15 x 15 and threshold set to 3: here

• The original stereo sequence in ".iss" format is available here
  
  
• The left images (size 640x480) of the original sequence in ".bmp" format are available here
  
  
• The right images (size 640x480) of the original sequence in ".bmp" format are available here
  
  
• The rectified left images (size 640x480) of the original sequence in ".bmp" format are available here
  
  
• The rectified right images (size 640x480) of the original sequence in ".bmp" format are available here
  
  
• The disparity maps (size 640x480) in ".bmp" format obtained processing the rectified stereo sequence with the SMP algorithm:
  
       - disparity range 64, subpixel 1/8, window size 15 x 15 and threshold set to 0: here
  
       - disparity range 64, subpixel 1/8, window size 15 x 15 and threshold set to 1: here
  
       - disparity range 64, subpixel 1/8, window size 15 x 15 and threshold set to 2: here
  
       - disparity range 64, subpixel 1/8, window size 15 x 15 and threshold set to 3: here
  

This section provides experimental results concerned with a robust real-time Change/Intrusion Detection approach, described in [10], which jointly exploits depth information coming from a 3D device and 2D brightness information. Information on scene changes is recovered by means of two different strategies. The former, referred to as 3D Output, mainly relies on depth information, and aims at being robust to camouflage, shadows and sudden illumination changes. The latter, referred to as 2D Output, aims at obtaining robustness with regards to sudden illumination changes as well as accuracy in the foreground segmentation. The final change masks determined by the two outputs will be referred to as, respectively, C_2D and C_3D.  As depicted the following figure, the proposed approach, using as 3D device a stereo vision system, can be outlined as a 4-stage algorithm.  The overall system relies on the SMP algorithm [1] for real time dense depth measurements.

The following figure shows preliminary experimental results obtained processing a challenging stereo sequence, referred to as "Office", with the 3D Change/Intrusion Detection application. In this indoor sequence, acquired with a rectified color stereo camera, the strong photometric distortions (clearly visible comparing the 9 frames shown in the following figure) are induced by switching lights on and off. Moreover, it is worth observing that the same sequence is also affected by severe shadow and camouflage problems. The overall 3D Change/Intrusion Detection application, includng the disparity maps generation step, runs in real-time on a standard personal computer.

• The rectified left images of the original sequence in ".bmp" format are available here
  
  
• The rectified right images of the original sequence in ".bmp" format are available here
  
  
• The disparity maps (size 320x240) in ".bmp" format obtained processing the rectified stereo sequence with the SMP algorithm are available here.
  
  
• 2D Output in ".bmp" format available here.
  
  
• 3D Output in ".bmp" format available here.