My Auto Panorama!
The goal of the project is to implement an end-to-end pipeline to do image panorama stitching of unordered images.
Language: MATLAB
AlGORITHM:
1. Capture Images
Simply caputure a set of unordered images as shown in Fig. 1, Fig. 2, Fig. 3.
2. Find the Corners
The aim of this step is to detect corners spread all across the image to avoid weird artifacts in warping. Implement the following steps for ANMS:
1) Detect corner features: Detect corner features using cornermetric().
The output is a matrix of corner scores. Visualize the output using imagesc(). The result is shown in Fig. 4.
2) Find Strong Corners: Find the N strong strongest corners using the Matlab function imregionalmax(). More than 3000 "corners" could be found, some of which are not real corners.
Then ANMS algorithm is used to choose the best corners in the image. I chose the strongest 490 corners and the results are shown in Fig. 5.
3. Feature Descriptor
Next step is to describe each feature point by a feature vector. Take a patch of size 40 x 40 centered (this is very important) around one corner point. Then apply gaussian blur.
Now, sub-sample the blurred output to 8 x 8. Then reshape to obtain a 64 x 1 vector. Standardize the vector to have zero mean and variance of 1.
4. Feature Matching
Pick one point in image 1, compute sum of square difference
between all points in image 2. Take the ratio of best match (lowest distance) to the second best match (second lowest distance) and if this is below some ratio keep the matched pair or reject it. Repeat this for all points in image 1. I am left with only the confident feature correspondences and these points will be used to estimate the transformation between the 2 images also called as Homography. Finally use the function dispMatchedFeatures() to visualize the result as shown in Fig. 6 and Fig. 7.
5. RANSAC to estimate robust Homography
Use RANSAC to compute homography. The steps are:
1) : Select four feature pairs (at random), pi from image 1, p1i from image 2.
2) : Compute homography H (exact). Use the function est homography given to you.
3) : Compute inliers where SSD (p1i;Hpi) ¡ thresh. Here, Hpi computed using the apply homography() function given to you.
4) : Repeat the last three steps until you have exhausted Nmax number of iterations (specified by user) or you found more than a percentage of inliers (say 90% for example).
5) : Keep largest set of inliers.
6) : Re-compute least-squares H estimate on all of the inliers. Use the function est homography() given to you.
6. Cylindrical Projection
When the FoV is large, to overcome the distortion problems at edges, we will be using cylindrical projection on the images before performing other operations. For this project. We don’t need cylindrical projection. But for panorama with large FoV, cylindrical projection is necessary. However, I still use the original images to find the inliers, perform cylindrical projection for the inliers and use the projected inliers to calculate homography H.
A comparison of the original image and cylindrical image is shown in Fig. 8 and Fig. 9.
7. Blending images to get a seamless panorama
To blend the image, I firstly perform transformation for the images. Instead of using Guassian filter, I use the nearestneighbor
algorithm to fill the black lines of the transformed image. When transforming, I calculate the offset of coordinates of different images. And apply this offset when blending the images. Then I build a mask to mark where the pixel is not black. Add the mask of all the images to get where the pixels overlap. The final image is the sum of all the images by the mask. Results are shown in Fig. 10.
Fig. 1 Captured Image 1.
Fig. 2 Captured Image 2.
Fig. 3 Captured Image 3.
Fig. 4 Vusualize the features.
Fig. 5 Choose the most reliable features.
Fig. 6 Match Results(1).
Fig. 7 Match Results(2).
Fig. 7 Final Result.
Fig. 8 Image before cylindrical transformation.
Fig. 4 Vusualize the features.
Fig. 9 Image after cylindrical transformation.
