The challenge of super-resolving a low-resolution wide-angle image (WAL) based on another high-resolution arrow angle reference (NAH) lies in the following aspects:
- The dense correspondence betweent he WAL and the NAH. This part is essential for transfer of high resolution contents from NAH to WAL. We have some previous work on dense correspondence matching, which is highly relevent to this sub-problem.
Jie Chen, Junhui Hou, Yun Ni, and Lap-Pui. Chau, “Accurate Light Field Depth Estimation with Superpixel Regularization over Partially Occluded Regions,’’ IEEE Transactions on Image Processing (IEEE TIP), vol. 27, no. 10, pp. 4889-4900, 2018. [pdf] [code]
- With a camera pose estimation that linkes the positions of the WAL and the NAH, we can use the grid postions as guide for super-resolution operations. This is highly similar to light field spatial super-resolution on which topic we have some prior research.
Henry W. F. Yeung, Junhui Hou, Xiaoming Chen, Jie Chen, Zhibo Chen, and Yuk Ying Chung, “Light Field Spatial Super-Resolution Using Deep Spatial-Angular Interleaved CNN,’’ IEEE Transactions on Image Processing (IEEE TIP).
- The major challenge we face for this problem however, is the extrapolation of high frequency conetents from liminted angle of view, and the propagation of it to wider angles. We have some on-going research, which is currently under review process at a CV publication venue. The solution we proposed was called a Stratified Labelling for Surface Consistent Parallax Correction and Occlusion Prediction, which uses a Generative Adversarial Network (GAN) to extend the camera baseline up to 10 times wider, based on the given narrow baseline references as input. We show some preliminary results here. In these results, we used a narrow baselined 3 captures to render the scene’s appearance that is from a wider baseline angle. (Currently, this problem set up is a little different from WAL super-resolution based on NAH, however the solution can be largely similar.)
