From Invisible to Visible: Filling Data Holes Through Prediction and Sensor Enhancements

Like any instrument, image sensors exhibit limited sensitivity to the world being limited in, e.g., spatial extent, spectral range, resolution and brightness range. As examples, a camera captures only the unobstructed parts of a scene; at a degree of resolution determined by the number of pixels; in gray scale, color, or beyond, e.g., in the infrared range; having good color/brightness distinction only in a certain dynamic range and for detectable contrast values; and all of this within the field of view.
Past two decades have witnessed remarkable progress in overcoming these limitations, by compensating for the missing data by learning to make image-based, data-driven predictions, or extending the capabilities of the cameras to avoid the misses to begin with.

We will discuss some such methods for recovering (1) Missing parts of a 2D image, and (2) High spatial frequency details not resolvable with the low resolution of the image. We then discuss (3) Extension of this framework to fill spatio-temporal holes in 3D, video data, and (4) Detecting small, faint, moving objects in a spatio-temporally noisy background. All of these methods combine data driven, learning based approaches with suitable structural criteria. They jointly reconstruct appearance and motion attributes of the scene in a temporally coherent fashion, while exploiting physically grounded constraints such as scene specific geometry. The learning algorithms used do not require a large, exhaustive set of visual examples. Extension of a camera’s (5) field of view, (6) dynamic range and (7) contrast require changes in the basic imaging capabilities; we present examples of hardware enhancements of the cameras that provide such extensions.


Dr. Narendra Ahuja is a Research Professor in the Dept. of Electrical and Computer Engineering, Beckman Institute, and Coordinated Science Laboratory at University of Illinois at Urbana-Champaign (UIUC), and the Director of Information Technology Research Academy (ITRA) – a national initiative started by the Department of Electronics and Information Technology, Government of India, as a pilot for a new model of enhancing quality and impact of research and advanced education in India.
He received the B.E. degree with honors in electronics engineering from the Birla Institute of Technology and Science, Pilani, India, in 1972, the M.E. degree with distinction in electrical communication engineering from the Indian Institute of Science, Bengaluru, India, in 1974, and the Ph.D. degree in computer science from the University of Maryland, College Park, USA, in 1979. In 1979, he joined the UIUC faculty where he was Donald Biggar Willet Professor until starting at ITRA in 2012. During 1999-2002, he served as the Founding Director of the first Indian Institute of Information Technology at Hyderabad, since renamed as International Institute of Information Technology.

His current research is focused on extraction and representation of spatial structure in images and video; integrated use of multiple image-based sources for scene representation and recognition; computational cameras; and applications of computer vision to visual communication, image manipulation, and information retrieval, with focus on developing societies.

He is the author of over 450 journal and conference papers. He has co-authored the books Pattern Models (Wiley, 1983), Motion and Structure from Image Sequences (Springer-Verlag, 1992), and Face and Gesture Recognition (Kluwer, 2001); and co-edited the book Advances in Image Understanding (IEEE Press, 1996). He has supervised thesis research of about 50 PhD, 15 MS and 60 Undergrad students and 10 Postdoctoral scholars.

He received the 1999 Emanuel R. Piore award of the IEEE, and the 1998 Technology Achievement Award of the International Society for Optical Engineering, and 2008 TA Stewart-Dyer/Frederick Harvey Trevithick Prize of the Institution of Mechanical Engineers, 2008-11 Open Innovation Research Award from Hewlett-Packard, and a 2010 Google Research Award. With his students, he shared Best Paper Awards given by: Piero Zamperoni Best Student Paper Award, International Conference on Pattern Recognition (2014), Symposium on Eye Tracking Research and Applications (2014), First IEEE International Workshop on Computer Vision in Sports (2013), International Conference on Pattern Recognition (2012) and IEEE Transaction on Multimedia (2006). He received Distinguished Alumnus Award from University of Maryland Department of Computer Science (2008) and from Birla Institute for Technology and Science, Pilani (2012). He was selected as Associate for 1998-99 and 2006-07 and Beckman Associate for 1990-91 in the University of Illinois Center for Advanced Study. He received University Scholar Award (1985), Presidential oung Investigator Award (1984), National Scholarship (1967-72), President’s Merit Award (1966), and President’s Scout recognition (1966). He is a fellow of IEEE, American Association for Artificial Intelligence, International
Association for Pattern Recognition, Association for Computing Machinery, American Association for the Advancement of Science, and International Society for Optical Engineering. He has served on the editorial boards of the journals IEEE Transactions on Pattern Analysis and Machine Intelligence; Computer Vision and Image Understanding; Journal of Mathematical Imaging and Vision; Journal of Pattern Analysis and Applications; Int. Journal of Imaging Systems and Technology; Journal of Information Science and Technology; and IEE Japan Transactions on Electrical and Electronic Engineering; and a guest coeditor of the Artificial Intelligence Journal’s special issue on vision.

He has served on the Advisory/Governing Boards of IIIT, Hyderabad; IIT BHU; IIIT Delhi; Microsoft Research Connections, USA; and Microsoft Research, India; and Institute of Communication, Information and Perception Technologies, Scuola Superiore, Sant’Anna, Pisa, Italy. He has received 4 patents. He is the founding president of Vision Technology, a computer vision technology company that has developed and transferred several prototypes, and a co-founder of a start-up that is introducing a new technology for automated sports-videos analysis. His algorithms/prototype systems have been used by about 10 companies/organizations, including industrial systems at General Electric, Westinghouse, Lockheed and Honeywell.