LightCam: ENLIGHTENING THE CAMERA OBSCURA – WHERE PHOTOGRAMMETRY, COMPUTER AND ROBOTIC VISION MEET

In the last years, vision-based systems have flourished at an unprecedented pace, fuelled by developments in hardware components (higher resolution and higher sensitivity imaging sensors, smaller and smarter micro controllers, just to name a few), as well as in software or processing techniques, with AI (Artificial Intelligence) leading to a landmark revolution. Several disciplines have fostered and benefited from these advances, but, unfortunately, not always in a coordinated and cooperative way. When it comes to image-based sensing techniques, photogrammetry, computer vision and robotic vision have many contact points and overlapping areas. Yet, as for people of different cultures and languages, communicating among the three different communities can be very harsh and disorienting especially for beginners and non-specialists. Driven by a strong educational and inclusive ambition, the LightCam project is funded by the ISPRS Education and Capacity Building Initiatives 2020 (ECB). The project’s ambition is to act as an interpreter and ease the dialog among the three actors, i.e. photogrammetry, computer vision and robotics. Two intermediation tools will be developed to serve this aim: (i) a dictionary of concepts, terminology and algorithms, in the form of a knowledge base website, and (ii) a code repository, where pieces of code for the conversion between different formulations implemented in available software solutions will be shared.


Background and motivations
We perceive the world through our senses, where vision is the one we rely on the most. Since time immemorial, vision has represented our primary resource to observe, investigate and understand the nature and essence of things. Not surprisingly, we have mimicked this natural characteristic, making vision the core sensing approach in visual arts, science and technology. Drawings, paintings and images were the preferred form of representation, documentation and communication before the invention of photography, which marked a milestone not only in art but also in engineering. In fact, it was not long after its invention that photography was introduced to map cities (Jensen, 2007) or survey buildings (Albertz, 2002), applications that represented the birth of photogrammetry. Since then, imaging devices have significantly developed. The transition to digital and miniaturised sensors, coupled with algorithmic advances and increasing computing capabilities have opened new research and technological frontiers.
Vision, with its underlying mathematical and geometric principles, is a common ground for several scientific and technological domains, especially for photogrammetry, computer vision and robotics. Photogrammetry is the art or science of obtaining reliable measurements by means of photography (ASPRS, 1952). Its main applications are mapping, surveying and 3D modeling, with great emphasis on accuracy, reliability and completeness of the results (Förstner & Wrobel, 2016). Computer vision designs computational models able to reproduce the human visual system to support autonomous systems (Huang, 1996). It investigates theory and methods for the automatic extraction, analysis and understanding of information from single images or sequences. Robotics is a vast engineering discipline which deals with robots, i.e. programmable machines designed to perform a series of tasks autonomously, or semi-autonomously. Robotic vision is one of the technologies that enables the robots with sensing capabilities and, strongly related to computer vision, is the focus of this project. Whereas computer and robotic vision can be considered closely related, often sharing a common basis of terminology and theoretical formulations, in the past, these disciplines and photogrammetry have rarely crossed, growing up with different motivations in different backgrounds and developing their own nomenclatures and conventions. At the beginning of the 90's, photogrammetric methodology was still limitedly applied in robot vision because, at that time, the severe constraints imposed to assure high precision and reliability contradicted the needs of high-speed performance in robotics (Gruen, 1992).
With the advent of the internet and the easier possibility to access information, attempts have been made to create a bridge between photogrammetry and computer/robotic vision. A prominent example is the book edited by Gruen & Huang (2001) entitled 'Calibration and orientation of cameras in computer vision'. It was the result of a workshop on the same topic held in 1992 and covering topical aspect of machine (or computer) vision. The articles collected in the book showed that these themes were core subjects to both photogrammetry and computer vision, at the same time showing how they were handled from different perspectives. In (a), an optical scaling in the camera coordinate system is applied, resulting in image coordinates expressed in mm. The image coordinates are scaled from pixels to mm before lens distortion parameters are used to "correct" the measured image coordinates. The difference between the projected ideal and corrected points provides the residual, highlighted in bold. In (b), the lens distortion coefficients are used to 'distort' the projected ideal point in normalized units; the image coordinates are then scaled to pixel units. The residual (bold circle) is computed between the measured point and the "distorted" projected point.
In the computer vision community, the seminal work by Triggs et al. (1999) presented a survey of the theory and methods of photogrammetric bundle adjustment, addressed to potential developers in their community. Davison (2015) recognised that, before the beginning of 2000, when his pivotal paper on SLAM with a single camera was published (Davison, 2003), the mobile robotics community had almost completely lost interest in pure vision-based navigation approaches and, at the same time, the computer vision community had almost abandoned researching on real-time and robotics applications. In the last ten years, photogrammetry has tried to interact with computer vision; as a result, many tools developed in computer vision have been adopted in photogrammetry (Granshaw & Fraser, 2015), and many terms derived from computer vision have been included in the third edition of the photogrammetric terminology (Granshaw, 2016). The book by (Förstner & Wrobel, 2016), entitled "Photogrammetric Computer Vision", addresses concepts and methods developed in the computer vision community with a focus on the main application areas of photogrammetry, specifically mapping and image-based metrology. Still, software solutions and algorithms coming from different communities, although sharing a common theoretical basis, may entail different formulations and implementations ( Fig. 1

The ISPRS ECB Initiatives
The International Society for Photogrammetry and Remote Sensing (ISPRS) has introduced the Education and Capacity Buildings initiatives in 2017. The aim of the funding program is to consolidate the Society's international position in education training and networking in the domains relevant to the Society, i.e. of photogrammetry, remote sensing and spatial information sciences. The funded projects, whose expected duration is 132 months, are expected to focus on topics such as: education and learning programs, teaching and training materials and resources, e-learning techniques and platforms and online databases (GEOInformatics, 2019). As the ultimate outcome, the supported initiatives should enhance the visibility of the ISPRS, broadly promote its mission, widen its public and network of collaborations.

OBJECTIVES AND METHODOLOGY
Built upon the last years of efforts of interaction between the three communities, the LightCam project wants to offer two platforms to promote a common itinerary towards the convergence of a unified theoretical basis and a coherent vocabulary. The developed tools, envisaged as support to education and teaching, will be mainly addressed to students, beginners, and non-experts.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B5-2020, 2020 XXIV ISPRS Congress (2020 edition) Figure 2: Typical vision-based 3D measurement and modelling pipeline.

Image matching
Stereo correspondence problem Residuals of image coordinates Re-projection error Interior orientation parameters Intrinsic parameters Exterior orientation parameters Extrinsic parameters / Robot pose Image coordinates Camera coordinates Ground or object space coordinates World coordinates Dense image matching Multi-view stereo / densification Table 1: Examples of dictionary entries (adapted from Granshaw & Fraser, 2015).

The LightCam website
The first tool will be implemented as a knowledge base website, which is currently under construction (www.lightcam.fbk.eu), with the following envisaged sections: • Birth and history of photogrammetry, computer and robotic vision, with an emphasis on their contact points • Image formation with basics of optics and photography • Typical vision-based 3D measurement and modelling pipeline (Fig. 2), with emphasis on contact points and differences between the three communities • Dictionary for common concepts (Tab. 1) • List of relevant software solutions, websites, available courses resources and seminal articles from the three communities.

The LightCam code repository
The repository will contain pieces of code to transform from one common mathematical representation to another (e.g., different angle definitions, from one coordinate system to another, etc.). It will also show examples for the conversion between different software applications adopting different implementation strategies (e.g., Bingo, Photomodeler, Australis, OpenCV, etc.) and format files (*.nvm, *.out, etc.). The codes will be written in the most common programming language (e.g., Matlab, Python, C++, Java). An example is provided in Fig. 3.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B5-2020, 2020 XXIV ISPRS Congress (2020 edition) Figure 3: Example of exchange code, which will be included in the LightCam code repository.

EXPECTED OUTCOMES
LightCam specifically addresses the aim of 'generation of computer aided teaching and learning material (curriculum development, text books, etc.) in the photogrammetry, remote sensing and spatial information sciences' by providing cross community educational resources aiming at a stronger interconnection among photogrammetry, computer vision and robotic vision. LightCam will set an easy and lightweight internet platform covering from nomenclature to principles of visionbased techniques, as well as main algorithms from most popular open source framework or commercial software packages developed in the three different communities.
The web resources will be targeted to students, young researchers, new users and practitioners, fostering an easier dialogue and cooperation among the different communities. The benefits of the developed tools are expected to be even broader: Users coming from other fields (e.g. heritage, energy, water, healthcare, autonomous systems, transportation, traditional/robotic manufacturing, etc.) who employ vision-based methods will profit from the LightCam outcomes. This will further promote the ISPRS itself and its activities, pushing the liaisons with other communities beyond the current status quo. In line with the objectives of ISPRS, the LightCam ambition is to provide and disseminate photogrammetric knowledge to a more and more vast audience, highlighting the common interests with computer vision and robotics, but at the same time affirming the specificities of the three communities. The implementation simplicity of the LightCam web resources will facilitate its diffusion to a wide public, but, at the same time, the rigorousness of the shared information will be assured to guarantee high quality standards of the learning material.