Scalable Data Systems for LiDAR and Imagery Data Integration

The massive volumes of 3D LiDAR and 2D imagery data collected from remote sensing mappings are increasingly used in both independent and integrated form in geospatial applications. To facilitate the independent and integrated usage of these data, the development of scalable LiDAR and imagery integrated data systems is essential. However, currently, there’s no tailored technique such as an integrated index that performs the integration of LiDAR and imagery data. Moreover, existing image indexes are also not designed to index raw images. Equally, a binary encoding technique that could be leveraged for scalable LiDAR data management is not leveraged. Finally, there’s no qualitative method that guides the development of scalable LiDAR and imagery integrated data systems. To address these challenges, this thesis leverages the raw LiDAR and imagery data provided by the UrbanARK project, which funds this Ph.D. Thereby, this thesis innovates novel data management techniques that can be employed to develop scalable LiDAR and imagery integrated data systems, such as the UrbanARK data system presented in this thesis. Firstly, to guide the development of LiDAR and imagery integrated data systems, this thesis introduces a qualitative method. This method is presented as Scalable Data Systems(S) Approach(A) for LiDAR(L) and Imagery(I) Intergration(I), hence termed SALII. SALII methodology synthesizes the fundamental areas and lessons learnt from the literature. Using the SALII methodology, this thesis identifies a state-of-the-art integrated data system for the concrete implementation of the required indexes for raw imagery management LiDAR and imagery integration, and integrated data system necessary to leverage binary encoding for LiDAR data management. Secondly, to address the gaps in scalable indexing of raw imagery, this thesis introduces two index optimizations: the 4 Step Algorithm (4SA) and the Designing Adjacent Cell Merge Algorithms (DACMA), along with an index termed the 4-Dimensional Hilbert Index (4DHI). Both 4SA and DACMA support window queries, which are widely used in spatial data systems. The 4DHI is designed to perform approximate k-nearest neighbors search, an aspect that has received little attention in the literature. The methodology for all these indexes is based on the Hilbert Space Filling curve-based technique. Scalability evaluations demonstrate that all proposed indexes can significantly reduce index storage requirements while achieving improved query performance. Thirdly, in addressing the challenges in LiDAR data management, this thesis leverages a state-of-the-art binary data encoding technique built on Google Protocol Buffers. This thesis demonstrates the suitability of this encoding technique through an evaluation of LiDAR storage, ingestion, and querying, in comparison to alternative text encoding methods.Finally, to facilitate the effective integration of LiDAR and imagery, this thesis introduces the LiDAR and Imagery Spatial Integration Index, referred to as LISII. LISII is grounded in metadata relevant to LiDAR and imagery datasets and makes use of a combined R-tree and on-the-fly kD-tree index to optimise metadata filtering. Experiments conducted reveal that employing a 2D R-tree with an on-the-fly index significantly enhances query performance, particularly in scenarios involving scalability to larger regions and concurrent user queries.