While imaging technology consistently increases in power and sophistication, each individual method can only view a narrow range of parameters owing to its specific mode of operation. These methods excel in their targeted applications, but are limited in other modes. Fusion of multiple images collected using different modalities offers the capability of creating a single fused image which is "greater than the sum of its parts"; such a fused multi-modal image can have increased resolution in every modality by taking advantage of data dependence between its constituent components. The goal of image fusion is to create a mixed-modality image that enhances the spatial resolution and spectral resolution over that obtainable using a lone technique. Methods such as Raman imaging and ion mass spectrometry, for example, can exceed common resolution limits by fusion with optical microscopy images (potentially even super-resolution images). Such techniques have significant implications especially in medical imaging, where multiple modalities are commonly used to assess and diagnose the extent of injury and disease. This work consists of the development of a technique to fuse mass spectoscopy images of tumor or brain tissue with optical microscopy images of chromatic stains. The technique uses novel spatial feature extraction and fusion using custom-designed neural networks. If ultimately successful, the methodology would be readily accessible to anyone using any combination of imaging techniques. Feature extraction focuses on segmentation (i.e., edge and region differentiation): by identifying features that correspond to real microstructures, extracted optical features have a significant likelihood of being related to specific chemical (i.e., spectral) signatures. Fusion via neural networks offers several advantages over traditional multivariate regression, as there are an arbitrary number of spatial features that are potentially nonlinearly related to a large number of chemical features (e.g. mass spectroscopy can have thousands of features).

Sensor fusion for biological imaging

256th National Meeting (2018)

The official archive and repository for all ACS Meetings & Expositions abstracts and select posters or presentations.