Robust multimodal collaborative visual recognition with missing data

摘要

Data contamination is one of typical difficulties that many computer vision practitioners encounter in real-world applications. With the increasing popularity of multi-view learning, the problem of imperfect data is even more pronounced. Specifically, there could be random missing features or even missing entire sensing channels in the testing phase, possibly due to interferences or bandwidth limits. In addition, cross-view paired correspondences could also be missing in the training data.;In this thesis, a series of missing data robust multi-view visual recognition methods are proposed to address these challenges. For the systematic and random missing of features in the testing data, a latent space based multi-view learning framework is developed. Paired with two types of information preserving projections and manifold embeddings algorithms, this framework effectively addresses the aforementioned data degradations and achieves superior recognition performances.;Inspired by the Regularized Generalized Canonical Correlation Analysis (RGCCA) and label information encoding, the Discriminative Canonical Correlation Analysis (DCCA) is proposed as the first type of supervised embedding algorithm. Alternatively, inspired by the recent success of metric learning and domain transfer learning, the Similarity Learning Canonical Correlation Analysis (SLCCA) is proposed to optimize the latent space with explicit category-preserving optimization constraints.;In addition, two variants of the aforementioned missing data problem are considered. In addition to the missing features in the testing phase, there could be missing correspondences among the training data. A new algorithm is proposed, which combines the effective subspace alignment technique and supervised information preserving embedding based on the squared-loss mutual information criterion. Alternatively, an asymmetric multimodal Convolutional Neural Network based approach is also proposed to jointly reconstruct the feature residuals and carry out classification.;With these recognition frameworks and new algorithms, multimodal visual classification is carried out on multiple benchmark datasets. We prove that these methods are robust against various data imperfections and outperform common baselines.