Learning from heterogenous data

Paper details:
“learning from heterogenous data”.
For example the problem of learning representations of human faces (this is a very broad problem that is relevant
to face recognition, expression recognition, 3D face modelling, face editing and many other topics). Typically the
pipeline involves:
1. collecting a dataset
2. transforming it into a homogenous format
3. apply machine learning (e.g. manifold learning)
4. new data can be projectedÕÉ≠†Њ† into the representation so long as it is in the same form as the training data
As a concrete example, the famous Active Appearance Model would look like this:
1. data is 2D images of faces
2. face images are labelled with consistent set of landmark points
3. representation is learnt by applying PCA to normalised shape and appearance information
4. a new 2D face image can be represented if it is labelled with the same set of landmark points
The problem with this sort of pipeline is that every approach needs a dataset that exactly conforms to the format
requirements. In reality there are a huge range of datasets with different properties, e.g. 2D versus 3D, labelled
versus unlabelled, different noise properties, different resolutions, some missing data (e.g. 3D scans with holes),
3D shape only versus shape and texture, same person at different ages versus single age, multiview versus single
view and so on.
Approach is a neural network (probably a CNN) that has multiple inputs. E.g. have inputs for 2D image (perhaps
at different resolutions), labels, landmark positions, 3D data and so on. then provide one or more inputs to the
network and feed forward to the representation. probably tweak the architecture to allow prediction of some inputs
given other ones.