Title: Training multimodal systems for classification with multiple objectives
Authors: Armitage, Jason
Thakur, Shramana
Tripathi, Rishi
Lehmann, Jens
Maleshkova, Maria 
Language: eng
Keywords: Machine Learning;Multimodal Data;Probabilistic Method
Issue Date: 1-Jan-2020
Document Type: Conference Object
Journal / Series / Working Paper (HSU): CEUR Workshop Proceedings
Volume: 2611
Page Start: 1
Page End: 15
Published in (Book): CLEOPATRA 2020: cross-lingual event-centric open analytics : proceedings of the 1st International Workshop on Cross-lingual Event-centric Open Analytics
Publisher Place: Heraklion
Conference: 1st International Workshop on Cross-lingual Event-centric Open Analytics co-located with the 17th Extended Semantic Web Conference (ESWC 2020) Heraklion, Crete, Greece, June 3, 2020
We learn about the world from a diverse range of sensory information. Automated systems lack this ability as investigation has centred on processing information presented in a single form. Adapting architectures to learn from multiple modalities creates the potential to learn rich representations of the world - but current multimodal systems only deliver marginal improvements on unimodal approaches. Neural networks learn sampling noise during training with the result that performance on unseen data is degraded. This research introduces a second objective over the multimodal fusion process learned with variational inference. Regularisation methods are implemented in the inner training loop to control variance and the modular structure stabilises performance as additional neurons are added to layers. This framework is evaluated on a multilabel classification task with textual and visual inputs to demonstrate the potential for multiple objectives and probabilistic methods to lower variance and improve generalisation.
Organization Units (connected with the publication): Universität Bonn
ISSN: 16130073
Publisher DOI: 10.48550/arXiv.2008.11450
Appears in Collections:6 - Publication references (only metadata) of your publications before HSU

Show full item record

CORE Recommender

Google ScholarTM




Items in openHSU are protected by copyright, with all rights reserved, unless otherwise indicated.