A method for training convolutional neural networks for image recognition using image conditioned mask language modeling

摘要

Problem to be solved: to provide a method of pre training convolutional neural network for image recognition based on mask language modeling.The method isEnter an image into a convolution neural network;Outputting a visual embedded tensor of a visual embedded vector from a convolution neural network;Step into tokens the captions;Randomly selecting one of the tokens in the list of masked tokens;Computing the latent expression of a token using a language model neural network;Step by step pooling visual embedded vectors in a visual embedded tensor;Steps to predict a masked token;Determining the prediction loss associated with the masked token;AndThe prediction loss is convoluted and propagated back to the neural network.Includes adjusting the parameter.Diagram