Data manipulation based on real world object manipulation

摘要

An aspect of the invention provides a method in the field of manipulating data based on real physical object manipulation. The method comprises receiving, by a computing device, a primary content dataset from a remote server, the primary content dataset including a subset of known visual references stored by the remote server that were determined to be most popular by the remote server, wherein each known visual reference corresponds to a virtual object model; storing, by the computing device, the primary content dataset in a local memory of the computing device; after receiving the primary content dataset, capturing, by an optical sensor of the computing device, an image of a physical object, the image including a captured visual reference associated with the physical object; searching, by the computing device, the primary content dataset stored in the local memory of the computing device based on the captured visual reference to identify a virtual object model corresponding to the captured visual reference associated with the physical object; in response to determining that the captured visual reference is not included in the primary content dataset in the local memory of the computing device, transmitting, by the computing device, the image to the remote server, causing the remote server to search the known visual references stored by the remote server based on the captured visual reference to identify the visual object corresponding to the captured visual reference; receiving, from the remote server, the virtual object model corresponding to the captured visual reference associated with the physical object; displaying, based on the virtual object model, a virtual image over a live image of the physical object; in response to detecting a movement of the computing device relative to the physical object caused by the user, modifying the virtual image; detecting a physical manipulation of the physical object; identifying a virtual function assigned to the physical manipulation of the physical object; and performing the identified virtual function, the virtual function controlling a physical device electrically unconnected to the physical object.