ES2407105B1 - METHOD AND SYSTEM OF RECOVERY OF INFORMATION OF IMAGES IN MOBILE DEVICES USING METADATA. - Google Patents

Abstract

Method and system for recovering image information on mobile devices using metadata comprising the use of a mobile phone (100) and a remote server (102), and where essentially, a plurality of metadata is used as the rotation of the device, the acceleration at the time of the capture, audio metadata, network data, accuracy and GPS coverage, size of the region of interest, angle with respect to the horizontal, degree of exterior luminosity, metadata obtained by reverse geocoding, ground height and weather conditions for the interactive improvement of the identification of the object previously selected by the user.

Description

METHOD AND SYSTEM FOR RECOVERY OF IMAGE INFORMATION ON MOBILE DEVICES USING METADATA

5

  The present invention relates to a method based on a client / server architecture for retrieving information from images obtained by mobile devices.

1 o

Prior art

fifteen

The search for information about an object from its photography is useful in those problems where it is easier to describe the query through an image than through a textual description. Currently, there are many information retrieval systems from images. This task, which highlights systems such as Google images [http: / jimages.google.com], is known as content-based image retrieval.

2 or 2 5

To carry it out, a series of image characteristics (local or global) are normally extracted, processed using information grouping techniques, and finally a ranking of images similar to the query present in a database of information is obtained. reference. You can identify or classify the object by consulting, for example, the category of the first image of the ranking {for this it is essential that the database is labeled).

3 o

The main advantage of advanced mobile devices is the taking of photographs from any point and sending them over the internet. The recovery of image information obtained through these devices is a problem of recent study. Currently, the best known application that addresses this problem is Google Goggles [described in US20100260373], which is capable of recognizing wine brands, books, works of art, monuments and commercial logos.

3 5

Although recognizing any type of image in general is an extremely complicated problem, it can be simplified in part by using mobile devices, since these provide additional data (metadata) on the space / time location, user information and data provided by the

sensors to complement the image information.

5

These metadata can be useful to narrow your search. For example, if we take a picture of an object outside and know the time of the shot, we could improve the recognition by selecting only the images that have been taken around that time, since the lighting conditions will be similar.

1 O 15 2 O

We can find in the literature various techniques for extracting characteristics, both local (SIFf, SURF, etc.) and global (color, texture, etc.). After obtaining the characteristics of the image, distance-based metrics, clustering or histograms can be applied, or learning techniques, such as closer neighbors, support vector machines, or boosting, can be used to obtain the image class, or Image with greater similarity. To improve the results in the image search, it is convenient to restrict the number of objects with which to compare the image of the query. Metadata such as date or GPS location can facilitate the identification of the scene or object by reducing the search space. For example, using only camera metadata, good results can be obtained to discriminate between indoor and outdoor scenes [US2005 / 0105776].

2 5

The literature has studied the contribution of metadata provided by cameras [US2oosjo105776], or by mobile devices, although in a basic way [US20100260373].

Explanation of the invention.

3O 35

   The present invention has a number of differentiating characteristics. The first is the inclusion of a plurality of new metadata to improve the search, within a multimodal scheme. Another distinguishing feature is that it is an interactive system, in which the user selects the region of interest around the object and also validates the response given by the system, so that it can improve in future consultations. More specifically, the method of retrieving image information on mobile devices using metadata employs mobile devices that comprise

means for capturing images, as well as means for transmitting said captured images together with a plurality of metadata to a remote server.

5

The method is characterized in that it comprises a step of capturing an image by means of the mobile device, as well as the selection of a region within the image itself by the user of said mobile device; where in that region there is an object from which you want to obtain some information.

1 o

Subsequently, the selected region of the image is sent together with a plurality of metadata. After that, it is identified on the remote server and the identified object is classified, sending said information to the mobile device.

fifteen

Finally, the response of the remote server on the mobile device is validated by the user, indicating when that information is correct and, otherwise, labeling the object for return and storage in a database of the remote server.

20 25 3 O

In a second aspect of the invention, the system implementing the method described above is claimed. As you can see, it is a system and a customizable method, in the sense that it is intended for the user to decide what types of object you want to classify, and collaborative, in the sense that the images that a user is adding can be use to improve the queries of other users. Throughout the description and the claims the word quot; comprehequot; and its variants are not intended to exclude other technical characteristics, additives, components or steps. For those skilled in the art, other objects, advantages and features of the invention will be derived partly from the description and partly from the practice of the invention. The following examples and drawings are provided by way of illustration, and are not intended to be limiting of the present invention. In addition, the present invention covers all possible combinations of particular and preferred embodiments indicated herein.

3 5

Brief description of the drawings

   FIG 1.

Schematically shows the recovery system and method of

Image information on mobile devices using metadata.

Detailed presentation of an embodiment

5

The general scheme can be seen in the attached figure. The user can take

a photograph using the mobile device (100) and select the region

corresponding to the object from which information is to be obtained (106). The picture

included in this region is sent by network to the remote server (102) together with the

metadata described below. The remote server (102) later

1 o

Identifies or classifies the object and returns the information to the user, who can validate

the answer. The user can validate the answer, indicating if it is correct and, in case

otherwise, specifying the type or class of object.

For example, if the system identifies that a motorcycle is being photographed

fifteen

but it is a bicycle, the user can indicate that the answer is not correct

to add the tagged image to the server database.

It is proposed to use the metadata described below, extracted from

the sensors that some mobile devices have:

twenty

Device rotation. A gyroscope can provide information about

Tait-Bryan angles. These represent the axes of pitch, direction and

warping, from which more metadata such as angle can be obtained

respect to the horizontal.

25

Orientation. If the mobile device is rectangular, there are four

Possible orientations This data is relevant to rotate the image to

send it to the server, and also to calculate the angle with respect to the horizontal.

GPS position The coordinates at the time of the shot can be used to,

for example, choose only those objects close to the location of the

30

device. The true potential of the GPS position can be revealed

by reverse geo-coding.

Acceleration Data Using an accelerometer you can get the acceleration

respect each of the axes at the time of taking. These dates

they provide an indication on the stability of the device, and this

35

information can be used to select the most stable images as

   representative of a class.

Date. It is more likely, for example, to take a picture of a flower in

spring, or a snowman in winter, or a pool in summer. Information about the date can be useful for discerning between certain types of objects or scenes. Hour. The time it was made can also provide relevant information. It can be used, for example, to narrow the search on those objects that have been photographed at that time, thus comparing images with similar lighting conditions. Camera parameters Data such as shutter speed, aperture, exposure time, focal length, ISO, or flash mode are also relevant for this task. Focal length is a representative parameter of the distance of photographed objects, but currently the cameras of advanced mobile devices usually have fixed focal lengths. Audio metadata The decibel level at the time of taking or at a time interval around it can indicate whether the user is in a noisy environment (such as a shopping center) or quiet (such as a rural area). Network data If there is a nearby WiFi network, it is most likely that the mobile device is in a populated area. The density or quantity of wireless networks detected at the time of the capture can be used as an indication to obtain data about the environment. Accuracy and GPS coverage. As a general rule, at least one satellite is always detected in outdoor locations, but in indoor areas they are not detected because ceilings and walls prevent the signal from reaching the device. Therefore, the accuracy of the GPS position (in meters) or coverage (in number of satellites detected) are also relevant data for this task, and can indicate whether the scene is indoor or outdoor. User ID The same user tends to take photographs of similar objects (animals, office objects, etc.) and to repeat searches on objects already photographed. Therefore, it is possible to prioritize the categories of images previously taken by the user. Size of the region of interest. The height and width of the region of interest selected by the user can help to know if the object is elongated or compact.

From the above metadata, a series of additional metadata can be derived that reveal the true potential of mobile devices for this task:

Angle with respect to the horizontal. This information is relevant to know the size of the photographed object or, for example, to discern between objects located on the floor or placed on a wall. The angles of Tait-Bryan obtained by means of a gyroscope can be combined with the orientation of the mobile device to calculate the angle with respect to the ground at the time of the taking. Degree of exterior brightness. From the date, time and GPS latitude, it can be calculated if it is day, night, or if it is sunrise or sunset. This data can be used, for example, to deduce that if the shot has been taken at night and without a flash, an indoor scene is more likely to be treated than outside. Metadata obtained by reverse geocoding. Using systems such as simplegeo, google, geonames, etc., relevant data can be obtained from a location, such as:

o The type of place where the taking was made {zoo, university, forest, city, beach, etc.)

o The zip code of the area

o The city, town or region

o The country

o The population density in the area These characteristics are of interest to restrict the classification of certain objects. For example, it is very unlikely to find an elephant in a city, but it can be found in a zoo. Height above ground By reverse geocoding you can obtain the altitude above the sea in a certain area. By comparing this altitude with the altitude obtained by GPS, it can be deduced (with a certain margin of error) if the shot was made from, for example, a multi-storey building or at ground level. Weather conditions. Data relating to temperature, relative humidity or if it is raining or sunny can also be extracted from the GPS position, locating the nearest weather station and obtaining its information through systems such as simplegeo or geonames. This data can be relevant to know if the shot has been exterior or interior, if snow may appear in the image, etc. Orientation of the view on the map. The angles of Tait-Bryan in

Combined with the GPS position, it is possible to know the point towards which

He has taken the picture. For example, in an area with four points of interest,

if the photo has been taken in the central place of them, these could be used

angles to determine which point is being referenced.

5

Additional metadata from the camera metadata. For example,

from the exposure time and the aperture value you can obtain the

scene energy, which is directly correlated with the type of

scene and the different degrees of brightness.

1O

In the classification and identification of the image (103) the invention allows the use of

any existing system of classification or identification of images.

Normally, this task is done in three steps: feature extraction,

grouping of information and classification or identification. Depending on the

problem, the selected characteristics can be both points of interest (SIFf,

fifteen

SURF, Harris, etc.) as descriptors of color, texture, shape, etc.

Subsequently, the characteristics are usually grouped using quot; bag of wordsquot ;,

histograms, principal component analysis, etc. to reduce the cost

computational and improve ranking results. Optionally, you can

2O

use geometric verification to check the spatial coherence of

features.

Finally, the grouped characteristics of an image are compared with those of all

images from a tagged database (or with a subset of them)

2 5

to get a ranking of similar images. The most image tags

Similar provide information about the class of the object. The comparison is

you can perform calculating the nearest neighbors, given a defined distance or

using learning techniques to classify the sample, such as quot; support vector

machinesquot ;, Bayes networks, neural networks, etc.

30

Metadata can complement the image information to improve the

classification. Integration can be done in different ways:

Classification with image characteristics and subsequent filtering using

metadata

35

Initial filtering with metadata and subsequent classification with features

   of image.

Joint classification linking image characteristics with metadata.

In all three cases, machine learning techniques can be used, and in the case

from

that there is interactivity with he Username, these must be incremental The

information

can be integrated using a net Bayesian or other techniques from

5

statistical classification

In the generation of information (104),

one time the remote server (102) has

identified or classified the object, information about it is sent to the device

mobile (100). This information includes the type of object for the user to

1 o

validate it (optionally) and information additional. By example, yes he system

identify a specific picture of an artist, can provide information

about the author or similar works. This information can be textual, visual or

sound (for example, for an audio guide).

fifteen

Finally, the tagged sample is incorporated into the database (105).

validate the server response, indicating if the class is correct or not. In case

Yes, the new image is added to the database with the label obtained. In

case the answer

no is correct, the user can indicate the answer

correct and this will be added to the database with its new tag to improve

2 o

   Future consultations

Claims (4)

1.-Method of recovering information from images on devices

5

mobile using metadata; wherein said mobile devices (100) comprise

means for capturing images and means for transmitting said images

plus a plurality of metadata captured to a remote server (102);

that is characterized because

(i) capturing an image using a mobile device (100) and the

1 o

user selection of said mobile device (100) of a given

region within the image itself, where said region corresponds to a

purpose of which you want to obtain some information (106);

(ii) sending the selected region of the image along with a plurality of

metadata;

fifteen

(iii) identification (103) on the remote server (102) and classification of

identified object, sending said information (104) to the mobile device (too); Y

(iv) validation of the response of the remote server (102) on the device

mobile (100) by the user, indicating when that information is correct and, in

otherwise, labeling the object for return and storage in a

2 o

remote server database (102).

2. Method according to claim 1 wherein the metadata

They comprise at least the rotation of the device, the acceleration at the time of

takes, audio metadata, network data, accuracy and GPS coverage, size of the

2 5

region of interest, angle to horizontal, degree of exterior brightness,

metadata obtained by reverse geocoding, height above ground and

weather conditions.

3. Method according to claims 1 and 2 wherein the information

3 o

sent by the remote server (102) includes the type of object and information

additional regarding that object.

4.-System for retrieving image information on devices

mobile using metadata; wherein said mobile devices (100) comprise

3 5

means for capturing images and means for transmitting said images

plus a plurality of metadata captured to a remote server (102);

that is characterized because

implements means configured for the execution of the method of claims 1 to 3 and consisting of means for the selection by the user of said mobile device (100) of a region within the image itself, which corresponds to an object of which you want to get some information (106); means configured for sending the selected region of the image together with a plurality of metadata, comprising at least the rotation of the device, the acceleration at the time of the capture, audio metadata, network data, accuracy and GPS coverage, size of the region of interest, angle with respect to the horizontal, degree of exterior luminosity, metadata obtained by reverse geocoding, 1 O height above ground and weather conditions; and means for identification on the remote server (102) and classification of the identified object, by sending said information (104) to the mobile device (100).