JP2001034632A - Image retrieving method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make easily retrievable an image by specifying, as reference data, the photographing place, the position or name of a subject and finding to what extent the data match the information attached to the data of a stored image. SOLUTION: The direction and photographic power of a camera at the time of photographing are superposed on map information and a photographed subject is specified and stored in a database in sequence. A specification pattern for retrieval is read in and superposed on the map information to perform conversion to retrieval specification information of a user, i.e., positional information on the subject. Then, an original image database is read in and the read information is collected with tabled specific patterns for retrieval. Here, the columns and rows of the table are collated sequentially and matching columns or rows, when found, are taken out as a result of retrieval. Then the degree of matching (similarity) with a plurality of candidate subjects in the original image database is calculated and retrieved images are outputted in the decreasing order of the points of the degree of matching.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image retrieving method, and more specifically, a method of extracting an image of a subject from position information, camera information and map information at the time of photographing, and using the extracted image for retrieval. To an image search method.

[0002]

2. Description of the Related Art In recent years, for the purpose of taking recorded photographs,
There are cameras on the market that can automatically record when and where a picture was taken. This camera is a GPS (global posit) position measurement device using artificial satellites that is often used in car navigation systems (car navigation systems).
ioninng system) and a combination product of a direction sensor and a camera (Photo Industry, 1996-7, pp.12-1)
5, 84-87).

[0003] In the role of photography, the importance of recordability and testimony of photographs is one of the essential values possessed by photographs. Therefore, in the recorded photograph,
Ancillary information indicating “when and where” was taken plays an important role. Conventional cameras have provided some information about "when" but not information about "where".

The camera introduced in the above-mentioned photo magazine (hereinafter referred to as a camera with a built-in GPS) is, in this sense,
It can be said that the recorded photograph fulfills its original function.
That is, first, the difference between the date function of the conventional (previously) so-called camera with a date function and the time display of the camera with a built-in GPS is a time that can be determined privately by the former, whereas The testimony is crucially different in that it is the "official time" transmitted from the satellite.

On the other hand, the position information indicated by the GPS built-in camera is not exactly the position of the subject but the position of the camera and the direction of shooting. Therefore, when the photograph is a distant view, the position of the subject is not always clear. Also in this case, if necessary, the position of the subject can be accurately calculated from the distance between the plurality of subjects in the image.

[0006]

The inventor first obtains at least one piece of photographing information of a photographing position, a photographing azimuth and a photographing magnification in association with a photographed image. And Japanese Patent Application No. Hei 10-16213, which discloses a photographic printing method for obtaining additional information related to at least a part of the method and adding the obtained additional information to a photographic print.
No. 0 proposes.

According to the above-described photographic printing method and apparatus, the customer can easily recall the situation at the time of shooting by looking at the image based on the shooting information added in association with the shot image. This is effective not only in enhancing the memo function of the photo print, but also in providing the photo print with entertainment (may be referred to as topic providing).

[0008] The present invention has been made to make the above technology easier to use.
When a customer searches for a desired image from among a large number of stored images, the above-mentioned position information, camera information, and map information at the time of shooting added in association with the shot image are used. Accordingly, an object of the present invention is to provide an image search method that enables easy image search.

[0009]

In order to achieve the above object, an image search method according to the present invention extracts a shooting location or a subject name and a position on a map from shooting information and map information at the time of shooting. The extraction result is attached to the image data as attached information and stored, and the position or name of the shooting location, or the position or name of the subject is specified as reference data, and the attached information attached to the stored image data and Is obtained, and an image having a degree of coincidence equal to or more than a predetermined value is read out.

In the image search method according to the present invention, the photographing information is acquired in association with a photographed image, wherein a first data structure comprising a photographing position, a second data structure comprising a photographing position and a photographing direction, It is characterized by one of a third data configuration consisting of a shooting position and a shooting magnification, and a fourth data configuration consisting of a shooting position, a shooting direction and a shooting magnification.

The image retrieval method according to the present invention is characterized in that an approximate depth is designated by arranging a patterned subject in a virtual map space as reference data relating to the photographing position and the subject position. . Further, in addition to the extracted information, a position and a size of a shooting place or a subject in a screen are extracted, and the extracted locations and sizes are added to the additional information.

[0012] In the image search method according to the present invention, it is preferable that, in addition to the extraction information, a position and a size of a shooting place or a subject in a screen are added to the designation information.

Further, the image retrieval method according to the present invention derives distance information of a photographing screen from position information, camera information and map information at the time of photographing, and stores the derivation result attached to image data as additional information. It is characterized in that the distance information of the shooting screen is designated as reference data, the degree of coincidence with the attached information attached to the stored image data is obtained, and an image having a degree of coincidence equal to or more than a predetermined value is read.

The image retrieval method according to the present invention is characterized in that, as reference data relating to the distance information of the photographing screen, the depth of each region in the photographing screen is roughly specified. In addition, an outline of the depth is designated by arranging a patterned subject in a virtual map space as reference data relating to the distance information of the shooting screen.

In the image retrieval method according to the present invention, a difference between simulation image data of a photographed image created from position information, camera information and map information at the time of photographing and an actual photographed image is stored. When outputting an image, the storage data capacity is reduced by outputting an image obtained by adding the stored difference to simulated image data of a captured image created from the position information, camera information, and map information at the time of shooting. Is preferred.

[0016]

Embodiments of the present invention will be described below in detail with reference to preferred embodiments shown in the accompanying drawings.

FIGS. 1 and 2 are diagrams for explaining an image retrieval method according to an embodiment of the present invention. FIG. 1 shows an original image 10 on a film taken by a camera with a GPS function.
FIG. 2 shows a search specification pattern 20 input by the user using a mouse or the like on the screen of the terminal device when searching for the original image 10.

In the original image 10 shown in FIG. 1, reference numeral 11 denotes a person who is a main subject, reference numeral 12 denotes a mountain (hereinafter, referred to as a circle for convenience) which is a subject effective for specifying a shooting position described later, and reference numeral 13 denotes a mountain. Indicates a well-known building (also referred to as an X building) which is also a subject effective for specifying the shooting position.

The search specification pattern 20 shown in FIG.
Has expressed and input, for example, a desire such as "I want to search for a scene where there is a mountain somehow around the T1 area and there is a building somehow around the T2 area" as a search specification pattern. In the example, reference numeral 22 denotes a display indicating an existing position of a mountain, and reference numeral 23 denotes a display indicating an existing position of a building.

First, as a preparatory operation in the image search method according to the present embodiment, the outline of the operation of specifying the position and name of the subject whose position in the original image 10 shown in FIG. The operation will be described with reference to the operation flowchart shown in FIG.

As shown in FIG. 3, when an image is read, the GPS added at the time of the above-mentioned photographing is added to the image.
Information, the direction of the camera at the time of shooting, and the shooting magnification are also read (step 101). By overlaying the GPS information on the map information prepared in advance,
An imaging point is specified (step 102).

Next, the subject to be photographed is specified by superimposing the direction of the camera and the photographing magnification at the time of photographing on the map information (step 103). In this process, a simulation image may be created based on the camera direction and the photographing magnification at the corresponding position on the map, and the subject may be specified by pattern matching with the photographed image. The identified subjects are sequentially stored in the database (step 104).

Table 1 shows an example of information attached to image data of an original image created based on photographing position information added as additional information to the original image. As shown in Table 1, here, the position is known in advance based on the GPS information at the time of shooting and the direction of the camera at the time of shooting.
Mountains, x-buildings, etc. are recorded along with their positions and shapes on the screen, their positions and names on the map. Note that the shooting date and time, shooting location information (latitude, longitude, place name), and other camera information are also registered in the database as additional information of the shot image.

[0024]

[Table 1]

In the present embodiment, since the data recording step is automatically performed, it is possible to save the trouble of inputting a subject name as search data when registering image data in the database. Is obtained.

A user who wants to search for an image inputs position information (latitude, longitude) of a subject on the screen, a place name or a name, etc. as search information. This input may be performed using a mouse or the like on the screen of the terminal device, such as the designated pattern for search 20 shown in FIG. In this case,
By specifying the name of a mountain or a building, the position is specified indirectly.

Next, the flow of operation in this embodiment for searching for an image based on the data stored as described above will be described with reference to FIG.

First, a search specification pattern as shown in FIG. 4 is read (step 201), and is superimposed on map information to be converted into user search specification information, ie, subject position information (step 201). 202).
If the user inputs the position information, these steps may be skipped.

Next, the original image database is read (step 203), and the original image database is collated with the table-designated search designated pattern (step 204). This collation is performed for each column and each row of the table in order, and if there is a match, it is extracted as a search result (steps 205 to 209).

Here, in order to simplify the explanation,
In the case of matching, it was explained that "if there is a match, it is taken out." However, in practice, it is rare that there is a perfect match, so as described later, the match degree is defined and the calculated match degree is determined. However, it is preferable that those that exceed a predetermined threshold value be matched.

As a method of calculating the degree of coincidence described above, for example, a method using information such as center coordinates, size, and vertical direction in the auxiliary information shown in Table 1 is used for a plurality of candidate subjects in the original image database. The degree of coincidence (or similarity) is calculated, and the retrieved images are output in descending order of the degree of coincidence.

According to the image search method shown in this embodiment,
The search specification pattern 20 input by the user using a mouse or the like on the screen of the terminal device, the position information (latitude, longitude) of the subject in the screen input by the keyboard as search information, a place name or a name, etc. Thus, it becomes possible to search for an image that matches the instruction.

Next, another embodiment of the present invention will be described. In the above-described embodiment, an example in which a database created from the original image and its photographing information is used has been described.In this embodiment, however, scene determination or collation is performed using depth information in an image. explain. Note that the present embodiment is a case in which photographing position information and map information are not used. Here, as the depth information, it is preferable to use information derived from a multi-stage focus image / stereo image, which is a known camera technology.

FIG. 5 shows an example of the search designation pattern 30 including the depth information as described above. In the example shown in FIG. 5, a circle 31 is a person, and a farther place in the upper direction corresponds to a mountain (a horizontally long ellipse 32), and a square 33 on the right side.
Is equivalent to a building.

Specifically, in the example shown in FIG. 5, the person 31 is located at a distance of about 5 to 10 m as viewed from the camera, and the building 33 is located at a depth of about 100 m on the right side.
This expresses a scene in which a mountain 32 exists 1000 m behind the person. Note that the designation of the position of the subject on the screen shown in FIG. 2 and the arrangement based on the shape of the subject shown in FIG. 5 can be used in combination.

When the user specifies the image to be searched by expressing the depth of each rough area of the scene by the layout diagram as shown in FIG. 5, the contents and a plurality of candidates in the original image database are specified. The degree of coincidence is calculated for the subject, and the retrieved images are output in descending order of the points of the degree of coincidence.

According to the image retrieval method shown in this embodiment,
By adding a two-dimensional arrangement state of subjects to the search specification pattern 20 input by the user, there is an effect that it becomes easier to search for an image that matches the instruction. The image search method according to the present embodiment can be applied to a moving image. In this case, motion information between frames of a moving image can also be used.

Next, still another embodiment of the present invention will be described. In the present embodiment, a case will be described in which there is an arrangement diagram showing an arrangement state between subjects. Here, the arrangement diagram may be, for example, the one shown in FIG. 5, as long as the planar positional relationship of a mountain or a building with respect to a person who is a main subject can be understood.

Here, as the position information, the aforementioned GPS
Information or PHS (Personal Handyphone System)
(In this case, the accuracy of the position determination is currently much lower than that of the GPS) or more general position specification information (for example, XX city OO town △△ chome) Can be used.

Thereafter, a matching process is performed by comparing the layout on the map data with the layout map used for designation, and the degree of coincidence is calculated for each candidate. If there is no camera direction information, the camera direction is rotated by a predetermined angle (0 to 3).
60 °) to perform a simulation. If the accuracy of the position is low, the simulation is performed by performing parallel movement within a predetermined range.

According to the present embodiment, it is possible to search for an image using information such as terrain by a simple instruction, and it is possible to obtain an effect of increasing the efficiency of image search. .

By the way, conventionally, when searching for an image as described above, the position information and the viewpoint direction are not shown in FIG.
It was often given in the form shown in (b). That is, a camera position (GPS information) and a shooting direction (camera information). On the other hand, if the information as shown in FIG. 6A, that is, the information on the object position and the photographing direction is obtained, the following can be performed based on the information.

In other words, since it is known what is being photographed, an image of the photographed subject is taken out of a prepared image database, and
Create a computer graphic image based on this,
After performing position adjustment, size adjustment, and the like by comparison with an actual photographed image, a difference between the two is obtained, and the data is appropriately compressed as necessary to be stored data.

As the simulation image, in addition to the above-described computer graphic image, an image that has been previously photographed and is considered to be closest to the position information and camera photographing information, or a combination of this image and the above-described computer graphic image Are available. Whichever image is used, the amount of stored data can be reduced.

Each of the above embodiments is merely an example of the present invention, and it is needless to say that the present invention is not limited to these embodiments.

[0046]

As described above in detail, according to the present invention, when a customer searches for a desired image from among a large number of stored images, the above-described captured image By using the position information, camera information, and map information at the time of shooting added in association with each other, it is possible to realize an image search method capable of easily searching for an image.

In other words, according to the image search method of the present invention, a search specification pattern input by a user using a mouse or the like on a screen of a terminal device, or an object in a screen input by a keyboard as search information. It is possible to search for an image that matches the instruction based on the location information (latitude, longitude), place name, name, or the like.

In addition, by adding a two-dimensional arrangement of the subjects to the search designated pattern input by the user, it is possible to more easily search for an image that matches the instruction. Note that the image search method according to the present invention can basically be applied to moving images.

[Brief description of the drawings]

FIG. 1 is a diagram showing an original image 10 on a film taken by a camera with a GPS function, for describing an image search method according to an embodiment of the present invention.

FIG. 2 is a diagram showing an example of a search specification pattern 20 input by a user when searching for the original image 10 shown in FIG.

FIG. 3 is an operation flowchart showing an outline of an operation of specifying a position and a name of a subject whose position in the original image 10 can be specified as a preparation operation in the image search method according to one embodiment.

FIG. 4 is an operation flowchart showing a flow of an operation of searching for an image based on stored data in the image search method according to one embodiment.

FIG. 5 is a diagram illustrating an example of a search designation pattern including depth information according to another embodiment.

6A and 6B are diagrams showing a relationship between position information and a viewpoint direction. FIG. 6A shows a case where a subject is known, and FIG.
Indicates a case where the user can only know where the image was taken but cannot know what the subject is.

[Explanation of symbols]

 Reference Signs List 10 original image 11 person 12 mountain (○ mountain) 13 building (× building) 20 designated pattern for search 22 display showing location of mountain 23 display showing location of building 30 search designated pattern 31 circle corresponding to ( ) 32 Mountain oval oblong) 33 Building (equivalent square)

Claims (9)

[Claims] 1. A photographing place or subject name and a position on a map are extracted from photographing information and map information at the time of photographing, and the extraction result is attached to image data as attached information and stored. A name or a position or a name of a subject is designated as reference data, a degree of coincidence with attached information attached to the stored image data is obtained, and an image having a degree of coincidence equal to or more than a predetermined value is read. Image search method. 2. A first data structure comprising a photographing position, a second data structure comprising a photographing position and a photographing direction, and a first data structure comprising a photographing position and a photographing magnification, wherein the photographing information is acquired in association with a photographed image. 2. The image search method according to claim 1, wherein the image search method is any one of the data structure of No. 3, and a fourth data structure including a shooting position, a shooting direction, and a shooting magnification. 3. An approximate depth is specified by arranging a patterned subject in a virtual map space as reference data relating to the photographing position and the subject position.
Or the image retrieval method according to any one of 2. 4. The image search method according to claim 1, wherein, in addition to the extracted information, a position and a size of a photographing place or a subject in a screen are extracted and added to the attached information. . 5. The image search method according to claim 1, wherein, in addition to the extraction information, a position and a size of a shooting location or a subject in a screen are added to the designation information. 6. Deriving distance information of a photographing screen from position information, camera information and map information at the time of photographing, storing the derivation result attached to image data as additional information, and using the distance information of the photographing screen as reference data. Specify and determine the degree of coincidence with the attached information attached to the data of the stored image,
An image search method comprising reading an image having a degree of coincidence equal to or greater than a predetermined value. 7. The image search method according to claim 6, wherein a depth of each area in the shooting screen is roughly specified as reference data relating to distance information of the shooting screen. 8. The image search method according to claim 6, wherein an approximate depth is specified by arranging a patterned subject in a virtual map space as reference data relating to distance information of the photographing screen. 9. A difference between simulation image data of a photographed image created from position information at the time of photographing, camera information and map information and an actual photographed image, and when outputting a search result image, the position at the time of photographing is stored. The image search according to any one of claims 1 to 8, wherein an image obtained by adding the stored difference to simulation image data of a captured image created from information, camera information, and map information is output. Method.