JP2008310446A - Image retrieval system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image retrieval system capable of evaluating the similarity of a photographed image, without being affected by a disturbance, a shift in the photographed image and the like, and capable of finding precisely positional information of an image photographing terminal. <P>SOLUTION: This image retrieval system is provided with the image photographing terminal 10 for acquiring the photographed image with a photographed landscape in order to present a candidate of a current position using the photographed image, and a server 20 for judging the similarity of the photographed image photographed by the image photographing terminal 10 to the photographed image held on a database 30, for specifying the similarity-recognized photographed image held on the database 30, as a candidate image, and for reply-transmitting the positional information held on the database 30 correlatedly with the specified candidate image, to the image photographing terminal 10. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a position specifying method for specifying the position of an image capturing terminal such as a mobile phone, an image comparison method for improving the accuracy of comparing the similarity between a captured landscape image and an image stored in a database, and The present invention relates to an image search system used for realizing the method.

  Conventionally, portable terminals such as cellular phones use the GPS (Global Positioning System) technology to specify the current position and provide route guidance. Since this method uses radio waves from GPS satellites, it does not function in places where signals do not reach. Also, a method of specifying a position from the strength of the radio wave using a base station with which a mobile terminal communicates is known, but there is a problem that sufficient accuracy cannot be obtained.

For this reason, a place where radio waves from GPS satellites do not reach by specifying the position by comparing the landscape image captured by the mobile terminal with the data obtained by converting the map information held on the server into 3D (three-dimensional) However, a technique for specifying the position is disclosed (for example, see Patent Document 1).
JP 2003-111128 A

  However, in the conventional configuration, the image changes even when the same position is photographed due to the influence of disturbance such as building extension, reconstruction, removal or new construction, moving objects such as people and plants, or reflection of automobiles. Since there are many factors that adversely affect the similarity with 3D data, there are problems that the position cannot be determined and the possibility of erroneous determination is high.

  There is also a problem that maintenance is indispensable to prevent comparison 3D data from becoming useless due to expansion, reconstruction, removal, or new construction of a building.

  The object of the present invention is to take a picture without being affected by the expansion, reconstruction, removal or new construction of a building, the influence of disturbance such as a moving object such as a person or a plant or the reflection of an automobile, or the deviation of a photographed image. To provide an image search system that can evaluate the similarity of images and can obtain the position information of the image capturing terminal with high accuracy.

  An image search system according to the present invention is an image search system that presents a candidate for a current position using a captured image, and is captured by an image capturing terminal that acquires a captured image of a landscape and the image capturing terminal. The photographed image and the photographed image held in the database are judged to be similar, the photographed image retained in the database in which the similarity is recognized is identified as a candidate image, and the identified candidate image A candidate image determination unit that returns the positional information associated with and held on the database to the image capturing terminal.

  In this image retrieval system, it is possible to create a database only with captured images and their position information, and to specify position information from the captured images, so that the expansion, renovation, removal or new construction of buildings, movements of people, plants, etc. Because it is possible to evaluate the similarity of images without being affected by disturbances such as reflection of a certain object or car, or by the shift of the captured image, the position information of the image capturing terminal must be obtained with high accuracy. Can do.

  The candidate image determination unit stores a first image evaluation unit that extracts a plurality of comparison target objects from a captured image captured by the image capturing terminal and calculates a distance between the comparison target objects, and stores the comparison target object on the database. A second image evaluation unit that extracts a plurality of comparison target objects from a captured image and calculates a distance between the comparison target objects; a distance between the comparison target objects calculated by the first image evaluation unit; Based on the distance between the comparison target objects calculated by the second image evaluation unit, the similarity between the captured image captured by the image capturing terminal and the captured image stored on the database is determined. It is preferable to include a comparison processing unit.

  In this case, by evaluating the similarity of the image using only the distance between the objects to be compared, disturbances such as expansion, reconstruction, removal, or new construction of a building, moving objects such as people or plants, or reflection of a car It is possible to evaluate the similarity of images without being affected by the influence of the above and the deviation of the photographed image.

  The image photographing terminal preferably draws a guideline around the comparison target object when the comparison target object exists on the preview image.

  In this case, the user can smoothly place a plurality of comparison target objects in the captured image, and the number of comparison target objects used for evaluating the similarity of images increases, so that the similarity can be evaluated more accurately. Can do.

  The comparison processing unit notifies the image photographing terminal to increase the number of comparison target objects on the preview image when there are a plurality of candidate images similar to the photographed image photographed by the image photographing terminal. Preferably, the image photographing terminal presents the notification content to the user when receiving the notification.

  In this case, when the captured image does not match the image in the database or when it matches a plurality of images, a new comparison target is requested by requesting the user to include a new comparison target object in the captured image. Since the similarity can be evaluated by the object, the frequency with which the captured image matches the image in the database can be increased.

  The database preferably stores in advance a result of extracting a plurality of comparison target objects existing in the held captured image and calculating a distance between the comparison target objects in association with the captured image. .

  In this case, by extracting the comparison target object of the captured image in the database in advance, the process of extracting the comparison target object every time the similarity of the captured image is evaluated can be reduced.

  The database preferably holds position information for each comparison target object.

  In this case, by comparing the data in the database with the comparison target object unit, it becomes possible to compare the comparison target objects included in different images. Therefore, the comparison target object in the captured image and the comparison target object in the database When comparing the similarity with, the probability of matching can be increased, and the frequency with which position information can be specified can be increased.

  When the similarity between the captured image captured by the image capturing terminal and the captured image stored on the database is high, the candidate image determination unit selects the captured image stored on the database. It is preferable to replace the image with a captured image captured by the image capturing terminal.

  In this case, when the comparison object in the database has not been updated even though the map has actually changed due to expansion or renovation of the building, etc., the latest image is used and stored in the database. The captured image can be replaced with an image representing a correct landscape, and the accuracy of the subsequent similarity evaluation of the image can be increased.

  Preferably, the candidate image determination unit adds a new comparison target object to the database.

  In this case, by adding information about a new comparison target object, the number of comparison target objects increases, so that the accuracy of evaluating similarity can be increased.

  The candidate image determination unit determines the similarity between the captured image captured by the image capturing terminal and the captured image stored in the database, and when the similarity is recognized, the image capturing terminal It is preferable to calculate the inclination of the image and send it back to the image capturing terminal.

  In this case, since the tilt information can be used at the image capturing terminal, it is possible to present the tilt to the user or to use the tilt information inside the image capturing terminal.

  According to the image retrieval system of the present invention, it is affected by the expansion, reconstruction, removal or new construction of buildings, the influence of disturbances such as moving objects such as people and plants, or the reflection of automobiles, and the deviation of captured images. Therefore, since the similarity of images can be evaluated, the position information of the image capturing terminal can be obtained with high accuracy.

Hereinafter, image search systems according to embodiments of the present invention will be described in detail with reference to the drawings.
(First embodiment)
FIG. 1 is a block diagram showing a configuration of an image search system according to the first embodiment of the present invention. In the image search system shown in FIG. 1, the image capturing terminal 10 transmits image data 40 of a captured landscape to the server 20, and the server 20 determines the received image data 40, the image data, and the shooting position of the image data. Image data similar to the image data 40 is searched by performing matching processing with each image data in the database 30 in which the photographing position information to be represented is recorded in pairs. The server 20 transmits shooting position information 50 managed as a pair with the matched image data to the image shooting terminal 10, and the image shooting terminal 10 draws the shooting position information 50 on an LCD (liquid crystal display) 60. The shooting position information is presented to the user, and the user can specify the current position.

  FIG. 2 shows an example of a data configuration in the database 30 that manages image data and shooting position information of the image data in pairs. Here, the shooting position information is, for example, coordinates on the map and shooting direction.

  FIG. 3 shows a functional block diagram of the image capturing terminal 10 and the server 20 shown in FIG. 3 illustrates an example in which the database 30 illustrated in FIG. 1 is included in the server 20 as the database 26, the database may be separate from the server as illustrated in FIG. It may be provided inside the server. The same applies to other embodiments described below.

  The overall flow of the current position specifying process by the image search system shown in FIG. 1 will be described with reference to FIG. The camera image capturing unit 12 of the image capturing terminal 10 captures a captured image of a landscape captured by the camera 11 as image data. The camera image transmission unit 13 transmits the captured image data to the server 20.

  Here, the image data refers to a still image unless otherwise specified. However, since a moving image is a sequence of still images, it may be a moving image, and is not limited to a still image. Further, the image data may be compressed. In this case, the compression method of the image data may be any compression method, and the compression unit and the decompression unit are not related to the present invention, and thus the description thereof is omitted.

  The image receiving unit 21 of the server 20 receives the image data and temporarily stores the image data in the image data temporary recording unit 22. The comparison processing unit 24 of the server 20 monitors that the image data is received by the image receiving unit 21, and when the image data is received, temporarily stores the image data in the first image evaluation unit 23. An instruction is issued to extract image data from the unit 22. The first image evaluation unit 23 analyzes a comparison target object (hereinafter abbreviated as “object”) in the image data. In addition, a candidate image determination unit 29 is configured by the first image evaluation unit 23, the comparison processing unit 24, and the second image evaluation unit 25.

  In the present embodiment, a sign is used as an example of an object. However, the object is not particularly limited to a sign, and various objects may be used as long as they are stationary on the landscape. The user of the image search system can register object information representing an object in the object database 28.

  The first image evaluation unit 23 obtains object information to be passed to the comparison processing unit 24 from the received image stored in the image data temporary recording unit 22 and the object information stored in the object database 28. The object information to be obtained is information representing the position of the object (the position in the horizontal direction, the vertical direction and the depth direction) with the lower left position of the image data captured by the image capturing terminal 10 as the origin.

  The object information (object position) obtained here is obtained by converting the position obtained in pixel units into an actual scale so as not to be affected by the camera resolution of the image capturing terminal 10. This is because it is necessary to compare images captured at various resolutions stored in the database 26 on the server 20 side with captured images captured by the image capturing terminal 10.

  There are various actual scale units such as meters, inches, degrees / minutes (units mainly used for expressing latitude and longitude), and any unit may be used. In the following description, a meter will be described as an example. In addition, since the position of the object in the first image evaluation unit 23 may be a relative position when an arbitrary point in the three-dimensional space is set as the origin, the origin position of the object position is not necessarily the lower left position of the image data. The origin is not limited to any point in the image data.

  FIG. 4 shows an example of the data structure of the object database 28 shown in FIG. The object database 28 holds object type, object image data, and object size for each object. Here, the type of object is an identifier for identifying the object, and various things such as numbers and symbols can be used as long as they are unique. Further, in this embodiment, for simplification, description is made using the horizontal size (W) and the vertical size (H) as the size of the object. Various other parameters may be added to represent complex shapes. Furthermore, the shape of the object is not limited to a quadrangle, and when the shape is complicated, a rectangle including the object may be used as the size of the object.

  First, the first image evaluation unit 23 compares the image data held in the image data temporary recording unit 22 with the object image data in the object database 28, and determines the object position in pixels (horizontal and vertical positions), The object size (horizontal and vertical positions) is obtained. As an example, FIG. 5 shows an example in which the three objects A, B, and C shown in FIG. 4 are extracted from the image data ID and their positions and sizes are obtained.

  Since the comparison method can use an existing image recognition technology, a specific description of the image recognition technology is omitted in this embodiment, but an example of the existing image recognition technology is a camera. Since images are usually subject to perspective correction, for example, Masakazu Iwamura and three others, "Character recognition in scene images by high-speed template matching that allows deformation", Image Recognition and Understanding Symposium (MIRU2006) It is desirable to use a technique that can be applied even to images subjected to perspective correction, such as the technique disclosed in Proceedings, IS2-42, pp.906-911, (2006-7). The image data and the object image data are compared using these image recognition techniques, and the position and size of the object in units of pixels are acquired.

  Next, a method for converting the object position (horizontal direction and vertical direction) obtained in pixel units into an actual scale (meters) will be described. The calculation formula is shown below.

X = xx (W / w) (1)
Y = y × (H / h) (2)
In the above formulas (1) and (2), X and Y are the horizontal and vertical object positions on the real scale, x and y are the horizontal and vertical object positions in pixel units, and W and H are The horizontal and vertical object sizes, w and h in the object database 28 are the horizontal and vertical object sizes in pixel units.

  Next, a method for obtaining the depth at the object position will be described. The calculation formula is shown below.

Z = Zccd × (W / w ′) (3)
In the above equation (3), Z is the object position in the depth direction of the actual scale, Zccd is the focal length of the camera 11, and W is the object size in the horizontal direction in the object database 28. w ′ is a value calculated by the following equation.

w ′ = w × (Wccd / wp) (4)
In the above formula (4), Wccd is the size of the photographing element (CCD) in the camera 11 (unit: meter), and wp is the horizontal size (unit: pixel) of the image data ID.

  Through the above processing, each position (X, Y, Z) in the horizontal direction, vertical direction, and depth direction of the real scale object having the origin at the lower left position of the image data ID is obtained. FIG. 6 shows an example of the actual scale object positions of the objects A, B, and C shown in FIG.

  The comparison processing unit 24 requests the second image evaluation unit 25 to extract an object. The second image evaluation unit 25 extracts an object from the image data in the database 26 using the same algorithm as the first image evaluation unit 23 and returns object information to the comparison processing unit 24. The comparison processing unit 24 performs image data matching processing using the object information extracted by the first image evaluation unit 23 and the second image evaluation unit 25.

  FIG. 7 is a flowchart for explaining the matching process in the comparison processing unit 24. In the following description, it is assumed that the image data evaluated by the first image evaluation unit 23 is a captured image, and the image data evaluated by the second image evaluation unit 25 is a map image.

  When the matching process starts (S10), the comparison processing unit 24 first selects one object (1) in the captured image (S11). Next, the comparison processing unit 24 selects another object (2) from the captured image (S12), and sequentially selects the map image (3) from the database 26 (S13).

  Next, the comparison processing unit 24 selects the object (4) from the selected map image (3) using the second image evaluation unit 25 (S14), and the type of the object (1) and the object (4) ) Is compared (S15). When the types are the same, the comparison processing unit 24 extracts another object (5) from the map image (3) (S16), and compares the type of the object (2) with the type of the object (5) ( S17). When the types are the same, the comparison processing unit 24 calculates the distance L1 between the objects (1) and (2) and the distance L2 between the objects (4) and (5). The calculation formula of the distance L1 is shown below.

L1 = √ {(X2−X1) ² + (Y2−Y1) ² + (Z2−Z1) ² } (5)
In the above formula (5), (X1, Y1, Z1) is the position of the object (1), and (X2, Y2, Z2) is the position of the object (2). The formula for calculating the distance L2 is the same as above.

  Next, the comparison processing unit 24 compares the distance L1 and the distance L2 (S18). If the distance is the same, the captured image and the selected map image (3) have a correlation (similarly). The evaluation value managed in map image units is increased (S19). Here, the evaluation value managed in map image units is initialized at the start of the comparison process.

  Note that the distance comparison method is not necessarily limited to the case where the distances are the same, and a slight difference in distance considered as an error on the map may be regarded as the same. As an example of the slight distance, the actual distance per pixel may be an error by obtaining the actual distance per pixel from the actual width of the object and the number of pixels of the object width on the captured image. .

  Thereafter, the comparison processing unit 24 confirms whether another map image (3) can be selected in order to compare with another map image (S22). If another map image (3) exists, the comparison processing unit 24 returns to the selection process (S13), and if not, checks whether another object (2) can be selected from the captured image (S23). ). If another object (2) exists, the comparison processing unit 24 returns to the selection process (S12), and if not, checks whether another object (1) can be selected from the captured image (S24). The comparison processing unit 24 returns to the selection process (S11) if another object (1) exists, and ends the matching process if it does not exist (S25).

  If the result of the comparison process (S17) between the type of the object (2) and the type of the object (5) is different from the comparison process (S18) of the distance between objects, the comparison processing unit 24 displays the map image. It is confirmed whether another object (5) can be selected from (3) (S20). The comparison processing unit 24 selects the object (5) again if another object (5) can be selected (S16). If the other object (5) cannot be selected, the comparison processing unit 24 selects another object (4) from the map image (3). It is confirmed whether selection is possible (S21). If the object (4) can be selected, the comparison processing unit 24 extracts the object (4) again (S14). If the object (4) does not exist, the comparison processing unit 24 confirms the selection of another map image (3) (S22). Return to). If the type is different in the comparison process (S15) between the type of the object (1) and the type of the object (4), the comparison processing unit 24 returns to the confirmation process (S21) of the selection of the object (4).

  In the flowchart of FIG. 7, in the object selection process from the captured image and the object selection process from the map image, the selection may fail due to insufficient number of objects, but this is a general abnormality process. Therefore, the description is omitted.

  After the above matching processing is completed, the comparison processing unit 24 compares the evaluation values aggregated for each map image data, and the map that is considered that the map image data having the largest evaluation value has taken a point that approximates the captured image. It identifies that it is image data, and identifies the position of the image photographing terminal 10 based on the photographing position information managed as a pair with the map image data.

  Next, a calculation formula for specifying the position of the image capturing terminal 10 is shown below. Note that the lower left position of the camera 11 portion of the image capturing terminal 10 is expressed as the position of the image capturing terminal 10 unless otherwise specified.

  First, the position (EyeX, EyeY, EyeZ) of the image capturing terminal 10 on the coordinate system having the origin at the lower left of the image in the database 26 is obtained by Expression (6). In the equation (6), (X1, Y1, Z1), (X2, Y2, Z2), (X3, Y3, Z3) are the positions of the objects with the lower left position of the image data ID as the origin, (AX1, AY1, AZ1), (AX2, AY2, AZ2), and (AX3, AY3, AZ3) are object positions when the lower left of the image in the database 26 is set as the origin. Further, (SX, SY, SZ) is the horizontal tilt of the image capturing terminal 10, (UX, UY, UZ) is the vertical tilt of the image capturing terminal 10, and (DX, DY, DZ) is the image. This is the inclination in the line-of-sight direction of the photographing terminal 10. Wp and Hp are the horizontal and vertical sizes of the actual scale image data ID, and are calculated by the following equations.

Wp = wp × (W / w) (7)
Hp = hp × (H / h) (8)
In the above formulas (7) and (8), W, w, H, and h are the same as in formulas (1) and (2). wp and hp are the horizontal and vertical sizes (unit: pixels) of the image data ID.

  Next, the comparison processing unit 24 obtains the absolute position of the image capturing terminal 10 using the following formula. In equations (9) and (10), latitude and longitude calculation formulas are shown as examples of absolute positions.

Latitude = A0_Z + {(EyeZ × cos α) / 111111} (9)
Longitude = A0_X + {(EyeX × cos α) / 111111} (10)
In equations (9) and (10), A0_X and A0_Z are the longitude and latitude of the lower left absolute position of the image in the database 26, and α is the orientation (north) of the image capturing terminal when the image in the database 26 is captured. It is a value indicating how many times the direction is deviated clockwise from the direction, and the deviation in the altitude direction is ignored), and “111111” is a constant and is a meter amount per latitude / longitude of 1 °.

  The comparison processing unit 24 sends the position of the image capturing terminal 10 to the information transmitting unit 27, and the information transmitting unit 27 notifies the information receiving unit 14 of the image capturing terminal 10 of the position information. The display processing unit 15 of the image capturing terminal 10 receives the position information from the information receiving unit 14 and displays the position information using the LCD 60.

  Further, the comparison processing unit 24 determines the similarity between the captured image captured by the image capturing terminal 10 and the captured image held on the database 26, and if the similarity is recognized, the position information The inclination of the image capturing terminal 10 is calculated using the same method as in the above case, and inclination information representing the inclination is transmitted to the image capturing terminal 10 using the information transmitting unit 27. In this case, it is possible to display information indicating the inclination on the LCD 60 of the image capturing terminal 10 without using a sensor for detecting the inclination, or to use the inclination information for other controls.

  In addition, when the comparison processing unit 24 of the server 20 has all the evaluation values managed for each map image data because the number of objects of the photographed image is insufficient, the map image that matches in the database 26. In order to indicate that the data was not found, the server 20 sends an object request notification to the image capturing terminal 10.

  Upon receiving the request, the image capturing terminal 10 notifies the user of the object extraction result based on the object position (horizontal direction and vertical direction) and the object size (horizontal direction and vertical direction). FIG. 8 is an example of an object extraction result notification screen when an object addition request is made to the image capturing terminal 10. In the example of FIG. 8, frame lines F1 and F2 are displayed as examples of guidelines for the objects O1 and O2, respectively, and a message M1 indicating how many objects are required is displayed. In addition, a guideline is not specifically limited to said broken line, A solid line, a 1-dot chain line, a dashed-two dotted line, etc. may be used and a various change is possible.

  The required number of objects indicated by the message M1 is a value depending on the comparison processing unit 24. For example, if there is no candidate image, request one object, and if there are multiple candidate images as described later and there are two unevaluated objects in the candidate image, request two. There is a method for determining such values. However, the value determination method is not limited to these examples, and it may be a method of requesting one by one or a method of using statistical information on a map. Also, any method other than the method shown here may be used as long as the user can know the object extraction result.

  Thus, even when the candidate image cannot be specified, the user can take measures such as increasing the number of objects in the captured image. When the matching processing of the captured image is performed again, the comparison processing unit 24 can be used. The number of object comparison patterns can be increased, and the possibility of matching with map image data can be increased.

  Further, when there are a plurality of map image data with the same evaluation value managed for each map image data in the comparison processing unit 24 of the server 20 because there are not enough objects in the photographed image, the same applies. By notifying the image capturing terminal 10 from the server 20 to increase the number of objects, the number of comparisons by objects can be increased, and map image data closer to the captured image can be found.

  In addition, the comparison processing unit 24 has a function of replacing the corresponding map image with the corresponding captured image by regarding the closest evaluation value between the captured image and the map image as the same point. At this time, the comparison processing unit 24 may directly rewrite the information shown in FIG. 2 held in the database 26, or notifies the image photographing terminal 10 from the server 20 and displays a confirmation display as shown in FIG. M2 may be performed, and the database 26 may be rewritten according to a user instruction.

  Further, when there are a plurality of map images having the same evaluation value as the photographed image, the server 20 notifies the image photographing terminal 10 of the candidate location in the same manner, and the replacement candidate location is selected by the user as shown in FIG. A message M3 to be presented is displayed, and the user is made to select a rewriting point. By replacing the map image with a captured image, the map image can always be kept up-to-date.

  As described above, in the present embodiment, the database 26 can be created from only the photographed image and its position information, and the position information can be specified from the photographed image. Since the similarity of images can be evaluated without being affected by disturbances such as moving objects such as plants or the reflection of automobiles, or by deviations in the captured images, the location information of the image capturing terminals can be obtained. It can be obtained with high accuracy.

(Second Embodiment)
Next, an embodiment in which an object is extracted and held in advance for each map image data in the server will be described as a second embodiment. In the second embodiment, when registering map image data in the database, the administrator of the image search system executes object extraction at the server and registers the object extraction result in advance in units of map image data. .

  FIG. 11 shows a block diagram of a server 20a to which a function for registering map image data and objects in the database 26 is added in the image search system according to the second embodiment of the present invention. The image capturing terminal 10 that communicates with the server 20a is the same as that in the first embodiment, and thus illustration and detailed description thereof are omitted.

  The image input unit 32 receives input of map image data, shooting position information, and shooting direction by an administrator of the image search system, and transmits the map image data, shooting position information, and shooting direction to the image registration unit 31. The image registration unit 31 requests the comparison processing unit 24 to evaluate the map image. The comparison processing unit 24 requests the first image evaluation unit 23 to analyze the map image. The first image evaluation unit 23 extracts objects using the object database 28. The object extraction method is the same as the method described in the first embodiment. The image evaluation unit that evaluates the map image is not limited to the first image evaluation unit 23, and the map image data is analyzed using the second image evaluation unit 25 to extract the object and its attribute information. May be performed.

  When the first image evaluation unit 23 finishes extracting the object, the first image evaluation unit 23 notifies the image registration unit 31 of the object information via the comparison processing unit 24, and the image registration unit 31 includes the map image data, the shooting position information, and the shooting The object information is registered in the database 26 together with the direction.

  FIG. 12 shows an example of a data structure managed by the database 26. As shown in FIG. 12, the database 26 stores the shooting position information (latitude and longitude) and the shooting direction for each map image data, and the type and position (latitude, longitude and altitude) of the objects included in the map image. ) Is remembered.

  Thus, in this embodiment, since the object has been extracted for each map image data in the database 26, the second image evaluation unit 25 does not need to analyze the map image data and extract the object. . Therefore, the comparison processing unit 24 can directly read the object information included in the map image data from the database 26. Other processes are the same as those in the first embodiment.

  Note that, after specifying the position information of the captured image captured by the image capturing terminal 10, the attribute information of the object extracted from the captured image may be registered as map image data in the database 26 together with the captured position information.

  Further, the database 26 may store in advance a result of extracting a plurality of objects existing in the held map image and calculating the distance between the objects in association with the map image. . In this case, by extracting the map image object in the database 26 in advance, it is possible to reduce the process of extracting the object each time the similarity with the photographed image is evaluated.

(Third embodiment)
Next, an embodiment in which data is stored in units of objects together with position information on a map at a server will be described as a third embodiment. In the third embodiment, when registering the map image data in the database, the administrator of the image search system executes object extraction and position information identification at the server, together with the position information for each object. The attribute information is registered in advance.

  Since the functional block diagram of the image search system according to the present embodiment is the same as that in FIG. 11, the overlapping illustration is omitted. Therefore, the description will be omitted with reference to FIG. .

  The difference between the present embodiment and the second embodiment is that the first image evaluation unit 23 calculates the position information for each object together with the extraction of the object, and notifies the image registration unit 31 of the difference. The image registration unit 31 notified of the object information and its position information is to register the object information and the position information in the database 26. Details will be described later.

  FIG. 13 shows an example of a data structure managed by the database 26. In the present embodiment, the data in the database 26 is map object data, and the entity is attribute information and position information in units of objects. Therefore, the second image evaluation unit 25 does not need to analyze the map image data and extract an object, and the comparison processing unit 24 can directly read the object information included in the map image data from the database 26.

  FIG. 14 is a flowchart for explaining matching processing in the comparison processing unit 24 of the image search system according to this embodiment.

  When the matching process starts (S30), the comparison processing unit 24 first selects one first object (1) from the photographed image (S31), and then another second second object from the photographed image. The object (2) is selected (S32). The comparison processing unit 24 further selects one third object (3) from the database 26 (S33), and compares the type of the object (1) with the type of the object (3) (S34).

  If the types are the same, the comparison processing unit 24 extracts the fourth object (4) different from the third object (3) from the database 26 (S35), and the type and object (2) of the object (2) Comparison with the type of 4) is performed (S36). When the types are the same, the comparison processing unit 24 calculates the distance L1 between the objects (1) and (2) and the distance L2 between the objects (3) and (4). The formula for calculating the distance is the same as that in the first embodiment.

  Next, the comparison processing unit 24 compares the distance L1 and the distance L2 (S37). If the distances are the same, the objects (1) and (2) in the photographed image and the object (3) being selected are selected. , (4) is determined to have a correlation, and the evaluation value managed in units of objects in the database 26 is increased (S38). Note that the evaluation value managed in units of objects in the database 26 is initialized at the start of the comparison process.

  After that, the comparison processing unit 24 compares with another object in the database 26, and confirms whether another fourth object (4) can be selected (S39). If another object (4) exists, the comparison processing unit 24 returns to the selection process (S35), and if not, checks whether another object (3) can be selected from the database 26 (S40). ). If another object (3) exists, the comparison processing unit 24 returns to the selection process (S33), and if not, checks whether another object (2) can be selected from the photographed image (S41). . If another object (2) exists, the comparison processing unit 24 returns to the selection process (S32), and if not, checks whether another object (1) can be selected from the captured image (S42). The comparison processing unit 24 returns to the selection process (S31) if another object (1) exists, and ends the matching process if it does not exist (S43).

  If the result of the comparison process (S36) between the type of the object (2) and the type of the object (4) is different from the comparison process of the distance between objects (S37), the comparison processing unit 24 26, it is confirmed whether another object (4) can be selected (S39). If another object (4) can be selected, the comparison processing unit 24 selects the object (4) again (S35). If the other object (4) cannot be selected, the comparison processing unit 24 can select another object (3) from the database 26. (S40). If the object (3) can be selected, the comparison processing unit 24 extracts the object (4) again (S33). If the object (3) does not exist, the comparison processing unit 24 performs confirmation processing for selecting the object (2) from the photographed image. Perform (S41). Further, when the types are different in the comparison process (S34) between the type of the object (1) and the type of the object (3) (S34), the comparison processing unit 24 performs a confirmation process for selecting the object (3) (S40). .

  In the flowchart of FIG. 14, in the object selection process from the captured image and the object selection process from the database, the selection may fail due to an insufficient number of objects, but this is a general abnormal process. The description is omitted. Further, after specifying the position information of the photographed image photographed by the image photographing terminal 10, the attribute information of the object extracted from the photographed image may be registered as map object data in the database 26 together with the photographing position information.

  After the above matching processing is completed, the comparison processing unit 24 compares the evaluation values collected for each object in the database 26, and calculates the distance between the object having the largest evaluation value and the object (photographed image). And another object in the database 26 that matches the object in the image) are both included in the photographed image, and the photographing position in the photographed image is identified using position information managed in pairs with the object. To do.

  Next, a calculation formula for specifying the position of the image capturing terminal 10 is shown below.

  Here, the position (EyeX, EyeY, EyeZ) of the image capturing terminal 10 calculated by Expression (11) is a position in the coordinate system in the database 26. In the equation (11), (AX1, AY1, AZ1), (AX2, AY2, AZ2), and (AX3, AY3, AZ3) are object positions in the database 26. Further, in the matrix F, fv = 1.0 / tan (π × fovy / 180.0), F is a perspective correction matrix, and is a camera-specific parameter of the image capturing terminal 10, fovy (vertical direction) Is a fixed value obtained from aspect (horizontal viewing angle), aspect (horizontal viewing angle), and f, n (focal length). Other symbols in Equation (11) are the same as in Equation (6).

  The procedure for transmitting the shooting position to the image shooting terminal 10 after specifying the shooting position of the image shooting terminal 10 by the above algorithm is the same as the procedure described in the first embodiment.

  Next, with respect to the database 26 in FIG. 11, an object on an image captured by the image capturing terminal 10 or an object on map image data registered by the image input unit 32 (hereinafter referred to as a registration target object). A method for updating an object to be registered by adding or deleting will be described. Note that the processing request for registering an object may be a processing request from the image capturing terminal 10 as described in the first embodiment, or a process request from the image capturing terminal 10 for an unregistered object. Even if there is no request, it may be automatically registered in the server 20a.

  As an addition method of the registration target object, after the object extraction by the first image evaluation unit 23, the comparison processing unit 24 and the second image evaluation unit 25 compare the extracted object with the object in the database 26, If the object extracted by the first image evaluation unit 23 does not exist in the database 26, it is added to the database 26 as a registration target object. If the object and position information exist in the database 26 but no object exists on the captured image, the object in the database 26 is deleted. If the type of the object in the database 26 existing at the same position as the object on the captured image is different, the information on the object in the database 26 is overwritten with the information on the object on the captured image.

  Next, details of a procedure for registering an object to be registered in the database 26 will be described. An expression for converting the position of the registration target object from the value in the coordinate system with the origin at the lower left position of the captured image to the value in the coordinate system in the database 26 is shown below.

  Here, in Expression (13), (AX ′, AY ′, AZ ′) is the position of the registration target object after conversion (value in the coordinate system in the database 26). Also, (X, Y, Z) is the position of the registration target object before conversion (value in the coordinate system with the lower left position of the captured image as the origin). Other symbols are the same as those in Expression (11). With the above procedure, the object position information can be registered in the database 26 together with the object.

  As described above, in the present embodiment, since the data on the database 26 is held in units of objects, it is possible to compare objects included in different images. When comparing the similarity with the object in 26, the probability of matching can be increased, and the frequency with which the position information can be specified can be increased.

(Fourth embodiment)
Next, a third image evaluation unit and an object database equivalent to the first image evaluation unit 23 and the object database 28 in FIG. 3 are provided on the image shooting terminal side, and the object extraction result is displayed on the preview image being shot. An embodiment for displaying (for example, displaying a frame line around an object) will be described as a fourth embodiment. In this embodiment, since the object extraction result is known to the user at the time of shooting, the case where selection fails due to an insufficient number of objects in the image data matching process (flow chart in FIG. 7) on the server 20 is reduced as much as possible. It becomes possible.

  FIG. 15 is a functional block diagram of an image capturing terminal and a server of the image search system according to the fourth embodiment. The processing in the server according to the present embodiment is the same as that in the first embodiment, and thus detailed description thereof will be omitted, and the overall flow of the current position specifying process by the image search system will be described with reference to FIG. Among these, object extraction processing by the image capturing terminal will be mainly described.

  The camera image capturing unit 12 of the image capturing terminal 10 a captures the scenery being captured by the camera 11 as image data and stores it in the captured image recording unit 16. The third image evaluation unit 18 obtains object information to be passed to the display processing unit 15 from the object information stored in the captured image recording unit 16 and the object database 17. The method for obtaining the object position (horizontal direction and vertical direction) and the object size (horizontal direction and vertical direction) in pixel units from the image data of the photographed image recording unit 16 and the object database 17 is the same as in the first embodiment. is there.

  The object information required by the third image evaluation unit 18 is the position of the object (horizontal direction and vertical direction) with the lower left position of the image capturing terminal 10a as the origin. Here, since the obtained object information is only used in the image capturing terminal 10a, it is not necessary to obtain the position of the object on an actual scale, and may be in units of pixels. Further, since the purpose is to display a frame line or the like on the LCD 60, the position of the object (in the depth direction) is also unnecessary.

  The display processing unit 15 notifies the user of the object extraction result by displaying a frame line around the object on the LCD 60 based on the object information. For example, as shown in FIG. 8, the display processing unit 15 performs object extraction based on the object position (horizontal direction and vertical direction) and the object size (horizontal direction and vertical direction) obtained by the third image evaluation unit 18. Notify the user of the result. The notification method is the same as in the first embodiment.

  While checking the object extraction result on the LCD 60, the user determines the shooting range by moving the image shooting terminal 10a up and down, left and right so that the number of objects in the preview image increases.

  FIG. 16 shows an example in which the object O3 is newly recognized and the number of objects in the image reaches the required number of objects when the user moves the image capturing terminal 10a to the right from the state of FIG. A message M4 informs the user that shooting is allowed. The user sees the message M4 and presses the shooting button. Alternatively, the shooting button may be automatically pressed when the number of objects in the image reaches the required number of objects.

  When the shooting button is pressed, the camera image capturing unit 12 captures the scenery being captured by the camera 11 as image data and stores it in the captured image recording unit 16, and the captured image recording unit 16 passes through the camera image transmission unit 13. The image data is transmitted to the server 20. Processing after transmission to the server 20 is the same as in the first embodiment.

  According to the present embodiment, since the object extraction result is known to the user at the time of preview, there is a case where the selection fails due to an insufficient number of objects in the image data matching process (the flowchart in FIG. 7) on the server 20. It becomes possible to reduce as much as possible. As a result, it is possible to prevent the user from re-acquiring the camera image many times, and the time required for specifying the position can be reduced accordingly. In addition, since the number of times of image transmission to the server 20 can be reduced, it is possible to reduce communication costs for image transmission.

  According to the present invention, it is difficult to be affected by the expansion, reconstruction, removal or new construction of buildings, the influence of disturbances such as moving objects such as people and plants, or the reflection of automobiles, and the displacement of captured images. This is useful as a method for evaluating the similarity of images. In addition, by utilizing the fact that the similarity of images can be evaluated with high accuracy, it can be applied to the purpose of comparing and evaluating the similarity of images to obtain the position information of the image capturing terminal. There is no need to provide a location specifying means, and its industrial utility value is extremely high.

It is a block diagram which shows the structure of the image search system by the 1st Embodiment of this invention. It is a figure which shows an example of the data structure in the database which manages image data and imaging | photography position information of image data by a pair. FIG. 2 is a functional block diagram of an image capturing terminal and a server shown in FIG. 1. It is a figure which shows an example of the data structure of the object database shown in FIG. It is a figure which shows the example which extracted three objects shown in FIG. 4, and calculated | required these positions and magnitude | sizes. It is a figure which shows the example of the object position of the real scale shown in FIG. It is a flowchart for demonstrating the matching process in the comparison process part shown in FIG. It is a figure which shows an example of the notification screen of the object extraction result in the case of performing the addition request of an object to the image photographing terminal shown in FIG. It is a figure which shows an example of the confirmation screen in the case of replacing a map image. It is a figure which shows an example of the inquiry screen in the case of selecting a map image and replacing with a picked-up image. It is a functional block diagram of the server of the image search system by the 2nd Embodiment of this invention. It is a figure which shows an example of the data structure managed in a database, when managing the attribute information of the extracted object for every map image. It is a figure which shows an example of the data structure managed in a database, when managing attribute information for every object. It is a flowchart for demonstrating the matching process in the comparison process part of the image search system by the 3rd Embodiment of this invention. It is a functional block diagram of the image photographing terminal and server of the image search system by the 4th Embodiment of this invention. It is a figure which shows an example of the notification screen of the object extraction result when the number of objects in an image reaches the required number of objects.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10, 10a Image photographing terminal 11 Camera 12 Image capture part 13 Camera image transmission part 14 Information receiving part 15 Display processing part 16 Captured image recording part 17 Object database 18 3rd image evaluation part 20, 20a Server 21 Image receiving part 22 Image Data temporary recording unit 23 First image evaluation unit 24 Comparison processing unit 25 Second image evaluation unit 26, 30 Database 27 Information transmission unit 28 Object database 29 Candidate image determination unit 31 Image registration unit 32 Image input unit 60 LCD

Claims (9)

An image search system that presents candidates for the current position using captured images,
An image capturing terminal that acquires a captured image of a landscape;
Judge the similarity between the captured image captured by the image capturing terminal and the captured image stored in the database, and identify the captured image stored in the database where the similarity is recognized as a candidate image And a candidate image determination unit that returns position information stored in the database in association with the identified candidate image to the image capturing terminal. The candidate image determination unit
A first image evaluation unit that extracts a plurality of comparison target objects from a captured image captured by the image capturing terminal and calculates a distance between the comparison target objects;
A second image evaluation unit that extracts a plurality of comparison target objects from a captured image held on the database and calculates a distance between the comparison target objects;
A captured image captured by the image capturing terminal based on the distance between the comparison target objects calculated by the first image evaluation unit and the distance between the comparison target objects calculated by the second image evaluation unit. The image search system according to claim 1, further comprising: a comparison processing unit that determines similarity between the captured image and the captured image held on the database.   4. The image search system according to claim 2, wherein the image photographing terminal draws a guideline around the comparison target object when the comparison target object exists on the preview image. The comparison processing unit notifies the image photographing terminal to increase the number of comparison target objects on the preview image when there are a plurality of candidate images similar to the photographed image photographed by the image photographing terminal. Done
The image retrieval system according to claim 3, wherein the image capturing terminal presents notification contents to the user when receiving the notification.   The database stores, in advance, a result of calculating a distance between comparison target objects by extracting a plurality of comparison target objects existing in the held captured image in association with the captured image. The image search system according to any one of claims 2 to 4.   The image search system according to claim 2, wherein the database holds position information for each comparison target object.   When the similarity between the captured image captured by the image capturing terminal and the captured image stored on the database is high, the candidate image determination unit selects the captured image stored on the database. The image search system according to claim 1, wherein the image search system is replaced with a captured image captured by an image capturing terminal.   The image search system according to claim 1, wherein the candidate image determination unit adds a new comparison target object to the database.   The candidate image determination unit determines the similarity between the captured image captured by the image capturing terminal and the captured image stored in the database, and when the similarity is recognized, the image capturing terminal The image search system according to claim 1, wherein an inclination of the image is calculated and returned to the image capturing terminal.

Images