JP2010272054A - Device, method, and program for providing building relevant information - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a device for providing building relevant information, which allows recognition hardly affected by noise like a shadow on a signboard. <P>SOLUTION: An image photographed by an image acquisition means 102 of a mobile terminal 30 with a camera is transmitted to an information providing server 500 via a network 101 by an information provision request transmission means 103, and an image of a character string area is extracted from the image which is received by an image reception means 111, by a character string area extraction means 116, and a feature quantity of the extracted image is extracted by an image matching degree calculation means 112, and matching degrees between the feature quantity and feature quantities of images of partial areas of respective buildings stored in a building relevant information database 215 are calculated, and the matching degrees are defined as evaluation values of the buildings by an evaluation means 113, and relevant information of a building having high evaluation values is transmitted to the mobile terminal 30 with the camera by a provision information transmission means 114. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a system that provides information on an external building in real time based on information on an image of an external building, and provides information on a store and map information around the store to provide a current operator. In particular, the present invention relates to a building-related information providing apparatus, method, and program for handling an image of a signboard of a building.

  Conventionally, an object image captured by a camera-equipped mobile phone is used as a target and collated with the feature amount of the recognition target image stored in an existing database, and related information is displayed based on the result or the location of the information is determined. Systems / services to present have been proposed. These include, for example, a service such as “Mirara” described in Non-Patent Document 1 and a service such as “syncR” described in Non-Patent Document 2. There are various methods for collating feature quantities between images, and a technique for recognizing an image of a signboard of a building has been proposed (see Non-Patent Document 3).

  The general operation of such a system will be described with reference to the drawings. FIG. 6 is a block diagram of a prior art system. The system of FIG. 6 includes a camera-equipped mobile terminal 30 and an information providing server 50, and information is exchanged between them via a network 101.

  Each record of the building related information database 115 of the information providing server 50 needs to store an image feature amount and related information of each target building in advance. The image feature amount of the building is a feature amount extracted from the image of the specific area of the building for specifying the building, and the specific area is the area of the signboard of the building in this example. FIG. 7 shows an example of the contents of the building related information database 115. Among them, FIG. 8 shows a schematic diagram of the appearance of the record having the building number 1 and “Kanagawa No. 9 Building” corresponds to the central building 20.

  One billboard 21 is installed in this building 20, and what is registered is a feature amount extracted from the image of FIG. 9 taken in advance of the billboard 21 of “Yokosuka Shokai”. The feature quantity is generally represented by a vector as shown in FIG. 7, and this is called a feature vector. In addition to this, related information of each building is recorded in the building related information database 115. In this example, a URL representing an address where a peripheral map centering on the building is placed on the Web is recorded. .

  Next, a flow of processing for actually acquiring and displaying related information using the camera-equipped mobile terminal 30 will be described. The operator photographs the subject 10 using the image acquisition unit 102 of the camera-equipped mobile terminal 30. An example of the image is shown in FIG. Next, the camera-equipped mobile terminal 30 sends an image captured using the information provision request transmission unit 103 to the image reception unit 111 in the information provision server 50 via the network 101. An image feature amount is then extracted from the image sent to the image receiving unit 111 by the image matching degree calculating unit 112, and the feature amount is collated with the image feature amount of each record in the building related information database 115.

  The matching degree of all records is sent to the evaluation means 113, the record with the highest matching degree is selected there, and the information of the record is sent to the provision information sending means 114, where the corresponding information in the building related information database 115 The provided information transmitting unit 114 searches for the related information (the URL of the map in this example) of the record to be transmitted, and sends it to the information receiving unit 104 of the camera-equipped mobile terminal 30 via the network 101. Thereafter, this URL is sent to the information display means 105. The information display means 105 accesses the Web with this URL and displays the obtained map image (example is shown in FIG. 11).

  In addition, various services that use the GPS function installed in the mobile phone to navigate the current position, the route to the destination, etc. in conjunction with the map information have been implemented. These services include GPS / location information service and EZ Navi Walk, etc. The former displays the current location on the surrounding map based on the location information acquired by the GPS function built in the mobile phone, and reaches the destination. It is possible to provide a route and present store information around the route. On the other hand, the latter is a service that is equipped with a compass in addition to the GPS function and displays the current position and the traveling direction on the liquid crystal monitor.

  A method for extracting a character string region used in the present invention is described in Non-Patent Document 4, for example.

“Feature amount extraction technology (Mira)”, search on May 1, 2009, Internet URL <http: // capg. sbins. co. jp / products / case / milula / index. html> “SyncR by Olympus-Cross Media Realization”, May 1, 2009 search, Internet URL <http: // www. syncr. jp / outline4. html> Takayasu Yamaguchi, Minoru Takahata, Nobuyuki Hongo, “Information Retrieval Triggered by Signboard Shooting with Mobile Camera (Pattern Recognition / Media Understanding for Entertainment)”, IEICE Technical Report, IEICE Technology Research Report (PRMU), December 11, 2003, Vol. 103, no. 515 pp. 19-24 “Text Locating and Verification Algorithm for Scene Images Based on Gray-Scale Connected Component Analysis”, Image Electronics Society of Japan, July 2007, Vol. 36, no. 4, pp509-519

  The conventional method described above has the following problems.

  In GPS / location information service and EZ Navi Walk, etc., it is assumed that the GPS functions normally. However, GPS is not guaranteed to function normally in any location. For example, in the vicinity of tall buildings. It is known that position information may not be measured.

  In addition, in the service realized by “Mira” of Non-Patent Document 1 and “SyncR” of Non-Patent Document 2, the feature quantity of the image sent from the camera-equipped mobile terminal is stored in the database. This is a form in which the recognition target object having the highest degree of matching is selected by collating with the feature amount of the image of the object, and the information on the database linked to it is sent to the terminal. In such a system, the accuracy of matching between images becomes a problem. A certain percentage of errors cannot be avoided regardless of the matching technique between images. In this case, the mobile terminal is notified of recognition failure or related information on the recognition object that is different from the correct answer. Will be provided. As a cause of such recognition failure, camera shake and blur at the time of shooting with a mobile terminal, if the position and size of the object in the image shot with the mobile terminal greatly deviate from the expected range, There are various factors, such as a shadow on an object.

  FIG. 10B is an example, and this is an image in which a shadow of another building is cast on the lower part of the signboard when photographed with a portable terminal to request information provision, and the building related information in FIG. Compared with the image of FIG. 9 used for registering the image feature amount in the database 115, this thin shadow is greatly different. In such a case, for example, the image feature amount is likely to be closer to another category (for example, “DFF Building”) than “Kanagawa Building No. 9” in FIG. 7 which is the correct category. In the processing result of the evaluation means 113, the first candidate is “DFF building”, the second candidate is “Kanagawa ninth building”, and the information of “DFF building” is erroneously sent to the camera-equipped mobile terminal 30. It will be.

  This invention solves the said subject, The objective is to provide the building related information provision apparatus, method, and program which can perform the recognition which is hard to receive to the influence of noise, such as the shadow applied to the signboard. is there.

  The building related information providing device of the present invention for solving the above problems is a building related information providing device that provides information related to the building from the image of the building taken by the mobile terminal with camera. Then, for each set of buildings existing in the outside world, the feature amount of the partial region image obtained by extracting the character string region from the image of the specific region of the building and the related information of the corresponding building are stored in association with each other. A character string area image for extracting an image of a character string area from an image received by the image receiving means, an image receiving means for receiving an image transmitted from the camera-equipped mobile terminal, and an image received by the image receiving means Extracting means and feature amount of the image extracted by the character string region image extracting means, and extracting the feature amount and the partial area of each building stored in the building related information database Image matching degree calculating means for calculating the degree of matching with the feature amount of the image, evaluation means for using the degree of matching calculated by the image matching degree calculating means as an evaluation value of each building, and relative to the evaluation means Providing information transmitting means for transmitting related information of a building to which a high evaluation value is given to the mobile terminal with camera.

  According to the present invention, when collating images of a signboard of a building, the object of collation between images is limited to a character string area in the signboard, thereby being affected by noise such as a shadow on the signboard. It becomes possible to perform difficult recognition, and it is possible to provide a service with few errors in providing information.

It is a block diagram which shows the structure of the building related information provision system by the example of 1 embodiment of this invention. It is explanatory drawing which shows the example of a process of the image for calculating the feature-value of the building in the building related information database of one embodiment of this invention. It is explanatory drawing which shows the structural example of the building related information database in the example of 1 embodiment of this invention. It is a flowchart which shows the flow of a process of the building related information provision method by one embodiment of this invention. It is explanatory drawing which showed a mode that the area | region of the character string was extracted and cut out by applying this invention with respect to the image of the signboard of the building image | photographed in order to obtain the relevant information of a building. It is a block diagram which shows the conventional system configuration. It is explanatory drawing which shows the structural example of the building related information database in the conventional system of FIG. It is a schematic diagram which shows the example of the to-be-photographed object. It is explanatory drawing which shows the example of the image which image | photographed the subject's signboard. It is explanatory drawing which shows the example of the image which image | photographed the subject's signboard. It is explanatory drawing of the map of the building which is an example of the relevant information of a building.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited to the following embodiments. FIG. 1 is a block diagram showing the configuration of an embodiment of a building related information providing system to which the present invention is applied. 1, the same parts as those in FIG. 6 are denoted by the same reference numerals.

  The system according to the present embodiment includes a camera-equipped mobile terminal 30 and an information providing server 500, and information is exchanged between them via the network 101.

  The camera-equipped mobile terminal 30 transmits an image acquisition unit 102 that captures a specific area (for example, a signboard) of the building of the subject 10 with the camera, and transmits the captured image to the information providing server 500 via the network 101. Information provision request transmitting means 103 for requesting provision of information related to the building, and information reception for receiving information related to the building sent from the information providing server 500 via the network 101 as a result of the request. Means 104 and information display means 105 for displaying the received information.

  For each set of buildings existing in the outside world, the information providing server 500 extracts a character string area from an image of a specific area (for example, a signboard) of the building and characterizes the image of the partial area and the corresponding building. The building related information database 215 that stores the related information in association with each other, the image receiving unit 111 that receives the image transmitted from the camera-equipped mobile terminal 30 via the network 101, and the image receiving unit 111. A character string region image extracting unit 116 that extracts an image of a character string region from the image, and a feature amount of the image extracted by the character string region image extracting unit 116 are extracted. The image matching degree calculating means 112 for calculating the degree of matching with the feature amount of the image of the partial area of each building stored in the information database 215; An evaluation unit 113 that uses the degree of match calculated by the image match level calculation unit 112 as an evaluation value of each building, and related information of the building to which a relatively high evaluation value is given by the evaluation unit 113 Provided information transmitting means 114 for transmitting to the mobile terminal 30 with camera.

  The functions of the respective means of the information providing server 500 are achieved by, for example, a computer.

  The building related information database 215 of the information providing server 500 corresponds to the building related information database 115 described with reference to FIG. The differences are as follows.

  When the feature amount of the signboard image for each building is created, in the case of the building related information database 115 of FIG. 6 of the prior art, the feature amount from the entire image obtained by photographing the signboard shown in FIG. On the other hand, in the building related information database 215 of the present embodiment, for example, a character string region image extracting unit 116 from an image obtained by photographing a sign as shown by a dotted line region in FIG. The character string area is extracted by the above, and the image of the portion is cut out as shown in FIG. 2B and the feature amount is calculated from the image, and the image is stored in the building related information database 215 in advance. Others are the same as the building related information database 115. That is, in the building related information database 215, as shown in FIG. 3, related information of each building is recorded in the same manner as the building related information database 115, and in this embodiment, the building is centered. A URL representing an address where the surrounding map is placed on the Web is recorded.

  The building related information stored in the building related information database 215 is text or an image, video, audio, the position of the building in the surrounding map, or a combination thereof.

  Next, the flow of processing for actually acquiring and displaying related information using the camera-equipped mobile terminal 30 will be described with reference to the flowchart shown in FIG. First, the operator photographs the subject 10 using the image acquisition unit 102 of the camera-equipped mobile terminal 30 (step S1). Here, it is assumed that a signboard of “Kanagawa Building No. 9” is photographed and the image of FIG. Next, this image is sent to the information provision request transmission means 103 and sent to the image reception means 111 in the information provision server 500 via the network 101 (step S2).

  The image sent to the image receiving means 111 and received (step S3) is first extracted by the character string area image extracting means 116 so that the character string area image is extracted as a dotted area in FIG. The image of the area is cut out as shown in FIG. 5B (step S4). The shadow of the character string cut out in this way does not include a shadow on the lower part of the image. As a specific method for extracting the character string region, a method that has already been proposed (for example, the method described in Non-Patent Document 4) can be applied.

  Next, the image feature amount of the image of the character string region is extracted by the image matching degree calculation means 112 (step S5), and the feature amount is collated with the image feature amount of each record in the building related information database 215. The degree of coincidence, which is a measure of the degree of similarity between images, is calculated (step S6).

  In this process, in the case of the prior art, the image of FIG. 9 used for calculating the feature amount of the corresponding building in the building related information database 115 and the image of FIG. Since there is a difference between the images due to the shadow existing in the lower part of the image in FIG. 10B, the degree of coincidence of the images is low and a recognition error is likely to occur.

  On the other hand, in the technique of the embodiment, since the image of FIG. 5B cut out by the image matching degree calculation unit 112 does not have a shadow, the corresponding building in the building related information database 215 It is close to the image of FIG. 2B used for calculating the feature amount of the image, and therefore, the degree of coincidence of the image is high and recognition errors are unlikely to occur. In this way, in this embodiment, the image area to be compared by the image matching degree calculation means 112 is narrower than in the case of the prior art, so it can be said that the image is not easily affected by shadows. The degree of coincidence is calculated for all records, and the degree of coincidence for all records is sent to the evaluation means 113.

  The evaluation unit 113 selects the record with the highest degree of match, and sends the result to the provision information transmission unit 114 (step S7).

  As a result of the above processing, the provision information transmitting unit 114 has received the correct result “Kanagawa No. 9 Building” from the evaluation unit 113.

  Then, the provided information transmitting unit 114 searches for the related information (in this example, the URL of the map) of the corresponding record in the building related information database 215, and this information is received by the information receiving unit 104 of the camera-equipped mobile terminal 30 via the network 101. (Step S8).

  Next, the related information is received by the information receiving means 104 (step S9), the URL is sent to the information display means 105, and the information display means 105 accesses the Web with this URL and is obtained in the same manner as in the description of the prior art. The map image displayed is displayed (step S10).

  Note that the feature amount of the image extracted by the image matching degree calculation unit 112 may be stored in a memory and used, for example.

  The related information of the building is text or an image, video, audio, the position of the building in the surrounding map, or a combination thereof.

  In addition, a part or all of the functions of each means in the building related information providing apparatus of the present embodiment can be configured by a computer program, and the program can be executed using the computer to realize the present invention. It goes without saying that the procedure in the building related information providing method of the embodiment can be configured by a computer program, and the program can be executed by the computer, and the computer can read the program for realizing the function by the computer. Recording media such as FD (Floppy (registered trademark) Disk), MO (Magneto-Optical disk), ROM (Read Only Memory), memory card, CD (Compact Disk) -ROM, DVD (Digital V) rsatile Disk) -ROM, CD-R, CD-RW, HDD, and recorded in a removable disk, or stored, it is possible or distribute. It is also possible to provide the above program through a network such as the Internet or electronic mail.

DESCRIPTION OF SYMBOLS 10 ... Subject 20 ... Building 21 ... Signboard 30 ... Portable terminal with camera 101 ... Network 102 ... Image acquisition means 103 ... Information provision request transmission means 104 ... Information reception means 105 ... Information display means 111 ... Image reception means 112 ... Image matching Degree calculation means 113 ... Evaluation means 114 ... Provided information transmission means 115 ... Building related information database in the prior art 116 ... Character string region image extraction means 215 ... Building related information database in the embodiment of the present invention

Claims (4)

A building related information providing device for providing information related to a building from an image of the building taken by a mobile terminal with a camera,
For each set of buildings that exist in the outside world, a building that associates and accumulates feature quantities of partial region images obtained by extracting character string regions from images of specific regions of the building and related information of the corresponding building A related information database;
Image receiving means for receiving an image transmitted from the camera-equipped mobile terminal;
A character string area image extracting means for extracting an image of a character string area from the image received by the image receiving means;
The feature amount of the image extracted by the character string region image extracting means is extracted, and the degree of coincidence between the feature amount and the feature amount of the partial region image of each building stored in the building related information database is calculated. Image matching degree calculating means to perform,
An evaluation unit that uses the degree of matching calculated by the image matching level calculation unit as an evaluation value of each building;
A building-related information providing apparatus, comprising: provision information transmitting means for transmitting related information of a building to which a relatively high evaluation value is given by the evaluation means to the mobile terminal with camera. 2. The building related information providing apparatus according to claim 1, wherein the building related information is text or an image, video, audio, a position of the building in a surrounding map, or a combination thereof. Building related information providing device. For each set of buildings that exist in the outside world, a building that associates and accumulates feature quantities of partial region images obtained by extracting character string regions from images of specific regions of the building and related information of the corresponding building A building-related information providing apparatus having a related information database is a building-related information providing method for providing information related to a building from an image of the building taken by a mobile terminal with a camera,
An image receiving step for receiving an image transmitted from the camera-equipped mobile terminal;
A character string region image extracting means for extracting an image of the character string region from the image received by the image receiving means;
The image matching degree calculating means extracts the feature quantity of the image extracted by the character string area image extracting means, and the feature quantity and the feature of the image of the partial area of each building stored in the building related information database An image matching degree calculating step for calculating the degree of matching with the quantity;
An evaluation step in which the evaluation means uses the degree of matching calculated by the image matching degree calculation means as an evaluation value of each building;
The provided information transmitting means includes a provided information transmitting step of transmitting related information of the building to which the relatively high evaluation value is given by the evaluating means to the mobile terminal with camera. Information provision method. A building related information providing program for causing a computer to function as each means according to claim 1.

Images