JP2008059319A - Object recognition device, and image object positioning device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately recognize an object, even when a superficial difference caused by a situation when photographed and an image noise exists, in images of the same kinds of objects. <P>SOLUTION: A plurality of image feature amounts is extracted respectively as to every of a plurality of template images in a template image storage means, so as to find a correlation coefficient matrix between the extracted image feature amounts, then an input image of a recognition object is extracted out of a frame image acquired by an image acquiring means, a plurality of image feature amounts is extracted as to the input image, as in the template image, so as to find a correlation coefficient matrix between the image feature amounts, the correlation coefficient matrix of the found input image is compared with the correlation coefficient matrixes in the plurality of template images to calculate respectively distances between the input image and the plurality of template images, and the object of the template image having the minimum calculated distance is recognized as the object of the input image, in an image processing means provided in this object recognition device of the present invention. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an object recognition device for recognizing a specific object appearing in a video taken by a camera placed on a moving body, for example, and a video object positioning device for positioning a specific object appearing in a video.

  In car navigation map creation, road facility management, and water and sewage management in local governments, workers are required to visit the site and check the facilities one by one manually. Therefore, the work time is enormous and the cost burden such as labor cost for management is large. As a conventional technology to help improve the efficiency of equipment management to solve this problem, video, maps, and management information captured by the camera are displayed in conjunction with each other, and the object is automatically recognized from the video, resulting in recognition results. Is displayed by CG or the like (for example, see Patent Document 1). In this technique, cross-correlation between an image of a target object and a template is calculated to perform pattern matching, and a relative distance from the camera to the target object is calculated by using a calculation method similar to stereo stereoscopic vision. As another conventional technique, there is a method of performing appropriate guidance by synthesizing guidance information with a CG on a live-action image while the vehicle is running and presenting it to a driver or the like (see, for example, Patent Document 2). In this technique, only guidance contribution information that contributes to guidance is extracted from a live-action video and is generated as a CG image, and the CG image is combined with a real-time video at the time of current travel and displayed.

JP 2003-337947 A JP 2003-121167 A

The conventional techniques as described above have the following problems.
In reality, the appearance of objects on the image may change due to changes in sunshine, including nighttime, image noise during shooting, and partial omission due to shielding, etc., making it difficult to recognize images There is. This cannot be solved by the method of calculating the cross-correlation between the input image and the template image described in Patent Document 1. Further, the method according to Patent Document 1 has a problem that the recognition rate is lowered when the number of template images held is small. On the other hand, Patent Document 2 does not suggest a method on how to recognize road signs and the like that contribute to guidance. Therefore, a technique that can accurately recognize an image affected by a change in sunlight or an obstruction is desired. In particular, it is desirable to be able to efficiently specify, for example, the type of road sign.
Further, in the case of a normal positioning system such as GPS, a positioning error is always accompanied. However, in the case of a facility management efficiency support device that requires accurate positioning of an object, it is necessary to eliminate as much as possible disturbances associated with GPS positioning results, positioning errors associated with stereo stereoscopic vision, and the like.

The present invention has been made to solve the above problems, and even if there are apparent differences due to the situation at the time of shooting and image noise in the images taken of the same type of object. An object of the present invention is to obtain an object recognition device that can accurately recognize an object.
Another object of the present invention is to obtain a video object positioning device that can be used by being incorporated in an object recognition device, and that accurately measures an object on a video by reducing a GPS positioning error and an error associated with image recognition.

  The object recognition apparatus according to the present invention stores in advance a video information database for storing video information, an image acquisition means for acquiring an arbitrary frame image from the video information in the video information database, and a plurality of template images related to the object by type. For each of the template image storage means and the plurality of template images of the template image storage means, respectively, and a correlation coefficient matrix between the extracted image feature quantities is obtained, and then the image acquisition means An input image to be recognized is extracted from the acquired frame image, a plurality of image feature amounts are extracted from the input image in the same manner as the template image, a correlation coefficient matrix between the image feature amounts is obtained, and the obtained input image By comparing the correlation coefficient matrix of multiple templates and the correlation coefficient matrix of multiple template images, the input image and multiple template images The distance between each calculated, the calculated distance is an object of the smallest template image is obtained by an image recognition processing means for recognizing a target object in the input image.

  According to the present invention, by using a correlation coefficient or a correlation coefficient matrix, it is possible to accurately recognize an object on a video without being affected by luminance change, image noise, and the like.

Embodiment 1 FIG.
In the description of the present invention, “geographic coordinates” refer to latitude, longitude, and altitude in a geographic coordinate system. Further, “image coordinates” refers to coordinates on a two-dimensional image (normalized camera coordinates). Further, the “object” refers to an object that is the object of recognition on the image. For example, “objects” in the case of images used for road facility management include road signs, road white lines, road surface signs, manholes, illumination lights, traffic lights, and the like.

FIG. 1 is a block diagram showing a functional configuration of an object recognition apparatus common to the embodiments of the present invention.
In the figure, a video information acquisition device 90 is means for acquiring and collecting video information obtained by imaging a road using a camera and position information and posture information of a camera that has taken this video information. The video information database 100 is means for storing the video information acquired by the video information acquisition device 90 and captured by the camera on the road and the position information and posture information of the camera that has captured the video information synchronized with the video information. .
For example, as shown in FIG. 2, the video information acquisition device 90 has a configuration in which a high-precision positioning device 201, a camera 202, and an orientation sensor 203 are mounted on a moving body (vehicle) 204. A video information database 100 is also mounted on the moving body (vehicle) 204. The vehicle 204 is driven on the road in the information collection target area, and video information on the road is acquired by the camera 202. At that time, the position information of the camera 202 is measured by the high-precision positioning device 201 such as GPS, and the orientation of the camera 202 is measured by the direction sensor 203. Further, the measured camera position information and camera posture information are recorded in the video information database 100 in synchronization with each frame of the road video information acquired at that time. As a method of synchronizing in this case, for example, when video information is recorded every moment in the video information database 100, the computer system time at each time is recorded in the image header of the video information or the audio corresponding to the video. Record on the track. Similarly, when the position information such as the GPS positioning result is recorded every moment, the computer system time at each time is added to the position information and recorded. Therefore, the video information database 100 records video information synchronized with the computer system time, GPS positioning results (geographic coordinates of the camera), and camera attitude information. As a reference signal for synchronizing the GPS positioning result and the video information, signals such as the internal time of the video capture program and the GPS clock time may be used in addition to the computer system time.

  The camera 202 used in the video information acquisition apparatus 90 may be a single-lens camera or a plurality of cameras. Moreover, not only a visible camera but a line scan laser, an infrared camera, etc. may be sufficient. Therefore, the acquired video information is one video or a plurality of videos obtained by viewing the object at different angles. In addition, depth image information is obtained when a line scan laser is used, and thermal image information is obtained when an infrared camera is used. In the first embodiment, video information taken with a single-lens camera will be described as an example. However, the present invention can be similarly applied to other video information. The camera position information referred to here is information on the geographical coordinates of the camera measured in synchronization with each frame of the video information. Furthermore, the camera posture information is information such as triaxial angles (yaw angle, roll angle, pitch angle), angular acceleration, and the like, which are similarly measured in synchronization with each frame of video information.

  In FIG. 1, an image acquisition unit 101 is means for acquiring an arbitrary frame image from video information in the video information database 100. The template image storage unit 105 is a means for storing in advance a plurality of template images related to the object by type. Here, as the template image, for example, when the object is a road sign, it is assumed that a plurality of images taken under various circumstances are prepared for each type of road sign. The image recognition processing unit 102 is means for recognizing an object from a frame image acquired by the image acquisition unit 101 using a template image. The positioning processing unit 103 is a means for calculating the geographic coordinates of the object recognized by the image recognition processing unit 102 based on the image coordinates of the object and the position information and posture information of the camera. The display data storage unit 120 is a means for storing display data such as various types of map information to be displayed on the display 110 and management information related to the object. The display processing unit 104 is a means for performing processing for displaying the recognition result and positioning result of the object in association with the display target data.

Next, the overall operation of the object recognition apparatus will be described according to the operation procedure shown in FIG.
First, the image acquisition unit 101 acquires an arbitrary frame image from the video information stored in the video information database 100 (step ST101). The acquisition of the input image may be performed in real time during the collection of the video information from the video information acquisition device 90, or may be performed from the video information collected in advance. Next, in the image recognition processing unit 102, an input image to be recognized is extracted from the frame image acquired by the image acquisition unit 101, and a plurality of objects prepared in advance in the template image storage unit 105 are extracted from the input image. Using the template image, the target object is recognized by processing to be described later, and the image coordinates of the recognized object are obtained (step ST102). The positioning processing unit 103 obtains geographic coordinates for the object recognized by the image recognition processing unit 102 by a method described later (step ST103). Note that recognition may fail here, or an object that is not a recognition target may be measured. In this case, an object image may be manually selected from the video frame. Next, the display processing unit 104 synchronizes display target data such as various types of map information read from the display data storage unit 120 and live-action video of the video information database 100 with the geographical coordinates of the target obtained by the positioning processing unit 103. To display on the display 110. At that time, display processing such as an icon representing an object obtained by the image recognition processing unit 107 is superimposed and displayed on the position of the object on the synchronized display object data (step ST104).

Next, details of the processing of the image recognition processing unit 102 will be described.
The image recognition processing unit 102 recognizes a target object (for example, a road sign) from the frame image acquired by the image acquisition unit 101 using a correlation coefficient matrix of image feature amounts. The operation of the image recognition processing unit 102 in this case is performed according to the procedure shown in FIG.
First, the image recognition processing unit 102 reads an image of an object necessary for recognition, that is, a plurality of template images for each type related to road signs, from the template image storage unit 105 (step ST401). Next, the image recognition processing unit 102 extracts a plurality of image feature amounts for each template image (step ST402). The image feature amounts obtained here are, for example, horizontal image coordinates, vertical image coordinates, pixel values in the Y plane (luminance component) of the template image, pixel values in the Cb plane (color difference component), and Cr plane (color difference component). Pixel values, vertical edges, horizontal edges (edge information), and the like.

Next, the image recognition processing unit 102 obtains a correlation coefficient matrix between the image feature amounts obtained in step ST402 for each of the template images (step ST403). As a result, one correlation coefficient matrix is obtained for each template image.
For example, the feature vector f (x, y) at the image coordinates (x, y) is defined as the following equation (1).
f (x, y) = [x, y, Y (x, y), Cb (x, y), Cy (x, y), He (x, y), Ve (x, y)] (1)
Here, x is the image coordinate in the horizontal direction, y is the image coordinate in the vertical direction, Y (x, y) is the value of the Y plane in the image coordinate (x, y), and Cb (x, y) is the image coordinate (x , Y), the value of the Cb plane, Cy (x, y) is the value of the Cr plane at the image coordinates (x, y), and He (x, y) is the value of the horizontal edge at the image coordinates (x, y). , Ve (x, y) is the value of the edge in the vertical direction at the image coordinates (x, y).

  The image recognition processing unit 102 sequentially moves the search window on the frame image acquired by the image acquisition unit 101, and acquires an image in the search window (step ST404). In this case, the size of the search window is arbitrarily changed according to the target object. Hereinafter, the image in the search window acquired in step ST404 will be referred to as an “input image”. Next, in the same manner as in steps ST402 and ST403, an image feature amount is calculated for the acquired input image, and a correlation coefficient matrix between the image feature amounts is obtained (step ST405). Next, template matching is performed by sequentially comparing the obtained correlation coefficient matrix of the input image and the correlation coefficient matrices of the plurality of template images, and the distance between the input image and the template image is calculated (step ST406).

以上から、距離ｓが小さいほど入力画像とｋ番目のテンプレート画像との類似度が高く、距離ｓが大きいほど入力画像とテンプレート画像の類似度が低いものと解釈することができる。
ここで、上記の一般化固有値について説明する。一般化固有値は、入力矩形画像の相関係数行列をテンプレート画像の相関係数行列にアファイン変換する際の縮尺量（スケール）を意味している。一般化固有値が１に近いほど小さい縮尺量でアファイン変換可能であり、二つの相関係数行列は似通っていると判断できる。一方、一般化固有値が１から離れるほど大きな縮尺量でアファイン変換可能する必要があり、二つの相関係数行列は似通っていないと判断できる。 The distance between the input image and the template image is calculated using, for example, the following equation (4).

From the above, it can be interpreted that the smaller the distance s, the higher the similarity between the input image and the kth template image, and the lower the distance s, the lower the similarity between the input image and the template image.
Here, the generalized eigenvalue will be described. The generalized eigenvalue means a scale amount (scale) when the correlation coefficient matrix of the input rectangular image is affine transformed into the correlation coefficient matrix of the template image. As the generalized eigenvalue is closer to 1, affine transformation can be performed with a smaller scale amount, and it can be determined that the two correlation coefficient matrices are similar. On the other hand, it is necessary to perform affine transformation with a larger scale amount as the generalized eigenvalue is away from 1, and it can be determined that the two correlation coefficient matrices are not similar.

Next, an advantage of performing template matching using the correlation coefficient matrix between image feature amounts as described above will be described with an example.
FIG. 5 shows three images having the same appearance with respect to the same road sign, the average value of red, the average value of blue, and the correlation coefficient between red and blue in each image. The three images illustrated in FIG. 5 indicate that the average value of red and blue has changed greatly due to changes in luminance and noise because the situation at the time of shooting is different. On the other hand, the correlation coefficient of each image is represented as a substantially constant value that is not greatly affected by luminance change or noise. Similarly, a correlation coefficient such as an edge has a value that is not easily affected by a change in contrast of the image. That is, by using a correlation coefficient that is not affected by changes in luminance or the like, it means that an object can be accurately recognized from images with different appearances. Therefore, according to the image recognition processing unit 102 that matches the template image and the input image using the correlation coefficient matrix between the image feature amounts, the image recognition target 102 can also be applied to an image captured under a situation where there is a change in sunlight. It is possible to recognize the object accurately.

Next, details of the processing of the positioning processing unit 103 will be described according to the operation procedure shown in FIG. Here, an example is to obtain the geographical coordinates of the road sign on the video.
First, the positioning processing unit 103 acquires, as an input, image coordinates of an object recognized by the image recognition processing unit 102 (step ST901). Note that, when the image recognition processing unit 102 fails to recognize the image coordinates of the target object acquired as the input, the image of the image selected by the user using an input device such as a mouse or a touch pen after confirming the video. Coordinates may be used. Next, the positioning processing unit 103 detects the same object as the object for which the image coordinates were acquired in step ST901 from several frames before and after the frame image in which the object recognized by the acquired image recognition processing unit 102 is captured. Image coordinates of these objects are acquired (step ST902). As a method for detecting the same object, for example, matching using the correlation coefficient matrix of the image feature amount in the image recognition processing unit 102 described above is used, or Tuzel, O .; Porikli, F .; Meer, P., “Region Covariance: A Fast Descriptor for Detection and Classification”, European Conference on Computer Vision (ECCV), May 2006 (ECCV 2006). That's fine.

上記（６）式に示すコスト関数Ｅの第一項は対象物の画像座標誤差を最小にする項であり、第二項はＧＰＳの測位誤差を最小にする項であり、第三項はカメラ姿勢角の計測誤差を最小化するための項である。 Next, the positioning processing unit 103 acquires, from the video information database 100, the geographic coordinates and attitude angle vectors (yaw angle, roll angle, pitch angle) of the camera at the time when each of the plurality of frame images in which the object is photographed are captured. (Step ST903). Next, a cost function is generated using a plurality of image coordinates for the same target object, a plurality of geographical coordinates of the camera that has captured the target object, and a plurality of posture angles of the camera, and the generated cost function is minimized. The geographical coordinates of such an object are calculated (step ST904). The generated cost function is represented by the following equation (6).

The first term of the cost function E shown in the above equation (6) is a term that minimizes the image coordinate error of the object, the second term is a term that minimizes the GPS positioning error, and the third term is the camera. This is a term for minimizing the measurement error of the posture angle.

  Normally, the camera position information and the camera attitude information acquired by the video information acquisition apparatus as shown in FIG. However, by minimizing the cost function E consisting of the three terms of the above equation (6), the optimal object can be obtained while simultaneously minimizing the image coordinate error, GPS positioning error, and camera attitude measurement error of the object. Can be calculated from the video.

この場合、対数尤度関数を最大化して、対象物の地理座標、カメラ地理座標およびカメラ姿勢角ベクトルを求めることで、画像座標誤差の確率分布、カメラ位置測位誤差の確率分布、カメラ姿勢角度計測誤差の確率分布が、それぞれ任意の場合でも、最適な対象物の地理座標を算出することが可能である。 Moreover, the positioning process part 103 may obtain | require the geographical coordinate of a target object using the log likelihood function of following (7) Formula as a method different from the method of using the said cost function.

In this case, by maximizing the log likelihood function and obtaining the geographical coordinates of the object, the camera geographical coordinates, and the camera attitude angle vector, the probability distribution of the image coordinate error, the probability distribution of the camera positioning error, and the camera attitude angle measurement Even when the probability distribution of errors is arbitrary, it is possible to calculate the optimal geographical coordinates of the object.

  As described above, the positioning processing unit 103 uses a plurality of frame images before and after a frame image in which the object recognized by the image recognition processing unit 102 is captured, and a plurality of images of the object acquired from these frame images. Generating a cost function or log-likelihood function based on coordinates, multiple camera positions and multiple camera attitude angles at the time of capturing an object, minimizing the cost function or maximizing the log-likelihood function Thus, it is possible to calculate the optimum geographical coordinates of the object in consideration of the image coordinate error of the object, the positioning error of the camera position, and the measurement error of the camera attitude angle. Further, not only the optimum geographical coordinates of the target object but also the optimum values of the camera position and the camera posture angle can be obtained at the same time.

  In the above example, the positioning process for calculating the geographical coordinates of the recognized object based on the image coordinates has been described for the object recognized by the image recognition processing unit 102. Here, the positioning processing unit 103 may be separated from the image recognition processing unit 102 and may be configured as a video object positioning device that performs positioning of an object on a video other than the recognized target object. That is, using the video information collected in the video information database 100 and the camera position information and camera posture information synchronized with the video information, an arbitrary object on the video is selected using an input device such as a mouse or a touch pen. Then, by acquiring the image coordinates of the selected object and performing the same positioning process as described above, it is possible to calculate the geographic coordinates of an arbitrary object shown on the video. Further, in combination with a display processing unit 104 to be described later, it is possible to display an icon of an object that is positioned in synchronization with a live-action image or map information.

Next, details of the processing of the display processing unit 104 will be described.
As described in the operation of the video information acquisition device 90 of FIG. 1, the video information database 100 records video information synchronized with the system time of the computer, GPS positioning results (camera geographic coordinates), and camera attitude information. ing. Therefore, the display processing unit 104 can read out the geographic coordinates and camera posture information of the camera in association with the video information by referring to the system time of these computers. That is, it is possible to determine the geographic coordinates of the photographed image (camera coordinate coordinates) with respect to the photographed image.
On the other hand, map data stored in the display data storage unit 120, for example, a two-dimensional map such as a car navigation map and a piping map, also has geographical coordinates of each point. Therefore, when displaying a live-action video, the display processing unit 104 reads out map data having geographic coordinates that match the geographic coordinates of the frame image and displays them together on the display screen of the same display. Therefore, both corresponding images can be displayed in synchronization.

In addition, when geographical coordinates are given in advance in a bird's eye view in which a road is three-dimensionally displayed, it is also possible to display them in a similar manner. At that time, the display processing unit 104 synchronously displays the position of these objects based on the geographical coordinates of the recognized object or the object specified on the video, which has been subjected to the positioning process by the positioning processing unit 103. You may make it perform the process designated on the two-dimensional map and bird's-eye view which are present, or displaying with the icon of a target object.
Further, the display processing unit 104 determines the positions of these objects based on the geographical coordinates and the image coordinates of the recognized objects or the objects specified on the video, which have been subjected to the positioning processing by the positioning processing unit 103, by computer graphics. A process of superimposing and displaying a figure or a position mark of an object created in the process may be performed.

Display examples by the processing of the display processing unit 104 will be described with reference to FIGS. Here, an example of an object of equipment management is a manhole.
FIG. 7 is a facility management chart in which live-action images (vehicle images and manhole recognition results), camera viewpoint conversion images, two-dimensional maps, piping diagrams (also two-dimensional maps), and facility information are all synchronized based on geographic coordinates. This represents a state in which the same screen of the display 110 is displayed. The camera viewpoint converted video is generated by converting the camera viewpoint from the photographed image by the display processing unit 104. The facility information describes the manhole number, type (for water supply, for sewage, size, shape, etc.), installation address, etc., and is associated with geographic coordinates. Therefore, data corresponding to other display images is displayed. On the live-action video, an object (manhole in this example) recognized by the image recognition processing unit 102 and automatically positioned by the positioning processing unit 103 is displayed, for example, in a rectangular frame surrounded by color. Similarly, on the two-dimensional map or the piping diagram, the recognized and automatically positioned objects are displayed by, for example, color-coded icons. In addition, on the two-dimensional map, the position of the camera that is capturing the live-action video is displayed with an icon indicating the shooting direction (camera posture angle). Furthermore, on the live-action image, the position of a pipe such as a sewer pipe buried in a road read from a pipe diagram or the like is displayed, for example, with a superimposed broken line.

  On the other hand, FIG. 8 shows the situation in the snowy region, which is a live-action video of the current snowy road (present video and CG manhole position display), a past real-time video taken in the summer, etc. Manhole recognition result), car navigation screen, equipment information is displayed. In the snowy region, road equipment such as manholes are buried in snow during the winter, which often hinders equipment inspection work. Therefore, a CG composite arrow indicating the position of the manhole buried under the snow is displayed on the current live-action video of the snowy road to make it easier to search for the manhole. In addition, on the display screen, various pieces of facility information in the vicinity where the video is photographed are displayed in synchronization with geographical coordinates. In this way, in order to display the current live-action video at the same time as the past real-life video and display data, the display processing unit 104 causes the current position (position coordinates) of the camera or the current position (position coordinates) of the vehicle on which the camera is mounted ) On the basis of the past actual captured video and / or display data.

  As described above, according to the display processing unit 104, various map information and facility information are displayed together in synchronism with a live-action video based on an object automatically recognized from video information or a geographical coordinate of a specified object. Since the position of the object is displayed as an icon or CG on the map and the live-action image, it can be visualized so that it can be confirmed at a glance. Therefore, management work such as water and sewage management, power facility management, and map management for car navigation can be made more efficient, and it is possible to contribute to a drastic reduction in labor cost related to management. In particular, it is possible to accurately display and convey the position of an object that cannot be confirmed only with a live-action image such as a snowy area or a disaster-affected area, thereby enabling efficient facility inspection work.

  As described above, according to the object recognition apparatus of the first embodiment, the image recognition processing unit 102 uses a plurality of template images prepared in advance, such as image coordinates, luminance information, color information, and edge information. Each image feature amount is extracted, a correlation coefficient matrix between the extracted image feature amounts is obtained, and then an input image to be recognized is extracted from the frame image acquired by the image acquisition unit 101. Similar to an image, a plurality of image feature amounts are extracted to obtain a correlation coefficient matrix between the image feature amounts, and the input image correlation coefficient matrix is compared with a correlation coefficient matrix of a plurality of template images. And calculate the distance between the template images, and recognize the template image object with the smallest calculated distance as the object of the input image. And those, therefore, from the video information including video information and noise taken in situations where there is a change of daylight, etc., makes it possible to accurately recognize the object.

Further, according to the object recognition device of the first embodiment, the positioning processing unit 103 acquires the image coordinates of the object recognized by the image recognition processing unit 102, and before and after the recognized object is shown. The same object as the object is detected from the images of the plurality of frames, the image coordinates of the detected object are obtained, and the same object including the recognized object is obtained from the video information database 100. A camera position and a camera attitude angle at the time of copying are respectively acquired, and a cost function or a log likelihood function is generated based on the plurality of image coordinates, camera position and camera attitude angle of the acquired object, and the cost function is The geographic coordinates of the object, the geographic coordinates of the camera, and the camera attitude angle that are minimized or have the maximum log likelihood function are calculated. Therefore, it is possible to obtain the optimum geographical coordinates for the object recognized by the image recognition processing unit 102.
Furthermore, according to the object recognition apparatus of the first embodiment, the display processing unit 104 converts the actual captured video acquired from the video information database 100 and / or various map information acquired from the display data storage unit 120 into the positioning processing unit. The information is displayed on the display 110 in synchronization with the geographical coordinates of the object calculated in 104, and the position of the object is displayed as an icon or CG on a map and a live-action video. Therefore, it is possible to visually display a specific object that has been automatically recognized / positioned in a visually easy-to-understand manner, contributing to the efficiency of facility management work and the like.

Embodiment 2. FIG.
In the second embodiment, the image recognition processing unit 102 compares the correlation coefficient matrix for each partial region common to the input image and the template image acquired in the same manner as in the first embodiment. The use of a method for recognizing an object will be described.
FIG. 9 shows a matching operation example of the image recognition processing unit 102 according to the second embodiment. In this case, the image recognition processing unit 102 designates a plurality of partial regions of the image for each of the plurality of template images in the template image storage unit 105. Then, a plurality of image feature amounts are extracted for each designated partial region, and a correlation coefficient matrix between the extracted image feature amounts is calculated for each partial region. Next, an input image to be recognized is extracted from the frame image acquired by the image acquisition unit 101. For this input image, a plurality of partial areas corresponding to the template image are designated, a plurality of image feature quantities are extracted for each partial area, and a correlation coefficient matrix between the image feature quantities is obtained. Next, the correlation coefficient matrix of the partial area of the template image is compared with the correlation coefficient matrix of the corresponding partial area of the input image, and the distance between the partial areas is calculated.

また、各部分領域で求めた距離のうち、距離の小さい上位の数個の和をとり、その和を入力画像とｋ番目のテンプレート画像との間の距離としてもよい。 As described above, the image recognition processing unit 102 calculates the distance for each partial region and takes the sum thereof, so that the road sign is partially shielded, or numbers or characters are included in the road sign. Even when it exists, the degree of similarity can be calculated individually, thereby enabling more accurate image recognition.
The distance between the input image and the kth template image may be obtained using the following equation (10).

In addition, among the distances obtained in each partial region, the top several sums with a small distance may be taken, and the sum may be used as the distance between the input image and the kth template image.

  Usually, when there is a partial shielding object or the like with respect to the target object, the distance between the input image and the template image may be large in a partial region where the shielding object is reflected. However, as described above, the image recognition processing unit 102 removes the largest distance from the sum of the distances obtained in each partial area, or the upper number of values having a small distance among the distances obtained in each partial area. Since the sum of the individual is taken so as not to be affected by the partial area where the shielding object is involved, it is possible to accurately recognize the object even if it is partially shielded. .

  As described above, according to the object recognition apparatus of the second embodiment, the image recognition processing unit 102 designates a plurality of partial areas of the image for each of the plurality of template images, and the luminance for each designated partial area. A plurality of image feature amounts such as components, color information, and edge information are extracted, a correlation coefficient matrix between the extracted image feature amounts is obtained for each partial region, and then recognized from the frame image acquired by the image acquisition unit 101 A target input image is extracted, and for the input image, a plurality of image feature amounts are extracted for each of a plurality of specified partial areas in the same manner as the template image, and a correlation coefficient matrix between the image feature amounts is obtained, respectively. Compare the correlation coefficient matrix of the partial area and the correlation coefficient matrix of the corresponding partial area of the input image to calculate the distance between the two partial areas, respectively, Based on the distance between the regions respectively calculated distances between the input image and each template image, the object distance between the calculated image is the smallest template image, and to recognize as the object of the input image. Therefore, it is possible to accurately recognize the target object from the video information photographed in a situation where there is a change in sunshine or the like and the video information in which an object partially including shielding or specific display is captured.

Embodiment 3 FIG.
In the third embodiment, the image recognition processing unit 102 generates a correlation between characteristic image feature amounts as a correlation pattern from a plurality of template images for the same or the same type of object, and uses the generated correlation pattern A method for recognizing an image of a target object will be described.
FIG. 10 shows an operation procedure of the image recognition processing unit 102 according to the third embodiment.
First, the image recognition processing unit 102 reads a plurality of template images for the same or the same type of object prepared in advance from the template image storage unit 105 (step ST701). Next, select a plurality of partial areas of any size common to the read template images, and for each selected common partial area, select two image feature quantities having a significant correlation in the partial area, A correlation coefficient between the two selected image feature amounts is obtained, and a function for calculating the similarity based on the obtained correlation coefficient value (hereinafter referred to as a similarity calculation function) is created (step ST702).

Here, the method of creating the similarity calculation function in step ST702 will be described using the example of FIG. In the image recognition processing unit 102, first, one partial area 801 common to a plurality of template images for the same type of object is selected, and two image feature quantities having a useful correlation in the partial area 801 are selected as red image features. Select plane and blue plane. Then, a correlation coefficient between the selected red plane and blue plane is obtained for each of a plurality of template images. Next, a similarity calculation function 804 that calculates the similarity based on the calculated values of the respective correlation coefficients is created. Information that summarizes these partial regions 801, correlation coefficients, and similarity calculation functions is held as one correlation pattern. Similarly, different partial areas 802, 803,... Are selected, correlation coefficients and similarity calculation functions are obtained for the respective partial areas, and correlation patterns are generated and held. Furthermore, the generation of the plurality of correlation patterns is also performed for each type of template image). As a method of generating this correlation pattern, for example, Jerome Friedman, Trevor Hastie and Robert Tibshirani “Additive Logistic Regression: a Statistical View of Boosting”. Ann. Statist. 28, no. 2 (2000), 337407 A learning algorithm called Adaboost or Logiboost may be used, or other methods such as a support vector machine or a neural network may be used.

  Returning to the description of the operation in FIG. 10, the image recognition processing unit 102 sequentially moves the search window with respect to the input image acquired by the image acquisition unit 101, and acquires the image in the search window as the input image to be recognized ( Step ST703). In this case, the size of the search window is arbitrarily changed according to the target object. Also, the image coordinates of the input image at this time are calculated. Next, in accordance with the correlation pattern of the template image generated and held in step ST702, a corresponding partial region of the input image is extracted, a correlation coefficient between image feature amounts in the extracted partial region is calculated, and the calculated phase is calculated. The similarity is calculated by substituting the number of relations into the correlation pattern similarity calculation function (step ST704). Next, the processes in steps ST704 and ST705 are repeated for the number of correlation patterns obtained in step ST702, and the similarity calculated in each correlation pattern is added to calculate the final similarity (step ST706). Similarly, for other types of objects, the final similarity is calculated by adding the similarities obtained from the correlation pattern. From the final similarities for all types calculated in this way, the type having the largest similarity is obtained, and the obtained type of object is recognized as the target object (step ST707).

  In general, when there are a large number of template images, the method of matching these with input images one by one requires time and is inefficient. However, in the case of the image recognition processing unit 102 according to the third embodiment, several correlation patterns related to useful correlation coefficients are generated from the plurality of template images for the same or the same type of object, and these generated Since the correlation pattern is used to recognize the input image in units of the object type, the recognition processing time can be greatly improved.

  As described above, according to the object recognition device of the third embodiment, the image recognition processing unit 102 selects a plurality of partial areas having a common arbitrary size for a plurality of template images for the same type of object, For each selected common partial area, two image feature quantities having a significant correlation in the partial area are selected, a correlation coefficient between the two selected image feature quantities is obtained, and based on the obtained correlation coefficient value A similarity calculation function for calculating the similarity is created, and a plurality of correlation patterns including the obtained common partial region, two image feature amounts, and the similarity calculation function as a set are generated and stored, and then an image is acquired. The input image to be recognized is extracted from the frame image acquired by the unit 101, and the corresponding common partial region is extracted according to each of the plurality of correlation patterns generated and held for the input image. Select each, calculate the correlation coefficient between the image feature quantities for the selected common partial area in the same way as the template image, and assign the calculated correlation coefficient to the corresponding correlation pattern similarity calculation function, respectively The similarities are calculated, and the calculated similarities are added to recognize the same type of object of the template image as the object of the input image when the added value exceeds a certain value. I am doing so. Therefore, it is possible to accurately recognize the target object from the video information photographed in a situation where there is a change in sunshine or the like and the video information in which an object partially including shielding or specific display is captured. In addition, since the image recognition processing unit 102 according to the third embodiment performs the recognition process using the correlation pattern formed for each type of object, a process for performing pattern matching on individual template images. Compared to the above, the processing time can be greatly improved. This is particularly effective when using a large amount of template images.

  Although the object recognition apparatus of the present invention has been described with respect to management support for road facilities in the above embodiment, the apparatus of the present invention is also mounted on a train, for example, for collecting and managing route status information. Applicable. The apparatus of the present invention can also be used for car navigation using live-action video. For example, a positioning device such as GPS and a camera are mounted on a vehicle, and white line recognition, road signs, traffic signal recognition, pedestrians, bicycles, oncoming vehicles, buildings, etc. It recognizes and presents such information and the guidance information produced | generated based on these information to a driver. Furthermore, the device according to the present invention, for example, installs a surveillance camera in a building, school, or town, recognizes the face of a specific person or intruder by the image recognition processing unit 102, and secures the recognition result and its video. It can also be applied to security using video transmitted to the center. Furthermore, the apparatus of the present invention can be used for image search and classification on the Internet. For example, a large amount of images are collected on the Internet and stored in the video information database 100, and a specific person image or a specific object image is recognized from the image of the video information database 100 by the image recognition processing unit 102. Can be displayed on a personal computer or a mobile phone according to the user's request.

It is a block diagram which shows the function structure of the object recognition apparatus by each embodiment of this invention. It is explanatory drawing which shows the structure of the video information acquisition apparatus which concerns on Embodiment 1 of this invention. It is a flowchart which shows the operation | movement procedure of the object recognition apparatus which concerns on Embodiment 1 of this invention. It is a flowchart which shows the operation | movement procedure by the method of utilizing a correlation coefficient matrix for every partial area | region of the image recognition process part which concerns on Embodiment 1 of this invention. It is explanatory drawing which shows the advantage of the template matching using the correlation coefficient matrix between the image feature-values concerning Embodiment 1 of this invention. It is a flowchart which shows the operation | movement procedure of the positioning process part which concerns on Embodiment 1 of this invention. It is explanatory drawing which shows the example of a display by operation | movement of the display process part which concerns on Embodiment 1 of this invention. It is explanatory drawing which shows the other example of a display by operation | movement of the display process part which concerns on Embodiment 1 of this invention. It is explanatory drawing which shows the example of the matching method by the image recognition process part which concerns on Embodiment 2 of this invention. It is a flowchart which shows the operation | movement procedure by the method of using the correlation coefficient pattern of the image recognition process part which concerns on Embodiment 3 of this invention. It is explanatory drawing which shows the creation method of the similarity calculation function by the image recognition process part which concerns on Embodiment 3 of this invention.

Explanation of symbols

  90 video information acquisition apparatus, 100 video information database, 101 image acquisition unit, 102 image recognition processing unit, 103 positioning processing unit, 104 display processing unit, 105 template image storage unit, 110 display, 120 display data storage unit, 201 high accuracy Positioning device, 202 camera, 203 direction sensor, 204 moving body.

Claims (20)

A video information database for storing video information;
Image acquisition means for acquiring an arbitrary frame image from the video information of the video information database;
Template image storage means for storing in advance a plurality of template images related to the object by type;
For each of the plurality of template images in the template image storage unit, a plurality of image feature amounts are extracted, a correlation coefficient matrix between the extracted image feature amounts is obtained, and then from the frame image acquired by the image acquisition unit Extracting an input image to be recognized, extracting a plurality of image feature amounts for the input image in the same manner as the template image, obtaining a correlation coefficient matrix between the image feature amounts, and calculating the correlation coefficient of the obtained input image Compare the matrix and the correlation coefficient matrix of multiple template images to calculate the distance between the input image and multiple template images, respectively, and recognize the template image object with the smallest calculated distance as the input image object And an image recognition processing means.   2. The object recognition apparatus according to claim 1, wherein the image feature quantity extracted by the image recognition processing means is image coordinates, luminance information, color information, and edge information. A video information database for storing video information;
Image acquisition means for acquiring an arbitrary frame image from the video information of the video information database;
Template image storage means for storing in advance a plurality of template images related to the object by type;
For each of the plurality of template images of the template image storage means, a plurality of partial areas of the image are designated, a plurality of image feature quantities are extracted for each designated partial area, and a correlation coefficient matrix between the extracted image feature quantities Is obtained for each partial area, and then an input image to be recognized is extracted from the frame image acquired by the image acquisition means, and a plurality of input images are specified for each of the specified partial areas in the same manner as the template image. Image feature quantities are extracted to obtain correlation coefficient matrices between the image feature quantities, and the correlation coefficient matrix of the partial area of the template image is compared with the correlation coefficient matrix of the corresponding partial area of the input image. Calculate the distance between the areas, calculate the distance between the input image and each template image based on the calculated distance between the partial areas, The distance between the out image is the object of the smallest template image, the object recognition apparatus characterized by comprising an image recognition processing means for recognizing a target object in the input image.   4. The object recognition apparatus according to claim 3, wherein the image feature amount extracted by the image recognition processing means is an image coordinate, a luminance component, color information, and edge information.   5. The object recognition apparatus according to claim 3, wherein the image recognition processing means sets the calculated total distance between the partial areas as the distance between the input image and the template image.   The image recognition processing means uses the remaining sum obtained by removing the largest distance among the calculated distances between the partial areas as the distance between the input image and the template image. 4. The object recognition apparatus according to 4.   The image recognition processing means uses, as the distance between the input image and the template image, the sum of the top several of the smallest distances among the calculated distances between the partial regions. 4. The object recognition apparatus according to 4.   The object recognition apparatus according to any one of claims 1 to 7, wherein the image recognition processing means calculates the distance using the generalized eigenvalue when comparing the correlation coefficient matrices. A video information database for storing video information;
Image acquisition means for acquiring an arbitrary frame image from the video information of the video information database;
Template image storage means for storing in advance a plurality of template images related to the object by type;
For a plurality of template images for the same type of object read from the template image storage means, a plurality of common partial areas of any size are selected, and each selected common partial area has a significant correlation in that partial area. Selecting two image feature amounts, obtaining a correlation coefficient between the two selected image feature amounts, creating a similarity calculation function for calculating a similarity based on the calculated correlation coefficient value, and generating the common partial region , Generating and holding a plurality of correlation patterns having the two image feature quantities and the similarity calculation function as a set, and then extracting an input image to be recognized from the frame image acquired by the image acquisition means For the input image, the corresponding common partial area is selected in accordance with each of the plurality of correlation patterns to be generated and held, and the selected common area is selected. Similar to the template image, the correlation coefficient between the image feature amounts is calculated for each segment area, and the calculated correlation coefficient is assigned to the corresponding similarity calculation function of the correlation pattern. The degree of similarity is calculated, the final degree of similarity is calculated by adding the calculated degrees of similarity, and the degree of similarity obtained from the correlation pattern is also added to other types of objects through similar processing. Image recognition processing means for calculating a degree, obtaining a type having the largest similarity from among the final similarities for all types calculated, and recognizing an object of the obtained type as a target object An object recognition apparatus characterized by comprising: The video information database stores video information taken by the camera on the road and position information and posture information of the camera that shot the video information synchronized with the video information.
Acquire the image coordinates of the object recognized by the image recognition processing means, detect the same object as the object from the images of multiple frames before and after the recognized object, and detect these The image coordinates of the target object is acquired, and the camera position and the camera posture angle at the time of copying the same target object including the recognized target object are acquired from the video information database, and a plurality of these acquired target objects are acquired. And a positioning processing means for generating a cost function based on the image coordinates, the camera position and the camera attitude angle, and calculating the geographic coordinates of the object, the camera geographic coordinates, and the camera attitude angle at which the cost function is minimized. The object recognition device according to claim 1, wherein:   11. The object recognition apparatus according to claim 10, wherein the cost function generated by the positioning processing means includes an image coordinate error term, a positioning error term, and a camera attitude angle measurement error term. The video information database stores video information taken by the camera on the road and position information and posture information of the camera that shot the video information synchronized with the video information.
Obtain the image coordinates of the object recognized by the image recognition processing means, detect the same object as the object from the multiple frame images before and after the recognized object, and detect these The image coordinates of the object are acquired, and the camera position and the camera posture angle at the time when the same object including the recognized object is copied are acquired from the video information database, and a plurality of these acquired objects are acquired. A logarithmic likelihood function is obtained based on the image coordinates, the camera position and the camera attitude angle, and positioning processing means is provided for calculating the geographical coordinates of the object, the camera geographical coordinates, and the camera attitude angle that maximize the log likelihood function. The object recognition apparatus according to claim 1, wherein the object recognition apparatus is any one of claims 1 to 9. A display data storage means for storing display data including a two-dimensional map having geographical coordinates, a bird's eye view having geographical coordinates displaying roads in three dimensions, and management information having geographical coordinates is provided inside or outside the apparatus,
A live-action video read from the video information database is displayed on the display, and the display data image selected and read out from the display data storage means is synchronized with the geographical coordinates of the frame image of the real-action video. 13. The object recognition apparatus according to claim 10, further comprising display processing means for performing a display process for displaying together with the photographed video.   The display processing means generates a viewpoint converted video by performing camera viewpoint conversion on the live-action video, and performs display processing for displaying the viewpoint-converted video together with the real video and display data on the display screen. The object recognition apparatus according to claim 13, wherein:   The display processing means performs display processing for displaying an icon of an object corresponding to the geographic coordinates calculated by the positioning processing means at a position of the object on a two-dimensional map and / or a bird's eye view. The object recognition device according to claim 13 or claim 14.   14. The display processing means displays a live-action image and / or an image of display data on a display in synchronization with a current position of a camera or a current position of a vehicle equipped with the camera. The object recognition device according to claim 15.   The display processing means performs display processing for displaying a figure or a position mark of an object corresponding to the geographical coordinates calculated by the positioning processing means at a position of the object on a live-action image. The object recognition device according to claim 1. A video information database for storing video information captured by a camera on a road and position information and posture information of a camera that has captured the video information synchronized with the video information;
An input device for selecting any object on the video read from the video information database;
The image coordinates of the selected object are acquired, the same object as the object is detected from images of a plurality of frames before and after the object is captured, the image coordinates of these detected objects are acquired, and the video information A camera function and a camera attitude angle at the time of copying the same object including the selected object are respectively acquired from the database, and a cost function is based on the plurality of image coordinates, the camera position and the camera attitude angle of the acquired object. And calculating the geographic coordinates of the object, the geographic coordinates of the camera, and the camera posture angle that minimize the cost function.   19. The video object positioning apparatus according to claim 18, wherein the cost function generated by the positioning processing means includes an image coordinate error term, a positioning error term, and a camera attitude angle measurement error term. A video information database for storing video information captured by a camera on a road and position information and posture information of a camera that has captured the video information synchronized with the video information;
An input device for selecting any object on the video read from the video information database;
The image coordinates of the selected object are acquired, the same object as the object is detected from a plurality of frame images before and after the object is captured, and the image coordinates of these detected objects are acquired from the video information database. , Acquiring a camera position and a camera posture angle at the time of copying the same object including the selected object, and a log likelihood function based on the plurality of image coordinates, camera position and camera posture angle of the acquired object And calculating the geographical coordinates of the object, the geographical coordinates of the camera, and the camera attitude angle that maximize the log likelihood function.

Images