JP2015032133A - Object search system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an object search system capable of efficiently searching a specific type of object from a video without omission.SOLUTION: The object search system includes: a video input unit 1 for inputting video data; an object extraction unit 3 for extracting an object region from an image data frame configuring the input video data; an object type identification unit 2 for identifying an object type in the object region on the basis of one or more first featured values of the image in the object region; an object type certainty calculation unit 4 for calculating object type certainty on the basis of one or more second featured values of the extracted object region; an object information storage unit 5 for storing the object type and the object type certainty in association for each image data frame; and a search unit 6 for searching the image data frame and the object type certainty corresponding to the object type to be input as search conditions from the object information storage unit 5, and for outputting them to a display unit 7.

Description

  The present invention relates to an object search system that detects an object of a specific type from an image.

In order to ensure security, with the increasing demand for surveillance cameras, the importance of analysis support technology for recorded images is increasing. In the case of an application for the purpose of detecting a specific target, conventionally, since it was detected by visual observation from a large amount of images, it took a lot of manpower and time. In order to improve detection efficiency, a function that can narrow down the search range from the target attribute information is required. In particular, there is a demand for a technique that can narrow down the search target based on information such as the type of detection target, for example, whether the target is a person, a motorcycle, or a vehicle.

  Japanese Unexamined Patent Application Publication No. 2005-210573 (Patent Document 1) is a technique for searching for an object from recorded video. In Patent Document 1, individual objects appearing in video data are detected, and a period in which the same object appears in a plurality of image frames is defined as one video period. What is highlighted is listed. This makes it possible to grasp the video content in a shorter time.

JP 2005-210573 A

  However, according to Patent Document 1, although the same object in the video data can be highlighted, the identification of the type of the object included in the video data and the certainty of the type of the identified object are considered at all. Not. Therefore, even if individual objects appearing in the video can be searched, it is not possible to search by narrowing down to a specific type of object, for example, “two-wheeled vehicle”. For this reason, even a user who wants to search for only “two-wheeled vehicles” must refer to individual objects including those other than two-wheeled vehicles in order.

  Therefore, the present invention provides an object search system that can efficiently search for a specific type of object from video data.

  The present invention includes a video input unit that inputs video data, an object extraction unit that extracts an object region from an image data frame that constitutes the input video data, and one or more first ones of images in the object region. An object type identifying unit for identifying an object type in the object region based on a feature amount, an object type certainty factor calculating unit for obtaining an object type certainty factor based on one or more second feature amounts of the extracted object region, An object information storage unit that stores an object type and an object type certainty factor in association with each image data frame, and an image data frame and an object type certainty factor corresponding to the object type input as a search condition. It has a search part which searches from an information storage part and outputs it to a display part.

  According to the present invention, it is possible to provide an object search system that can efficiently search for a specific type of object from video data.

  Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

It is a functional block diagram of the object search system concerning Example 1 of the present invention. It is a functional block diagram of an object classification identification part. It is explanatory drawing of the feature-value calculated in a feature-value calculation part. It is explanatory drawing of the feature-value extracted by the feature-value calculation part with respect to the image of a car. It is explanatory drawing of the feature-value extracted by the feature-value calculation part with respect to the image of a bicycle. It is explanatory drawing of the feature-value extracted by the feature-value calculation part with respect to the person's image. It is a figure which shows the example of the information stored in the object information storage part. It is a figure which shows the example of the identification reference | standard stored in the identification reference | standard storage part. It is a processing flow figure by an object classification judging part. It is explanatory drawing of each data item stored in an object information storage part. It is a figure which shows the example of the information stored in the object information storage part. It is explanatory drawing of the determination method using a classification score. It is a figure explaining the process by the tracking part between frames. It is a figure which shows the state of the object information storage part after an inter-frame tracking process. It is a figure which shows the state which an object hides in a shield during a movement. It is a figure showing the state which an object area | region touches the edge part of an image data frame. It is a figure showing the relationship between object area extraction and object area area fluctuation | variation. It is a processing flowchart of an object classification reliability calculation part. It is a figure explaining calculation of the object identification reliability in the situation where an object hides in a shield. It is explanatory drawing of the state which calculated the object classification reliability and stored in the object information storage part. It is an example of a screen displayed on a display part. It is a search result list by a search part. It is a processing flow figure by a search part. It is a thumbnail creation list for storing object information to be displayed in a thumbnail display area. It is a functional block diagram of the object search system which concerns on Example 2 of this invention. It is a functional block diagram of a certainty factor considering object type identification unit. It is explanatory drawing of the state of an object information storage part. It is explanatory drawing of the state of an object information storage part. It is state explanatory drawing of the object information storage part after tracking object comprehensive determination. It is explanatory drawing of the time change of the tracking object comprehensive determination score. It is a functional block diagram of the object search system which concerns on Example 3 of this invention.

Hereinafter, examples will be described with reference to the drawings.

  FIG. 1 is a functional block diagram of the object search system according to the first embodiment of the present invention. The object search system includes a video input unit 1, an object extraction unit 3, an object type identification unit 3, an object type certainty factor calculation unit 4, an object information storage unit 5, a search unit 6 and a display unit 7. The object extraction unit 3, the object type identification unit 2, the object type certainty factor calculation unit 4, and the search unit 6 are realized by a CPU, a memory such as a RAM and a ROM, and the CPU executes a program stored in the memory. By doing so, the processing described later is performed. In this embodiment, video data is assumed to be taken by a camera installed on an outdoor road, and an object to be detected from video data, that is, a person, a two-wheeled vehicle, and a vehicle will be described as an example. Is not limited to this. Details of each of the above parts constituting the object search system will be described below.

  The video input unit 1 inputs video data captured by a surveillance camera or the like. The video data is sequentially input in time series, and is input in order as an image data frame at time t, an image data frame at time t + 1, an image data frame at time t + 2,. The video data may be input directly from the camera, or the video data read from the buffer may be input after being recorded in a buffer (not shown).

  The object information storage unit 5 is a memory that stores information related to an object extracted by processing to be described later with respect to image data frames constituting input video data.

  An example of data items stored in the object information storage unit 5 will be described. FIG. 10 is an explanatory diagram of each data item stored in the object information storage unit. The time when the object is detected is stored in the time column, and the ID of the object detected at time t is stored in the object ID column. This number is unique by the combination of the time and the object ID. The object area coordinate field stores the coordinates of the object area extracted by the object extracting unit 3 described later. The position where the object is detected in the image data frame is stored by the x and y coordinates of the start point and end point of the circumscribed rectangle, for example. The feature amount is an area for storing a feature amount value calculated during processing of the object identification unit 2 described later. As an example, three types of feature amounts 1 to 3 are shown, but the number of feature amounts is not limited to this. The type column is an identification result of the type of the object identified by the object type identification unit 2. The type score column is a type score that indicates the type-likeness that is calculated when a determination is made using a classifier or the like during the identification process of the object type identification unit 2. The tracked object ID column is a tracked object ID that is uniquely assigned to each tracked object when it is determined that the object is the same object as a result of matching with an object at a past time. The tracked object type column is the object type as a result of comprehensively determining the object type from the object type determination result at the current time and the past object type determination result for the same object. Details of the calculation method of the data items in each column and the timing of storage will be described in order hereinafter.

  The object extraction unit 3 extracts an object region from the image data frame input from the video input unit 1. As a method for extracting an object from an image, any one of general methods as described in “Digital Image Processing” (CG-ARTS Association, 2004) supervised by the Digital Image Processing Editorial Committee is used. For example, a feature value related to background difference, inter-frame difference, or object, which is a method for extracting a moving region by taking a difference between images, is learned from a large number of samples, There is a method of extracting an area as an object area.

  In this embodiment, as an example, a case where an inter-frame difference method for extracting a moving region by taking a difference between successive image data frames in a video will be described as an example. For the extracted object area, the coordinates of the circumscribed rectangle are calculated and stored in the column of the object area coordinates in FIG.

  The object type identification unit 2 identifies the type of the object from one or more feature amounts calculated from the object region extracted by the object extraction unit 3.

  Next, a detailed configuration of the object type identification unit 2 will be described. FIG. 2 is a functional block diagram of the object type identification unit. The object type identification unit 2 is calculated by a feature amount calculation unit 21 that calculates a feature amount used to determine the object type of the object region, an identification reference storage unit 22 that stores a reference for identifying the object type, and a feature amount calculation unit 21. An object type determination unit 23 that compares the feature quantity with the identification standard stored in the identification reference storage unit 22 to determine the object type, an interframe tracking unit 24 that identifies the object detected at a past time, and a tracking process As a result, the object type comprehensive determination unit 25 that comprehensively determines the object type from the object type determination result at the current time and the past object type determination result. Information such as the determined object type is stored in the object information storage unit 5.

  The feature amount calculation unit 21 calculates a feature amount used for determining the object type of the object region. To determine the object type, one or more feature quantities effective for identifying the object to be identified are selected, and the object type is determined based on a numerical value obtained from the feature quantity.

  In the present embodiment, a case where three types of objects “car”, “motorcycle”, and “person” are identified will be described as an example. Features that identify these three types of objects include the number of horizontal edges, the number of circles (including ellipses), and matching between the current image data frame and the image data frame at the previous time (hereinafter referred to as the previous frame). Use the maximum correlation value. FIG. 3 is an explanatory diagram of the feature amount calculated by the feature amount calculation unit. A horizontal edge is a horizontal edge detected from an image. As shown in FIG. 3 (a), it is a feature quantity that can be detected in a large amount with respect to the car, such as the roof of the car, the upper and lower ends of the window, and the bumper. For example, as described in Japanese Patent Application Laid-Open No. 2008-299458, the horizontal edge can be calculated by setting a kernel that can extract the edge in the horizontal direction and applying a Sobel filter to the image.

  A circle or an ellipse is a characteristic amount specific to a car or a two-wheeled vehicle that can be detected in the circle of the wheel as seen in FIG. 3 (a) or FIG. 3 (b). This can be detected by a technique such as Hough transform as described in “Digital Image Processing” (CG-ARTS Association, 2004) p213 to p214 supervised by the Digital Image Processing Editorial Board.

The correlation value of matching with the previous frame is a feature amount regarding the degree to which the object area detected in a certain frame is similar to the object area of the previous frame. FIG. 3C shows an image data frame 38 at time t 38 and an image data frame at time t + 1. A region 3 a in the image data frame 38 is an object region extracted by the object extraction unit 3. A region 3b in the image data frame 39 at t + 1, which is the next time, is a region of the position at the time t + 1 of the object indicated by the region 3a. The region 3b is obtained, for example, by matching the region 3a with the region having the most similar image feature using the region 3a as a template. As a matching method, “Digital Image Processing” (CG-ARTS Association, 2004) supervised by the Digital Image Processing Editorial Board can be used, but the similarity or difference calculated here is used as the feature amount. To do. The similarity is large because the change is small when the object is a car or a two-wheeled vehicle, but the similarity is generally characterized by a small value when the object is a person. In the following example, the similarity is calculated for the entire object region, but in order to more accurately capture the characteristics of changes in foot movement, the similarity is calculated only for the lower half of the object region. May be.

  The process of the feature quantity calculation unit 21 will be described. FIG. 4 is an explanatory diagram of the feature amount extracted by the feature amount calculation unit 21 with respect to the image of the car. It is an example of the feature-value extracted by the feature-value calculation part 21 with respect to the image of the car of the time t. An image 41 is an image obtained as a result of extracting the object region 46 of the object a by the object extracting unit 3, and an image 42 is an image obtained as a result of extracting a horizontal edge as a feature amount. The image 43 is an image obtained as a result of extracting an ellipse as a feature amount, the images 44 and 45 are images at time t-1 and time t, respectively, and the object region 4e extracted at the time t is used as a template, and the previous time The region obtained as a result of matching with the image 44 of t-1 is the object region 4d. The feature amount calculation unit 21 digitizes the feature amount extracted by the above processing and stores it in the object information storage unit 5. FIG. 7 is a diagram illustrating an example of information stored in the object information storage unit. Feature quantity 1 is “number of horizontal edges”, feature quantity 2 is “number of circles”, and feature quantity 3 is “similarity of matching result with previous frame”. As the feature amount of the object a, horizontal edges 47, 48, 49, and 4a are detected from the image 42 of the horizontal edge feature amount extraction result of FIG. 4, and a numerical value “4” is stored in the feature amount 1 column. In the feature 2 column, ellipses 4b and 4c are detected and a numerical value “2” is stored. In the feature amount 3 column, “similarity of matching result with previous frame” is stored, and in the example shown in FIG. 4, an image 44 at time t−1 using the object region 4 e in the image 45 at time t as a template. The object region 4d at is matched. In the case of the object a, since the change between the image 44 and the image 45 is small, the similarity 0.96 between the object region 4e and the object region 4d is stored.

  FIG. 5 is an explanatory diagram of the feature amount extracted by the feature amount calculation unit for the bicycle image. The image 51 is an image obtained as a result of extracting the object region 56 of the object b by the object extracting unit 3, and the image 52 is an image obtained as a result of extracting the horizontal edge as a feature amount by the feature amount calculating unit 21. An image 53 is an image obtained as a result of extracting the ellipse as a feature amount by the feature amount calculation unit 21, and an image 54 and an image 55 are images at time t-1 and time t, respectively, and the object region 5a extracted at time t is The region obtained as a result of matching with the image 54 at the previous time t−1 as the template is the object region 59. A value stored in the object information storage unit illustrated in FIG. 7 for the object b will be described. In FIG. 5, since the horizontal edge is not detected in the object region 57 from the image 52 obtained by extracting the horizontal edge as the feature amount, “0” is stored in the feature amount 1 column. As the “number of circles” of the feature amount 2, “2” is stored in the feature amount 2 column because the circles 5 c and 5 d are detected from the object region 58 in the image 53 of the circle feature amount extraction result of FIG. The The “similarity of the matching result with the previous frame” of the feature amount 3 is the similarity with the region having the highest similarity as a result of matching on the image 54 using the object region 5a as a template. In the case of the object b, the change between the image 54 and the image 55 is small, but since there is some change due to the movement of the foot or the like, a similarity of 0.90 is calculated and stored in the feature amount 3 column.

  FIG. 6 is an explanatory diagram of a feature amount extracted by a feature amount calculation unit for a human image. The image 61 is an image obtained as a result of extracting the object region 66 of the object c by the object extracting unit 3. The image 62 is an image obtained as a result of extracting the horizontal edge as a feature value by the feature value calculating unit 21, and the image 63 is an image obtained as a result of extracting the ellipse as the feature value by the feature value calculating unit 21. Are images at time t-1 and time t, respectively. An object region 69 is a region obtained as a result of matching with the previous image 64 at time t-1, using the object region 6a extracted from the image 65 at time t as a template. The values stored in the object information storage unit 5 shown in FIG. 7 for the object c will be described. As the feature amount of the object c, “0” is stored in the feature amount 1 column because no horizontal edge is detected in the object region 67 from the image 62 of the horizontal edge feature amount extraction result of FIG. Since no circle is detected in the object region 68 in the image 63 of the circle feature quantity extraction result in FIG. 6, “0” is stored in the feature quantity 2 column. As a result of matching on the image 64 using the object region 6a in the image 65 as a template, the similarity 0.8 between the object region 6a and the object region 69 is calculated, and “0.8” is stored in the column of the feature amount 3 in FIG. The Since the movement of the foot is large between the object region 6a and the object region 69, the similarity is determined as a low value of 0.8.

  As described above, the feature amount calculation unit 21 calculates a feature amount for each object region extracted by the object extraction unit 3 and stores the value in the object information storage unit 5.

  In the present embodiment, the case where three types of feature quantities are used for identification of “person”, “two-wheeled vehicle (bicycle)”, and “car” has been described, but the present invention is not limited to this. Another feature amount is time-series fluctuation in the moving direction. The movement direction detected by the difference between image data frames is generally a constant direction in the case of a “two-wheeled vehicle”, whereas in the case of a “person”, the movement direction is various in the vertical and horizontal directions. . Therefore, if it is characterized by time-series fluctuations in the moving direction, it is effective for distinguishing between “people” and “two-wheeled vehicles”. Note that time-series fluctuations in the moving direction can be detected by, for example, calculating an optical flow between two consecutive frames and calculating an average value of the direction of the optical flow generated in the object region. The optical flow can be obtained by using the method described in “Digital Image Processing” (CG-ARTS Association, 2004), p243 to p245, supervised by the Digital Image Processing Editing Committee. This process is continuously performed in time series, and the value of the change in the direction of movement or the variance in the direction of movement is calculated. Can be identified as a “two-wheeled vehicle”.

  Next, the identification reference storage unit 22 in FIG. 2 will be described. The identification reference storage unit 22 stores a reference for determining the type of the object from the feature quantity of the object region. An example of identification criteria is shown in FIG. Value of feature value 1 (number of horizontal edges), feature value 2 (number of circles), feature value 3 (similarity of matching result with previous frame) for each of "car", "motorcycle", and "person" For example, “car” is judged as a car that satisfies each criterion that feature quantity 1 is greater than 2, feature quantity 2 is 1 or more, and feature quantity 3 is greater than 0.9. Will be. These threshold values are set by, for example, obtaining feature values from many image samples and obtaining them inductively.

  Next, the object type determination unit 23 in FIG. 2 will be described. The object type determination unit 23 determines the object type from the feature amount extracted by the feature amount calculation unit 21 based on the identification reference stored in the identification reference storage unit 22.

  FIG. 9 is a process flow diagram of the object type determination unit.

  The object type determination unit 23 reads the feature amount extracted by the feature amount calculation unit 21 from the object information storage unit 5 for the i-th object region among the object regions extracted by the object extraction unit 3 (step s91).

  It is determined whether or not the feature quantity of the object i satisfies the “car” identification criterion stored in the identification criterion storage unit 22 (step s92). If the feature amount of the object i satisfies the identification criterion of “car”, it is determined as “car”, “car” is stored in the type column shown in FIG. 10, and the process proceeds to step s99 (step s93). If not, the process proceeds to the next step s94.

  It is determined whether or not the feature amount of the object i satisfies the “two-wheeled vehicle” identification criterion stored in the identification criterion storage unit 2 (step s94). If the feature quantity of the object i satisfies the identification criterion of “two-wheeled vehicle”, it is determined as “two-wheeled vehicle”, “two-wheeled vehicle” is recorded in the type column shown in FIG. 10, and the process proceeds to step s99 (step s95). If not, the process proceeds to the next step s96.

  It is determined whether or not the feature amount of the object i satisfies the “person” identification criterion stored in the identification criterion storage unit 22 (step s96). If the feature quantity of the object i satisfies the identification criterion of “person”, it is determined as “person”, “person” is recorded in the type column in FIG. 10, and the process proceeds to step s98 (step s97). If not, the process proceeds to step s98.

  When the feature quantity of the object i does not satisfy any of the identification criteria of “car”, “two-wheeled vehicle”, and “person” stored in the identification criterion storage unit 22, the type of the object i is set to “unknown” for the processing of s99. move on.

  It is determined whether or not the processing for all detected objects i has been completed (step s99). If it is determined that the process has been completed, the process of the object type determination unit 23 is terminated. If it is determined that the process has not been completed, the process proceeds to step s9a.

  The object counter i is incremented by 1, and the process for the next object i + 1 is returned to step s91 to repeat the above process.

  Of the above steps, the contents of the determinations in step s92, step s94, and step s96 are based on the example in which the present embodiment uses three types of object types, “car”, “motorcycle”, and “person”. Depending on how to determine the classification of identification, it is not necessarily limited to this.

  Further, in the present embodiment, the identification criterion shown in FIG. 8 is used for the determination based on the threshold value for each feature amount. However, the present invention is not limited to this. For example, for each type of object to be identified, a large number of sample images are prepared in advance, a classifier is created by learning, and the object type determination unit 23 inputs the feature quantity of the object to the classifier. A method of determining the type based on the output result can be used. As the type of classifier, methods similar to these, such as discriminant analysis, SVM, and neural network, can be used. In this case, as an output of the discriminator, a score indicating the degree of uniqueness of the object type can be calculated, and this is also stored as object information.

  Next, a calculation method for the type score shown in FIG. 10 and object type identification using the type score will be described. FIG. 11 is a diagram illustrating an example of information stored in the object information storage unit. Here, a case where two types of “person” and “motorcycle” are identified as object types is taken as an example. For each of a plurality of object regions extracted by the object extraction hand unit 3, the value of the feature amount extracted by the feature amount calculation unit 21, the type score calculated from the feature amount, and the object type determined from the type score are shown. It is.

  The contents of feature quantity 1 (number of horizontal edges), feature quantity 2 (number of circles), and feature quantity 3 (similarity) are the same as those described in FIG. In order to identify “person” and “two-wheeled vehicle” from these feature quantities, in this example, it is assumed that they are identified using a discriminant function created by discriminant analysis. Discriminant analysis is a method for predicting a classification variable by a plurality of variables. The details of the method are described in, for example, p99-p118 in “Introduction to Multivariate Analysis” by Satoshi Nagata and Masahiko Munechika (Science Inc., 2001). For discrimination, a discrimination function is created in advance by the method described in the above document using a large amount of feature amount data of an object region whose object type is known. The classification variable calculated by the discriminant function is the type score in the example of FIG. 11, and the type can be determined by dividing the type score by a certain threshold.

  The discriminant function created with the variables of feature quantity 1, feature quantity 2 and feature quantity 3 shown in FIG.

  For each object region, the type score is calculated by substituting the feature value into the discriminant function of Equation 1. The value calculated by substituting the value of feature amount 1 into X1, the value of feature amount 2 into X2, and the value of feature amount 3 into X3 is shown in the type score column.

  FIG. 12 is an explanatory diagram of a determination method using a type score. The graph which plotted the type score on the vertical axis | shaft is shown. In FIG. 12, a broken line 351 is a discrimination threshold value of the discriminant function. When the classification score is larger than the threshold value, “two-wheeled vehicle” is determined, and when the classification score is smaller than the threshold value, “person” is determined. Based on the threshold value, the object a and the object b are determined to be “two-wheeled vehicle”, the object c and the object d are determined to be “person”, and the type determination result is stored in the type column of FIG.

  The discrimination method based on the above-described type score calculation is a method for obtaining a value of a particular type from one or a plurality of feature amounts of an object region. The type of “motorcycle”, “person”, etc. can be determined by the determination, and the degree of individuality can be quantitatively determined as the type score, which has the effect of increasing the amount of information when using the determination result .

  The above is a specific example of the classification score calculation method and the identification method using the classification score.

  The object type and type score determined by the processing of the object type determination unit 23 are stored in the “type” and “type score” fields of the object information storage unit 5 shown in FIG.

  Next, the inter-frame tracking unit 24 in FIG. 2 will be described. The inter-frame tracking unit 24 tracks whether or not the object extracted by the object extracting unit 3 in the image data frame at the current time t exists in the previous image data frame. This refers sequentially to the object information of the image data frame at the latest time from the object information storage unit 5, and determines whether or not they are the same object based on the similarity.

  As a determination method, for example, the object information at the current time t and the time t−1 is compared, and objects with small changes in size and close positions are compared as the same object from the object region information. Specifically, from the information of the “object region” stored in the object information storage unit 5, (1) the difference in the area of the object region is less than or equal to the threshold, (2) the distance of the object region is less than or equal to the threshold, When the above condition is satisfied, it can be determined that the similarity is high and the objects are the same. In this way, the same ID is assigned to the object determined by the inter-frame tracking means 24 as the same object at different times.

  FIG. 13 is a diagram for explaining processing by the inter-frame tracking unit. The image 121 is an image at time t + n−1, and the object region 122 is extracted by the object extraction unit 3. The object area 122 is an area having an area of 12960, with the upper left coordinates being (300, 122), the lower right coordinates being (380, 284), and the rectangular center being (340, 203). On the other hand, the image 123 is an image at the next time t + n, and the object region 124 is extracted. The object area 124 is an area having an area 13041 in which the upper left coordinates are (308,124), the lower right coordinates are (389,285), and the center of the rectangle is (348,204). For example, if the threshold value is set as follows, both the conditions are satisfied in this example, and therefore the object region 122 and the object region 124 are determined to be the same object. (1) Area difference condition threshold: Range from current area to half of current area (2) Distance between regions: Distance between object regions <100 Threshold is information such as object resolution Set appropriately.

  In this example, an example in which only the object region information obtained from the object information storage unit 5 is collated is used. However, a method of determining by threshold value determination of the correlation value obtained by matching the object region between images. There is also.

  FIG. 14 is a diagram illustrating a state of the object information storage unit after the inter-frame tracking process. As described in FIG. 13, since the object region 122 and the object region 124 are determined to be the same object, the tracking object ID at the time t + n−1 and the tracking object ID at the time t + n respectively have the same ID “ 2 "is stored.

  Next, the object type comprehensive determination unit 25 will be described. The object type comprehensive determination unit 25 is capable of comprehensively determining which type is the same even when different object type determinations are made between frames for the same tracking object.

  For the object type comprehensive determination unit 25, several realization methods are conceivable. As a first realization method, the type determination results of each of a plurality of image data frames are totaled, and the largest type is set as the type of the tracked object. In this case, the comprehensive determination result in the last frame for tracking the same tracking object is the type of the tracking object.

  The second realization method is a method of assigning the type of the object when the determination of the specific type continues and the preset threshold count is exceeded. According to this method, the type of the tracking object can be determined when the threshold value is exceeded. When this method is used, the determined object type is stored in the tracked object type field shown in FIG. 14 when the threshold value is exceeded, and the object type is not changed even if the tracking continues thereafter.

  As a third realization method, when the type score is calculated by the object type determination unit 23, the object type is determined based on the average value of the type scores of the same object. In this method, for each frame, the score of the same object before that frame is accumulated, and the classifier is determined using the averaged value. The object type of the determination result is displayed in the tracking object type field of FIG. To store. For this reason, as in the first implementation method, the comprehensive determination result in the last frame of tracking of the same tracking object is the type of the tracking object.

  As described above, the implementation method of the object type comprehensive determination unit 25 has been described. Regardless of which method is used, the final object type of each tracking object is the newest of the records with the same tracking object ID, in the tracking order. In other words, the type stored in the tracking object type of the last record is the type of the tracking object.

  Next, processing of the object type certainty factor calculation unit 4 will be described. The object type certainty factor calculation unit 4 calculates how accurately the object region is extracted using the feature amount of the object region. That is, the probability of the extracted object region is calculated as the object type certainty factor.

  In the object type identification unit 2 already described, the feature amount of the object region is extracted from the object region extracted by the object extraction unit 3 on the assumption that the entire object region is extracted, and the object type is determined. For this reason, if the object region cannot be extracted properly by the object extraction unit 3, the object type determination may not be performed correctly because the feature portion of the object of that type cannot be extracted.

  FIG. 15 is a diagram illustrating a state where an object is hidden by a shield during movement. Image 141 is an image data frame at time t, and image 144 is a continuous time of time t + 1. In each image, an object “person” is walking beyond the implantation 142 that is a shield. Therefore, as the object region, only the object region 143 and the object region 145 of the upper part from the trunk are visible because the foot portion is hidden. In this case, since the movement of the foot, which is a feature for identifying “person”, is hidden, the similarity of the matching result between the object region 143 and the object region 145 is considered to be higher than when the foot is seen accurately. As a result, the object type identification unit 2 may erroneously determine this object as a “two-wheeled vehicle”.

  As described above, when the object region cannot be accurately extracted, the reliability of the result of determining the object type from the feature amount of the object region tends to decrease. In order to obtain the reliability of the object type determination result, the object type certainty factor calculation unit 4 calculates the degree to which the object region can be accurately extracted from the information of the object region extracted by the object extraction unit 3.

  Since the object area with high confidence is a condition that allows obtaining sufficient features for object type determination, the object type confidence is whether or not the object area protrudes from the screen, and whether the area of the object area is sufficiently large. It is determined from the information on the object region whether the object region is likely to be shielded by the shield. Specifically, the object type certainty factor is calculated by the following equation 2.

Object type certainty factor = factor 1 coefficient x factor 2 coefficient x factor 3 coefficient x factor 4 coefficient… Formula 2
One or more factors that are considered to be factors that cause a low degree of confidence are determined from the feature quantities of the object area. If the confidence level is low, the coefficient is small, and if the confidence level is high, the coefficient is high. Calculate as follows. The degree to which the object region can be accurately extracted comprehensively is calculated by the product of them.

  There are the following as feature quantities of the object region that cause the certainty factor to be calculated.

Factor 1 is the feature quantity of the size of the object region. When the extracted object region is small, for example, when the object is far away in the image, it is difficult for features to appear in the image in the object region.
For this reason, the coefficient of factor 1 is set as shown in Equation 3 below according to the size of the object region extracted by the object extraction unit 3.

  If the area of the object region is larger than a predetermined threshold value, the coefficient is set to 1.0, and if it is less than that, the coefficient is decreased in proportion to the size, and the coefficient is decreased as the object region becomes smaller. If Area_th is greater than this, the size with which the feature amount of the object can be acquired is determined in consideration of the resolution.

  Factor 2 is a feature amount indicating whether or not the object region is in contact with the end of the screen data frame. FIG. 16 is a diagram illustrating a state where the object region is in contact with the end of the image data frame. In the image data frame 151, the object region 152 is in contact with the left end of the screen. In this case, only a part of the object in the object region 152 may be extracted as the object region. It is difficult to determine whether or not the entire object is visible with image processing technology. Therefore, when the object region is in contact with the screen edge, it is highly likely that a part of the object region is hidden, and the factor 2 factor is decreased. The factor 2 factor is set as shown in Equation 4 below.

  Factor 3 is the feature quantity of the change in the aspect ratio of the object area. Assuming that it moves in a certain direction, it can be said that the apparent area of the object region changes between frames, but the aspect ratio does not change greatly.

  As a factor for changing the aspect ratio, as shown in FIG. 15, a case where a part of the object is hidden by a shield while moving is conceivable. In this case, a change occurs in the aspect ratio of the object region between the object region where the object is present in the region without the shielding object and the object region where the object is present in the region with the shielding object. In addition, there are other cases where the aspect ratio of the object area changes, such as an environment where illumination is not sufficiently applied to the object, such as at night, or an environment where illumination is applied only to the front side in the moving direction of the object. Under such an illumination environment, as the area of the extracted object region is smaller, the entire image of the object existing in the object region is not extracted, and the aspect ratio of the object region changes. Therefore, it is possible to detect that the object region is not accurately extracted by detecting the change in the aspect ratio of the object region between the image data frames.

  Using this, the time-series variance of the value of the aspect ratio of the object area being tracked is calculated, and if this value is smaller than a predetermined threshold, the fluctuation of the aspect ratio is small and the certainty of the object area is high. The coefficient of factor 3 is increased, and if the ratio is greater than or equal to a predetermined threshold, the variation in aspect ratio is large and the factor of factor 3 is decreased because the certainty of the object region is low. The factor 3 factor is set using Equation 5 below.

  If the aspect ratio variance of the object region is smaller than a predetermined threshold, the coefficient is set to 1.0. If the variance is greater than a predetermined threshold, the coefficient is decreased in inverse proportion to the size, and the coefficient is decreased as the variance increases. Var_th is determined in consideration of the variance when the object region can be stably extracted in time series. The variance value is calculated by acquiring object region data for the latest N frames from the object information storage unit 5 with the current time t as a reference.

  Next, as factor 4, there is a feature amount of vertical fluctuation of the area of the object region. In the case described with reference to FIGS. 15 and 16, it can be confirmed whether or not the object region can be accurately extracted by detecting the area variation. FIG. 17 is a diagram illustrating the relationship between the object region extraction and the object region area variation with respect to the vertical variation of the object region area. A graph 171 in FIG. 17 is a graph showing a fluctuation value of the object region area when the object region extraction is stably performed. In the figure, the vertical axis represents the area of the object region, and the horizontal axis represents the time axis. The area variation 172 in the graph 171 has an increasing tendency in time series because the object is moving from the back side to the near side in the image. When the area variation value of the continuous period 174 is linearly approximated, the residual with the area value at each time from the approximate expression 173 is small and changes with a constant slope.

  On the other hand, a graph 175 in FIG. 17 is a graph showing a variation value of the object area area when the object area cannot be accurately extracted in a situation where the lighting environment is insufficient, such as the nighttime described above. In the figure, the vertical axis represents the area of the object region, and the horizontal axis represents the time axis. In the area variation 176, since the object moves from the back side to the near side in the image, the area tends to increase in time series. When the area fluctuation value of the continuous period 178 is linearly approximated, the residual of the area value at each time from the approximate expression 177 is large, and the fluctuation from the approximate expression 177 is large.

  In this manner, a linear approximation formula is obtained for the change in area, and the variation in area can be detected from the magnitude of the residual between the approximation formula and the actual area value. The confidence factor due to the area variation of factor 4 is set by the following formula 6.

  Similar to the above-described variance of the aspect ratio, the linear approximation formula is calculated by acquiring the area of the object region for the latest N frames from the object information storage unit 5 with the current time t as a reference. The residual with the approximate expression is calculated from the area data for N pieces and calculated by taking the average. Although the calculated average value may be determined as a threshold value as it is, the size of the residual is also affected by the size of the original area, so the calculated average value is divided by the area of the object region. It can be obtained more generally by normalizing to what percentage of the area of the object region.

  Here, a processing flow of the object type certainty factor calculation unit 4 will be described. FIG. 18 is a processing flowchart of the object type certainty factor calculation unit.

  The object type certainty factor calculation unit 4 resets the factor counter i to 1 of the first factor number (step s181). Next, a feature amount for obtaining the i-th factor is acquired. Aspect ratio history that can be calculated from the area of the object area for factor 1 as described above, the upper left and lower right coordinates of the object area for factor 2, and the upper left and lower right coordinates of the object area for factor 3. In the case of factor 4, the area history is acquired (step s182). The coefficient of the i-th factor is calculated using the acquired feature quantity for obtaining the i-th factor. Here, the coefficient of factor 1 is calculated based on the above formula 3, the coefficient of factor 2 is calculated based on the above formula 4, the coefficient of factor 3 is calculated based on the above formula 5, and the coefficient of factor 4 is calculated based on the above formula 6.

  It is determined whether steps s182 and s183 have been completed for all factors i (step s184). If completed, the process proceeds to step s186. If not completed, the process proceeds to step s185. The factor counter i is incremented by 1, and the process returns to step s182.

  When the calculation of the coefficients for all the factors i is completed, the object type certainty factor is calculated based on the above equation 2 (step s186).

  Next, a calculation method of the factor 1 to the factor 4 and a calculation method of the object type certainty factor will be described with specific examples.

  FIG. 19 is a diagram for explaining calculation of object identification certainty in a situation where an object is hidden by a shield. FIG. 20 is a diagram illustrating screens in three states in a series of videos in which an object appears and passes in the video. The image data frame 241 is an image when the object is passing behind the shielding object, the image data frame 242 is an image when the object is out of the shielding object area and is not blocked, and the image data frame 243 Is a screen in which the area of the foot is hidden because the object has advanced further than the state of the image 242 and has gone outside the screen. The calculation results of the factor 1, factor 2, factor 3, factor 4, and the object type certainty factor using the object region information from these three states and the calculation process will be described.

  FIG. 20 is an explanatory diagram of a state in which the object type certainty factor is calculated and stored in the object information storage unit. In FIG. 20, the time period from time t to time t + 2 is the time when the object passes behind the shielding object in FIG. 19, and the time period from time t + 3 to time t + 5 is the state in which the object in FIG. The band, from time t + 6 to time t + 8, corresponds to the time period in which a part of the object in FIG. 19 is out of the screen.

  In FIG. 20, the data items are time, object ID, type, tracking object region, aspect ratio, area, aspect ratio variance, linear approximation of area and residual average, factor 1 coefficient, factor 2 coefficient, factor 3 coefficient, factor 4 Each column of the coefficient and the object identification certainty factor is provided. The time of each image data frame is stored in the time column, the ID of the object detected in each image data frame is stored in the object ID column, and the object type identification unit 2 identifies the type column. In the tracked object ID column, tracked object IDs uniquely assigned to tracked objects determined to be the same object by the inter-frame tracking unit 24 are stored. In FIG. 20, the tracking object ID “3” is stored as the same object from time t to time t + 8.

  The upper left and lower right coordinates of the object area extracted by the object extracting unit 3 are stored in the object area column, and the aspect ratio of the object area calculated from the coordinate value of the object area is stored in the aspect ratio column. Has been. Specifically, it is calculated by (the width of the object region in the x direction) / (the height in the y direction). Further, the area column stores the area of the object region calculated from the coordinate value of the object region. In the column of the aspect ratio variance, the variance indicating the variation in the aspect ratio in the time direction obtained in the coefficient calculation process of the above factor 3 is stored. This is a value obtained by obtaining the variance of the aspect ratio values for the past N frames from the current time as the starting point for the same tracking object. Here, the variance is calculated among the past three frames. For example, the aspect ratio variance at time t + 2 is obtained as the variance of the aspect ratio values of the three data at time t, time t + 1, and time t + 2. Since the calculation requires tracking of three frames or more, the columns of the aspect ratio dispersion at time t and time t + 1 when tracking is less than three frames are blank (no value).

  The “average residual with area linear approximation” is the value of the average residual with the linear approximation of area, which shows the variation in area obtained in the process of calculating the above-mentioned factor 4 coefficient. This is an average of residuals between a value calculated by the approximate expression and the actual area, by calculating an approximate expression for the variation of the area for the past N frames starting from the current time for the same tracking object. In the example of FIG. 20, N = 3 frames, and calculation is performed between the past three frames. For example, the value at time t + 2 is obtained from three data of time t, time t + 1, and time t + 2. Since the calculation requires tracking of 3 frames or more, the columns at time t and time t + 1 when tracking is less than 3 frames are blank (no value).

  Factor 1, factor 2, factor 3, and factor 4 are calculated for Equation 3, Equation 4, Equation 5, and Equation 6, respectively, as described above.

  Below, a specific example is demonstrated about each factor coefficient. The factor 1 coefficient is calculated using Equation 3. If the threshold Area_th is 18000, the factor 1 coefficients at the time t and the time t + 1 among the objects being tracked shown in FIG. 20 are 0.833 and 0.856, respectively, and are smaller than the value 1 of the factor 1 coefficient after the time t + 2. . This period is the state of the image data frame 241 in FIG. 19, and it can be confirmed that the area is small because the object is far away, and the factor 1 coefficient is small because the feature is difficult to extract.

  The factor 2 coefficient is calculated using Equation 4. Whether or not the object region is in contact with the end of the image data frame is determined using the upper left and lower right coordinates of the object region as described above. In FIG. 20, when the screen size is “640” in the x direction and “480” in the y direction, the start point x coordinate of the object area is “0”, the start point y coordinate is “0”, and the end point x coordinate is “639”. "When the end point y coordinate is" 479 ", it is determined that" the screen is in contact with the edge ". In the example of FIG. 20, since the end point y-coordinate value of the object area in the image data frame from time t + 6 to time t + 8 is “479”, the factor 2 coefficient of these object areas is “0.1” from Equation 4. . In FIG. 19, the period in which the factor 2 coefficient is 0.1 is a period in which the object region 247 protrudes from the lower end portion of the image data frame 243. Therefore, it can be confirmed that the factor 2 factor decreases.

  Factor 3 coefficients are calculated using Equation 5. Assuming that the aspect ratio variance threshold Var_th = “0.02”, the data of the object region in the image data frame at time t + 2 exceeds the threshold in the example of FIG. = 0.02 / 0.02785 = 0.718. This indicates that the past three values of the aspect ratio of the object area at time t + 2 have fluctuations greater than or equal to the threshold value. In FIG. 19, the period of time t + 2 is the case of the state of the image data frame 241. During this period, the aspect ratio may vary because the object passes behind the shield and the entire area is not visible. I can confirm.

  Factor 4 is calculated using Equation 6. Assuming that the residual average threshold value with respect to the linear approximation of the area is “16”, the data of the object region at time t + 2 in FIG. 20 exceeds the threshold value, so the factor 4 coefficient = 16 / 17.392 = 0.92. This indicates that the past three values of the area of the object region at time t + 2 have fluctuations other than monotonous increase / decrease. In FIG. 19, the image data frame at time t + 2 is the case of the image data frame 241. During this period, the object may pass behind the shield and the entire area may not be visible, so the area may vary. I can confirm.

  The object type certainty factor is obtained by accumulating the factor 4 coefficient from the factor 1 coefficient obtained in the above description using Equation 2, and stores the result in the object certainty factor column of the corresponding image data frame. . As described above, if there are multiple factors that decrease the certainty factor, it is considered that the certainty factor will decrease.Therefore, calculate the coefficient from the multiple factors that decrease the certainty factor of the object type. Ask for. In this embodiment, coefficients are calculated for factor 1 to factor 4 and the object type certainty factor is calculated as an integrated value thereof. However, the present invention is not limited to this, and the object type certainty factor may be calculated from only one factor coefficient. .

Next, the search unit 6 and the display unit 7 that displays the search results by the search unit 6 will be described.
FIG. 21 is an example of a screen displayed on the display unit. Date selection area 201 that can be selected and input as search conditions, time zone selection list area 202 that can be selected and input in the same manner, object type selection list area 203, "search" button 204 that accepts a selection instruction, and thumbnail display area that displays search results The screen is constituted by 205 and an “Other object type identification confirmation” button 209 that receives a confirmation instruction of other object types. The thumbnail display area 205 is provided with windows 206 and 207 for displaying object information of search results, and a display priority area 208 for receiving display priority instructions.

  The date selection area 201 displays a calendar, and a user can input a plurality of dates as selection conditions. The time zone selection list area 202 displays a plurality of time zones in a pull-down manner, and the user can input a plurality of time zones as selection conditions. The object type selection list area 203 displays a plurality of object types in a pull-down manner, and the user can input a plurality of object types as selection conditions. FIG. 22 shows a state in which the date “2013/2/6”, the time zone “16: 00-17: 00”, and the object type “two-wheeled vehicle” are selected as selection conditions. In this state, when receiving an instruction input to the “search” button 204 by the user, the search unit 6 searches the object information storage unit 5 for information corresponding to the search condition and extracts it.

  FIG. 22 is a search result list by the search unit, and FIG. 23 is a processing flow diagram by the search unit. FIG. 22 shows a search result in which the search unit 6 extracts records corresponding to the date, time zone, and object type input as selection conditions from the object information storage unit 5. Here, when searching for the object type, the search unit 6 refers to the tracking object type of the record with the newest time among the records with the same tracking object ID, and extracts information on the tracking object type that matches the set object type. To do.

  Here, in the window 206 and window 207 in the thumbnail display area 205 shown in FIG. 21, the date, time, object type, object type certainty factor, and object image are displayed as thumbnails as search results. As a result, the user can more easily grasp the object information of the candidate search results.

  FIG. 23 is a process flow diagram of the search unit. A process in which the search unit 6 displays search result candidates in the above-described window 206 and window 207 will be described with reference to FIG.

  First, the search unit 6 sets a tracking object processing counter to 0 in order to sequentially process the tracking object information (step s221). The appearance date and time of the i-th tracking object are acquired (step s222). Specifically, the search result list shown in FIG. 22 is referred to in order from the top, and since the first tracking object ID is “2”, information of the tracking object ID “2” is acquired. As the time when the tracking object ID “2” appears, the appearance date “2013/2/6” and the appearance time “16: 40: 0001” are acquired.

  Next, the maximum value of the object type certainty factor of the tracking object i, the object type, date, and time are acquired (step s223). Specifically, with reference to the records of the tracking object ID [2] shown in FIG. 22 in order, both the object type certainty factors at time “16: 40: 0002” and time “16: 40: 0003” are “1”. So get one of the records. When there are records having the same certainty value as described above, it is desirable to select, for example, a record having a large area of the object region in order to select an image that can easily capture the feature of the object.

  Next, the record acquired in step 223 is stored in a thumbnail creation list to be described later (step s224), and the tracking object processing counter is advanced by 1 (step s225). It is determined whether or not all the tracking objects have been processed. If completed, the process ends. If not, the process returns to step s221 (step s226). In the case of the search result list shown in FIG. 22, since the tracking object ID is “3” at time “16: 55: 0088”, the process returns to step s226 and the same processing is repeated.

  FIG. 24 is a thumbnail list for storing object information to be displayed in the thumbnail display area. The thumbnail list stores the appearance date, the appearance time, and the time with the highest object identification certainty for the specific tracking object, the thumbnail date and the thumbnail time together with the object type and the object type certainty factor. Here, by the steps s222, s223, and s224 described in FIG. 23, the appearance date “2013/2/6”, the appearance time “16: 40: 0001”, the thumbnail date “ 2013/2/6 ”, thumbnail time“ 16: 40: 0003 ”, object type“ two-wheeled vehicle ”, and object type certainty factor“ 1.0 ”are stored. For the tracking object ID “3”, the appearance date “2013/2/6”, the appearance time “16: 55: 0088”, the thumbnail date “2013/2/6”, and the thumbnail time “16: 55: 0088” The object type “two-wheeled vehicle” and the object type certainty factor “0.6” are stored.

  After creating the thumbnail creation list shown in FIG. 24, the search unit 6 refers to this thumbnail creation list and corresponds to the window 206 and window 207 in FIG. 22 with the thumbnail date, thumbnail time, object type, and object type certainty factor, respectively. The image data frame to be displayed is displayed.

  Further, when the search unit 6 accepts selection of the display priority condition from the display priority area 208 in FIG. 22 by the user, the search unit 6 sorts the thumbnail creation list in the order of the selected items. When “in order of high certainty” is selected, the thumbnail creation list shown in FIG. 23 is rearranged in descending order of object type certainty and displayed in the windows 206 and 207 in the thumbnail display area 205. Further, when the user selects a specific window in the thumbnail display area 205, the user can refer to the thumbnail creation list corresponding to the window and reproduce the video from the appearance date and time of the object, thereby allowing the user to An image including the selected object can be referred to.

  According to the present embodiment, it is possible to efficiently search for a specific type of object from the recorded video data. For example, as described above, it is often difficult to confirm the characteristics of an object on an image when the lighting conditions are poor such as at night or when there is an obstacle. If the information is simply referred to in the order of time, information that cannot be obtained in the end is also referred to in order, and time is wasted until reaching the effective information. On the other hand, when referring to the object type certainty degree order, since the certainty degree is high and the object region can be referred to in the order of the objects that are accurately extracted, it can be expected that the information can be referred to in the order in which the effective information is obtained visually.

  When displaying in order of object type certainty factor, a check box is displayed beside each window in the thumbnail display area 205 in FIG. 21 to prevent reference omission, and a check is made when the user confirms. Then, a configuration may be adopted in which a “confirmed flag” or the like is set in the object information case unit 5. With such a configuration, it is possible to search only unconfirmed object information when searching by the search unit 6.

  FIG. 25 is a functional block diagram of an object search system according to the second embodiment of the present invention. FIG. 26 is a functional block diagram of the certainty-considered object type identification unit. The difference from the first embodiment is that instead of the object type identifying unit 2, a certainty factor considering object type identifying unit 9 for identifying the object type in consideration of the object type certainty factor is provided. The same components as those of the first embodiment are denoted by the same reference numerals, and the description of the same components as those of the first embodiment will be omitted below.

  In this embodiment, the certainty-considered object type identification unit 9 identifies an object in consideration of the object type certainty factor calculated by the object type certainty factor calculation unit 4. Specifically, when the object type comprehensive determination unit 25 in FIG. 26 comprehensively determines the object type from the object type determination result for each image data frame, the calculation is performed in consideration of the degree of certainty.

  In FIG. 19, an image data frame 241 is an image while the object is passing behind the shielding object, and an image data frame 242 appears on the screen with the object leaving the shielding object area and being unobstructed. The image data frame 243 in a moving state is an image in which the area of the foot is hidden because the object has advanced further than the state of the image data frame 242 and has gone out of the screen. This series of images is input, and processing is executed by the feature amount calculation unit 21, the object type determination unit 23, and the inter-frame tracking unit 24 as in the first embodiment.

  The processing of the object type comprehensive determination unit 25 based on the processing result by the interframe tracking unit 24 and the object type certainty factor by the object type certainty factor calculation unit 4, which is a feature of the present embodiment, will be described below. 27 and 28 are explanatory diagrams of the state of the object information storage unit, FIG. 29 is an explanatory diagram of the state of the object information storage unit after the tracking object comprehensive determination, and FIG. 30 is a graph showing temporal changes in the picture tracking object comprehensive determination score. It is explanatory drawing. FIGS. 27 to 28 also show the tracking object type determination results when the object type belief is not taken into account in order to make the features of this embodiment easier to understand.

  As in the first embodiment, an example in which the certainty factor is considered for each of the three types of realization methods of the object type comprehensive determination unit 25 will be described.

  The first implementation method is a method in which the object type determination results for each of a plurality of frames are totaled and the most frequently used type is set as the type of the tracked object. The certainty-considered object type identification unit 9 excludes frames whose object type certainty is below a certain threshold from the totalization because the certainty is low.

  27, the image data frame from time t to time t + 2 corresponds to the period of the image data frame 241 in FIG. 19, and the image data frame from time t + 3 to time t + 5 is the period of the image data frame 242 in FIG. It corresponds to. The image data frame from time t + 6 to time t + 8 corresponds to the period of the image data frame 243 in FIG. As a processing result of the inter-frame tracking unit 24, the object extracted for the image data frame from the time t to the time t + 8 is in a state where the same tracking object ID “3” is stored as the same object. Here, the object type comprehensive determination unit 25 sequentially counts the types determined and stored in the type column for the same tracking object for each image data frame.

  When counting without considering the object type certainty factor, the counting results at the time t + 5 are the same number of “motorcycles” and “people” three times each, but from the time t + 6 to the time t + 8 The total number of “motorcycles” will be 6 times and will be superior. Therefore, in the column of the tracking object type (without certainty factor consideration), “two-wheeled vehicle” is stored as a determination result from time t to time t + 8.

  On the other hand, if the certainty threshold is set to 0.5, for example, and the tracking object type is determined in consideration of the certainty, the image data frame from time t to time t + 2 has a value smaller than the threshold. This is because the image data frame from time t to time t + 2 corresponds to the period of the image data frame 241 in FIG. That is, since a part of the extracted object region is hidden by the shield, the object type certainty factor is a low value. The object type totaling unit 25 reads out the object types for each image data frame stored in the type column and totals them for each object type, but at the same time refers to the value of the certainty factor column to determine the certainty threshold. If it is smaller than the threshold value as compared with the value 0.5, the aggregation processing is skipped and the processing at the next time is started. As a result of this processing, the certainty factors at time t, time t + 1, and time t + 2 are “0.1”, “0.2”, and “0.3”, respectively, and are below the threshold value of 0.5, so they are not reflected in the aggregation. As a result of judgment, “-” (no judgment) is stored in the column of the tracking object type (with certainty factor taken into account).

  Next, at time t + 3, time t + 4, and time t + 5, the object type comprehensive determination 25 compares the value of the certainty factor column with a threshold value as described above. Since the threshold value is exceeded as a result of the comparison, the three types of time determination are totaled. At time t + 3, the total value for each type is person “1”, motorcycle “0”, and car “0”, so the overall judgment at this point is “person”. At time t + 4, the total value for each type is person “2”, the motorcycle “0”, and the car “0”, so the overall determination at this point is “person”. At time t + 5, the total value for each type is person “3”, the motorcycle “0”, and the car “0”, so the comprehensive determination at this point is “person”.

  Next, at time t + 6, time t + 7, and time t + 8, the object type comprehensive determination unit 25 similarly compares the value in the reliability column with a threshold value. Since the result of the comparison is below the threshold value, these three time type determinations are not reflected in the aggregation. Therefore, the total result does not change between time t + 6, time t + 7, and time t + 8, so that the comprehensive determination at the time t + 8 is finally “person”.

  As described above, the object type certainty factor is considered, and when the certainty factor is less than or equal to a predetermined threshold, the totaling is skipped. If the object region extraction state is bad, the comprehensive determination is suspended and the extraction state is good. A reliable comprehensive judgment can be carried out in some cases. Even in the above example, if the certainty factor is not considered, a person is erroneously determined as a two-wheeled vehicle, but the erroneous determination can be prevented by considering the certainty factor. Thereby, the accuracy of the comprehensive determination of the type of the tracking object can be improved.

  Next, the second realization method of the object type comprehensive determination unit 5 described in the first embodiment will be described with respect to an example in which the certainty factor is considered.

  The second realization method is a method of assigning the type of the object when the determination of the specific type continues and the preset threshold count is exceeded. In the certainty-considered object type identification unit 9, as in the first method of realization, frames whose reliability is equal to or lower than a predetermined threshold are excluded from aggregation because the reliability is low.

  A specific example will be described. Here, the threshold value of the continuous number is 3 times. When counting the number of consecutive times of the same type without considering the certainty factor, the object types stored in the type column are called in order of time, and the number of consecutive times of the same type is counted. In the image data frame at time t, since the type determination is “two-wheeled vehicle”, the number of consecutive two-wheeled vehicles at this time is “1”, and since the type determination of the image data frame at time t + 1 is “two-wheeled vehicle”, Is `` 2 '', the type determination of the image data frame at time t + 2 is `` two-wheeled vehicle '', so the number of consecutive two-wheeled vehicles at this time is `` 3 '', and at this point, the type of the tracking object is determined as `` two-wheeled vehicle '' "Two-wheeled vehicle" is stored in the column of the tracking object type (without considering the certainty factor). All subsequent tracking object IDs “3” are “two-wheeled vehicles”.

  On the other hand, the process of the object type comprehensive determination unit 25 will be described by taking as an example a case where the certainty threshold is set to 0.5, for example. The object type comprehensive determination unit 25 reads the object types for each image data frame stored in the type column in order of time, and counts the number of consecutive times when the type matches the previous time. The value is referred to and compared with the certainty threshold 0.5, and if it is smaller than the threshold, the continuous count is skipped and the process proceeds to the next time. Since the certainty levels at time t, time t + 1, and time t + 2 are all lower than the threshold value 0.5, the object type comprehensive determination unit 25 skips the continuous count processing at these times, and the object information storage unit “-” (No determination) is stored in each of the columns of the five tracking object types (with consideration for the depth of purchase).

  Next, since the type of time t + 3 is “person” and the value of the object identification certainty column is “1” and the threshold value is 0.5 or more, the number of continuous humans at this time is set to “1”. Since the type at time t + 4 is “person” and the object type certainty factor is “1” and the threshold value is 0.5 or more, the continuous number of people at this time is set to “2”. Since the type at time t + 5 is “person” and the object type certainty factor is “1” and the threshold value is 0.5 or more, the number of consecutive people at this time is “3”. “Person” is confirmed, and “Person” is stored in the column of the tracking object type (consideration of certainty factor) from time t + 3 to time t + 5. All subsequent tracking object IDs “3” are “people”.

  As described above, even in the second implementation method, the object type certainty factor is considered, and when the certainty factor is equal to or less than the predetermined threshold, the process of skipping the continuous count is performed, so that when the object region extraction state is poor The determination is suspended, and a reliable comprehensive determination can be performed when the extraction state is good. Even in the above example, if the certainty factor is not considered, a person is erroneously determined as a two-wheeled vehicle, but the erroneous determination can be prevented by considering the certainty factor. Thereby, the accuracy of the comprehensive determination of the type of the tracking object can be improved.

  Next, the third realization method of the object type comprehensive determination unit 25 described in the first embodiment will be described with respect to an example in which the certainty factor is considered. The third realization method is a method of determining the object type based on the average value of the type scores of the same object when the type score is calculated by the object type determination unit 23. In this method, for each image data frame, the type score of the same object before that frame is accumulated, the type is determined by the classifier using the averaged value, and the object type of the determination result is displayed in the tracking object type field. It is a method to store in. In the certainty-considered object type identification unit 9, when accumulating the type scores and taking the average value, the average is obtained with a certainty weight.

  A specific example will be described. In FIG. 29, the type score column stores the type score of the object region at that time calculated by the object type determination unit 23 from the feature amount. Here, it is assumed that the object type score is a score for discriminating between “person” and “two-wheeled vehicle”, and the identification rule is 0.0 as a threshold value, “motorcycle” is set as a threshold value, and “person” is set as a lower value. To do. The type column stores the result of the type determination based on the identification rule based on the type score. In addition, for comparison of whether or not certainty is taken into consideration, the tracking object type determined from the tracking object comprehensive determination score, which is the comprehensive determination result when the certainty factor is not considered, and the tracking object comprehensive determination score, as in Example 1. Are respectively stored, a tracking object comprehensive determination score (without considering certainty factor) column and a tracking object type (without considering certainty factor) column. On the other hand, the tracking object comprehensive determination score that is the comprehensive determination result when the certainty factor is considered, and the tracking object type that is determined from the tracking object comprehensive determination score are stored, respectively, and the tracking object comprehensive determination score (with certainty factor being considered) ) Column and a tracked object type column (with certainty factor taken into account).

  When the certainty factor is not considered, the average of the type scores for each time is taken as described above. For example, the tracking object comprehensive determination score of the image data frame at the time t + 1 is the type score at the time t and the time t + 1. The average of (1 + 1.3) ÷ 2 = 1.15. The tracking object comprehensive determination score at time t + 2 is the average of the type scores of time t, time t + 1 and time t + 2, (1 + 1.3 + 0.5) ÷ 3 = 0.933333, etc. Thus, the calculation is performed by taking the average of the type scores of the tracked objects up to that time. The tracked object comprehensive determination score calculated in this way is stored in the tracked object comprehensive determination score (without considering certainty factor). The result identified from this score by the identification rule (identified as “person” and “motorcycle” with a threshold value of 0.0) is stored in the column of the tracking object type (without considering certainty factor).

  On the other hand, a specific example in the case of considering the object type certainty factor will be described. When the certainty factor is considered, the weight of the object type certainty factor is multiplied by the type score and averaged as follows, for example.

  That is, smoothing is performed with emphasis on the score at a time with a high certainty factor.

  For this reason, for example, the tracking object comprehensive determination score at time t + 1 takes the weighted average of object type certainty at time t and time t + 1 (0.1 * 1 + 0.2 * 1.3) / (0.1 + 0.2) = 1.2, the tracking object comprehensive judgment score at time t + 2 is the average of the object types at time t, time t + 1 and time t + 2 (0.1 * 1 + 0.2 * 1.3 + 0.3 * 0.5) ÷ ( For example, 0.1 + 0.2 + 0.3) = 0.85, and so on, by calculating the object type certainty weighted average of the type scores of the tracking data up to that time. The tracking object comprehensive determination score calculated in this way is stored in the tracking object comprehensive determination score (with certainty factor taken into account). The result identified from the score by the identification rule (identified as “person” and “motorcycle” with a threshold value of 0.0) is stored in the column of the tracking object type (with certainty factor taken into account).

  FIG. 30 is a plot of the type score, the tracking object comprehensive determination score (without considering the certainty factor), and the tracking object comprehensive determination score (with the certainty factor considered) in FIG. 29 on a time-series graph for comparison. The horizontal axis represents time, the broken line 301 is the type score of FIG. 29, the broken line 302 is the tracking object comprehensive determination score (without considering the certainty factor), and the solid line 303 is the tracking object comprehensive determination score (with certainty factor being considered). Further, a solid line 304 in the figure is a threshold (0.0) for identifying “person” and “two-wheeled vehicle”. If the threshold is exceeded, the “two-wheeled vehicle” type is set.

  Based on the classification score 301 for each time, based on the identification rule, the “motorcycle” type is determined from time t to time t + 2, “people” from time t + 3 to t + 5, and from time t + 6 to t Between +8, the type is determined as “motorcycle”. On the other hand, in the case of the tracking object comprehensive determination score 302 that does not consider the certainty factor, the value is smoothed by taking the average of the scores, and the fluctuation does not become large like the value for each time of the broken line 301. "At the time t + 8, the value finally becomes a slightly positive value (0.0111) due to the influence of many positive value scores to be determined. Finally, this tracking object is erroneously determined as" two-wheeled vehicle ". On the other hand, in the case of the tracking object comprehensive determination score 303 in which the certainty factor is considered, since the ratio of the scores of times t + 3, t + 4, and t + 5 having high certainty factor and high weight becomes high, the type score 301 is correct. At time t + 6, t + 7, t + 8, the value does not shift to a positive value, and as a result, at time t + 8, it becomes a negative value (-1.161). A positive determination is made.

  As described above, in the third implementation method, when the object type determination unit 23 performs identification determination based on the type score, the type determination is performed after averaging the type score with the weight of the object type certainty factor. There is an effect that comprehensive tracking object determination based on a high type score can be performed.

  According to the present embodiment, the type of the tracking object is determined in consideration of the object type certainty factor calculated based on the extraction condition of the object region. Therefore, the object region can be satisfactorily extracted and the feature amount can be appropriately acquired. The determination process can be performed using the conditions preferentially, and the object type determination accuracy can be improved.

  As for the display of the result after determination, even if the object type certainty factor is displayed in each of the windows 206 and 206 in the thumbnail display area 205 as in the first embodiment, the date and time, the time, the object type, and the object area are also displayed. An image may be displayed.

  As described above, in the first embodiment and the second embodiment, the case where the calculated object type certainty factor is stored in the object information storage unit 5 has been described as an example. However, the present invention is not limited to this. You may store in the storage part. In this case, the object type certainty factor and other information can be linked.

  FIG. 31 is a functional block diagram of an object search system according to Embodiment 3 of the present invention. The same components as those in the first and second embodiments are denoted by the same reference numerals.

  The object search system according to the present embodiment includes a road network 311, one or a plurality of video shooting devices 312 installed at a plurality of locations on the road network 311, and an image recognition process using video shot by the video shooting device 312 as input. The image recognition unit 313, the image captured by the image capturing device 312 and the communication network 315 that transmits information of the result processed by the image recognition unit 313, the image transmitted by the communication network 315, and the image recognition unit 313. A server 318 that stores information on the processing results and video data from the video photographing device 312, a communication network 316 that transmits information stored in the server 318, and an electronic terminal such as a PC that can refer to the information via the communication network 316 Composed.

  When a large number of video photographing devices 312 are installed, a large amount of video and image recognition result information is acquired, and it is difficult to transmit all of them in real time. For this reason, the server 318 and the electronic terminal 317 are connected to the plurality of video photographing devices 312 and the image recognition unit 313 via the communication networks 315 and 316.

  The image recognition unit 313 includes a video input unit 1, an object type identification unit 2, an object extraction unit 3, an object type certainty factor calculation unit 4, and an object information storage unit 5, and stores information stored in the object information storage unit 5. The transmission control unit 314 enables transmission to the communication network 315. The video input unit 1, the object type identification unit 2, the object extraction unit 3, the object type certainty factor calculation unit 4, and the object information storage unit 5 constituting the image recognition unit 313 perform the same operations as those in the first embodiment. Therefore, the description is omitted.

  The transmission control unit 314 has a function of preferentially transmitting information having a high object type certainty factor when transmitting information stored in the object information storage unit 5 to the communication network 315. That is, the transmission priority can be set according to the object type certainty factor.

  The server 318 receives information stored in the object information storage unit 5 transmitted from the transmission control unit 314 of the image recognition unit 313 via the communication network 315 and stores the information from the object information storage unit 5 ′ and the video imaging device 312. A video storage unit 319 for receiving and storing transmitted video data via the communication network 315 is provided.

  The electronic terminal 317 includes a search unit 6 and a display unit 7. As described in the first embodiment, the user can specify desired conditions from the date selection area 201, the time zone selection list area 202, and the object type selection list area 203 in which each search condition can be input. The search unit 6 accesses the server 318 via the communication network 316, executes the processing described in the first embodiment, and outputs the search result to the display unit 7.

  According to the present embodiment, even when a large number of video photographing devices are installed on a road network and video data is acquired in real time, information on an object that is determined to have a good photographing state by the object type certainty factor Therefore, it is possible to realize an efficient and effective object search system.

  In the present embodiment, the road network has been described as an example. However, the present invention is not limited to this, and the present invention can also be applied to a system including monitoring cameras that monitor a plurality of locations in a building.

  In addition, this invention is not limited to an above-described Example, Various modifications are included. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, it is possible to add, delete, and replace the configurations of other embodiments with respect to a part of the configurations of the embodiments.

1 Video input section
2 Object type identification part
3 Object extraction unit
4 Object type certainty calculation unit
5 Object information storage
6 Search part
7 Display
21 Feature amount calculator
22 Identification criteria storage
23 Object type determination unit
24-frame tracking unit
25 Object type comprehensive judgment section
9 Certainty-considered object type identification unit
312 Video equipment
317 electronic terminal
318 server

Claims (16)

A video input unit for inputting video data;
An object extraction unit for extracting an object region from an image data frame constituting input video data;
An object type identifying unit for identifying an object type in the object area based on one or more first feature values of the image in the object area;
An object type certainty factor calculating unit that obtains an object type certainty factor based on one or more second feature quantities of the extracted object region;
An object information storage unit that stores the object type and the object type certainty factor in association with each image data frame;
An object search system comprising: a search unit that searches an image data frame and an object type certainty factor corresponding to an object type input as a search condition from the object information storage unit and outputs them to a display unit. The object search system according to claim 1,
The object type certainty factor calculation unit includes at least the area of the extracted object region, the position of the extracted object region in the image data frame, and the extracted object region in a plurality of consecutive image data frames. An object search system characterized in that the object type certainty factor is obtained using any one of variance of aspect ratio and change in area of an object region as the second feature amount. The object search system according to claim 1,
The object type identification unit includes a feature amount calculation unit that obtains the first feature amount from an image in the object region, and an identification reference storage unit that previously stores a threshold value of the first feature amount for each object type And an object type determination unit that compares the feature amount obtained by the feature amount calculation unit with a threshold value stored in the identification reference storage unit and determines an object type. The object search system according to claim 2,
The object type certainty factor calculation unit compares the area of the extracted object region with a predetermined threshold value and, when the threshold value is equal to or less than the threshold value, a value obtained by dividing the area of the object region by the threshold value Object search system with object type identification confidence. The object search system according to claim 2,
The object type certainty factor calculation unit compares the variance of the aspect ratio of the object region in a plurality of continuous image data frames with a predetermined threshold value, and if the threshold value is larger than the threshold value, sets the threshold value to the object An object search system in which a value obtained by dividing an area variance is used as the object type certainty factor. The object search system according to claim 3,
The object type identification unit extracts an image data frame having an object region similar to the extracted object region with reference to the object information storage unit, and sets a tracking object ID indicating the same object as the object information It has an inter-data frame tracking unit that outputs to the storage unit,
The display unit displays, in a predetermined area on the screen, an image data frame and an object type certainty factor corresponding to the output from the search unit, and a plurality of tracking object IDs indicating the same object An object search system characterized by displaying an image data frame. The object search system according to claim 6,
The display unit displays on the screen a display priority area that enables selection input of at least one of the order of high object type certainty and the order of low object type certainty,
The search unit outputs an image data frame and an object type certainty factor corresponding to a display priority selected and input to the display unit. A video input unit for inputting video data;
An object extraction unit for extracting an object region from an image data frame constituting input video data;
An object type certainty factor calculating unit that obtains an object type certainty factor based on one or more first characteristics of the extracted object region;
An object type identification unit for identifying an object type in the object area based on one or more second feature values of the image in the object area and the obtained object type certainty factor;
An object information storage unit that stores the object type and the object type certainty factor in association with each image data frame;
An object search system comprising: a search unit that searches an image data frame corresponding to an object type input as a search condition from the object information storage unit and outputs the image data frame to a display unit. The object search system according to claim 8,
The object type certainty factor calculation unit includes at least the area of the extracted object region, the position of the extracted object region in the image data frame, and the extracted object region in a plurality of consecutive image data frames. An object search system characterized in that the object type certainty factor is obtained using any one of variance of aspect ratio and change in area of an object region as the first feature amount. The object search system according to claim 9,
The object type identification unit includes a feature amount calculation unit that obtains the second feature amount from an image in the object region, and an identification reference storage unit that previously stores a threshold value of the second feature amount for each object type And an object type determination unit that compares the feature amount obtained by the feature amount calculation unit with a threshold value stored in the identification reference storage unit and determines an object type. The object search system according to claim 9,
The object type certainty factor calculation unit compares the area of the extracted object region with a predetermined threshold value and, when the threshold value is equal to or less than the threshold value, a value obtained by dividing the area of the object region by the threshold value Object search system with object type identification confidence. The object search system according to claim 9,
The object type certainty factor calculation unit compares the variance of the aspect ratio of the object region in a plurality of continuous image data frames with a predetermined threshold value, and if the threshold value is larger than the threshold value, sets the threshold value to the object An object search system in which a value obtained by dividing an area variance is used as the object type certainty factor. The object search system according to claim 10,
The object type identification unit extracts an image data frame having an object region similar to the extracted object region with reference to the object information storage unit, and sets a tracking object ID indicating the same object as the object information An object search system comprising an inter-data frame tracking unit for outputting to a storage unit. A plurality of image processing devices including an imaging camera and an image recognition unit that processes video data input from the imaging camera;
A server connected to the plurality of image processing apparatuses via a communication network;
An electronic terminal connected to the server via a communication network;
The image recognition unit includes an object extraction unit that extracts an object region from an image data frame that constitutes input video data, and an object region based on one or more first feature values of an image in the object region. An object type identifying unit for identifying the object type, an object type certainty calculating unit for obtaining an object type certainty factor based on one or more second feature quantities of the extracted object region, and each image data frame The object information storage unit that stores the object type and the object type certainty factor in association with each other, the image data frame corresponding to the object type certainty factor, the object type, and the object type certainty factor through the communication network A transmission control unit for sending to the server,
The server stores the image data frame transmitted from the image processing apparatus in association with the object type and the object type certainty factor,
The electronic terminal has a search unit that searches for an image data frame corresponding to an object type input as a search condition and an object type certainty factor from at least the server and outputs them to a display unit. The object search system according to claim 14,
The transmission control unit transmits the image data frame, the object type, and the object type certainty factor to the server in descending order of object identification certainty values stored in the object information storage unit. The object search system according to claim 15,
The object type certainty factor calculation unit includes at least the area of the extracted object region, the position of the extracted object region in the image data frame, and the extracted object region in a plurality of consecutive image data frames. An object search system characterized in that the object type certainty factor is obtained using any one of variance of aspect ratio and change in area of an object region as the second feature amount.

Images