JP4449483B2 - Image analysis apparatus, image analysis method, and computer program - Google Patents

Description

  The present invention relates to an image analysis apparatus, an image analysis method, and a computer program that execute an object motion identification process based on image data. More specifically, for example, a feature portion such as a human face region is extracted from an image captured by a camera, and collation with a rule configured by simple data such as position data of the feature portion is performed, and the movement of the subject is determined. The present invention relates to an image analysis device to be identified, an image analysis method, and a computer program.

  For example, an indexed video data can be created by setting an index corresponding to a time point such as a topic change in a meeting based on shooting data such as a meeting.

  As a method for automatically setting such an index, for example, a speech analysis based on audio data, a human face extraction based on image data, or a motion detection method can be considered. However, for example, in order to determine the time of topic change by speech analysis based on speech data, there is a problem that enormous dictionary data and complicated analysis are required, and high-accuracy speech input is essential. .

  From this point of view, in recent years, much development has been made on a technique for efficiently assigning an index based on image data, not audio data. For example, an operation occurring in a meeting is identified based on image data. For example, by adding an index to the meeting video data, it is possible to improve the efficiency when browsing or searching the video data at a later date. For example, Non-Patent Document 1 discloses a camera arrangement configuration for acquiring captured images without duplication in a meeting room, and a processing configuration for acquiring image data useful for image analysis with less influence of changes in brightness. Is described.

As a prior art disclosing the operation identification processing based on an image, for example, there is a technique described in Patent Document 1. Japanese Patent Application Laid-Open No. 2004-228561 discloses a technique for analyzing a three-dimensional movement of a subject by photographing the subject from different directions with a plurality of cameras and analyzing the plurality of images. Japanese Patent Application Laid-Open No. 2004-228561 discloses a configuration for detecting the motion of the person with high accuracy by detecting the movement of the head and the direction of the line of sight based on the captured image of the person's head.
I. Mikic, K. Huang, and M. Trivedi, "Activity Monitoring and Summarization for an Intelligent Meeting Room," in proc.IEEE Workshop on Human Motion, Austin, Texas, Dec. 2000. JP 10-334270 A JP 2000-163196 A

  However, all of the above-described image data analysis processes shown in the prior art require analysis by an extremely complicated algorithm based on photographed image data, and are not efficient processes. There are problems such as necessity of construction of the system, high cost, and heavy processing load. For example, the process described in Patent Document 1 calculates the direction and intensity of movement for each pixel in a plurality of two-dimensional images captured from a plurality of imaging angles, and the calculated intensity of movement is a predetermined value. A procedure for identifying the above regions, associating the identified specific regions with a plurality of two-dimensional images, and calculating the three-dimensional position of the specific region and the three-dimensional movement of the specific region, This requires a process called a procedure for estimating which part of the imaging target corresponds to the movement of the specific area.

  The technique described in Patent Document 2 extracts edge information from a face area of a captured image, estimates the position of an eye on a human face based on the edge information, and generates a grayscale image in the estimated image. Then, the process of calculating the line-of-sight direction by analyzing the position of the black eye in the eye, and executing the process of determining that there was nodding when the amount of motion is equal to or higher than the threshold in the vertical direction of the eye position. There is a problem that many processing steps such as edge image generation, eye position calculation, and gray image generation are required.

  The present invention provides an image analysis apparatus, an image analysis method, and a computer program that can solve such problems and efficiently analyze the motion of a subject by simple image data analysis. For the purpose.

  In more detail, a specific area such as a human face area is extracted from the image captured by the camera, and the movement of the subject is identified by collating with a rule composed of simple data such as position data of the specific area. An image analysis apparatus, an image analysis method, and a computer program are provided.

  A first aspect of the present invention is an image analysis apparatus, which includes a feature extraction unit that extracts image features from input image data, a storage unit that stores a plurality of pieces of image mode definition information related to image features, and each input image data An image mode definition information that matches an image feature is selected from the storage unit, a definition information matching unit that sets the selection information as definition information corresponding to input image data, and the definition information matching unit is associated with input image data. And an image identification processing unit that identifies an image mode of a subject including the image feature based on the definition information.

  According to this configuration, the image feature is extracted from the input image data, and the image feature is compared with a plurality of image mode definition information related to the image feature to analyze the subject information of the image data. It is not necessary to perform matching of the entire image, and the posture and motion of the subject can be determined by simplified and efficient processing.

  Furthermore, in an embodiment of the image analysis apparatus of the present invention, the image mode definition information is configured as simple rule description information defining the image feature.

  According to this configuration, it is easy to add new image mode definition information, and it is possible to perform image analysis by setting definition information according to various subject postures and actions.

  Furthermore, in one embodiment of the image analysis apparatus of the present invention, the input image data is an image sequence according to a time series constituting a moving image.

  According to this configuration, it is possible to analyze the motion of the subject along the time axis, such as the motion of raising the subject and the motion of raising the hand, by analyzing the image sequence according to the time series constituting the moving image.

  Furthermore, in an embodiment of the image analysis device of the present invention, the image analysis device further includes a motion detection unit that detects a motion of a subject from an image frame constituting an image sequence according to a time series constituting a moving image. The feature extraction unit is configured to execute a process of extracting an area in which motion is detected between frames in the motion detection unit as an image feature region, and the rule matching unit is configured to detect an image in which motion is detected. The image frame including the feature region is configured to execute the association process with the image mode definition information.

  According to this configuration, only the region in which the motion is detected is extracted as a feature region, and the subject information of the image is obtained by collating the display mode of the feature region with the image mode definition information defining various feature region modes. Therefore, the load corresponding to the processing of data other than the moving part can be reduced, and the movement of the subject can be determined efficiently.

  Furthermore, in an embodiment of the image analysis apparatus of the present invention, the image mode definition information is simple rule description information defining an image feature, and includes position information of a specific area, aspect ratio information of a specific area, a specific area Size information, center-of-gravity position information of specific areas, distance information between specific areas, and definition data that defines at least one of these conditions.

  According to this configuration, since the configuration is such that the object information of the image frame is analyzed by collating with a simple rule that defines position information, aspect ratio information, size information, etc. as image mode definition information. It is possible to determine the movement of the subject.

  Furthermore, in an embodiment of the image analysis apparatus of the present invention, the image mode definition information is information configured by a conditional expression that defines the mode of the image feature.

  According to this configuration, since the image feature is compared with conditional expressions such as position information and size information to analyze subject information of the image, it is possible to efficiently determine the motion of the subject.

  Furthermore, in an embodiment of the image analysis apparatus of the present invention, the image identification processing unit includes image mode definition information associated with a plurality of image frames according to a time series constituting a moving image in the rule matching unit. The present invention is characterized in that the subject motion determination is executed based on time-series sequence data.

  According to this configuration, since the operation determination is performed based on the transition of the display mode of the continuous specific areas of the plurality of image frames, it is possible to accurately analyze the motion of the subject information even in the moving image.

  Furthermore, in an embodiment of the image analysis apparatus of the present invention, the feature extraction unit is configured to execute feature extraction from input image data based on color discrimination processing of image data.

  According to this configuration, feature extraction from an input image is executed by color discrimination processing of image data, so that feature extraction with few errors can be performed without using a system such as a dedicated face analysis device.

  Furthermore, in one embodiment of the image analysis apparatus of the present invention, a motion detection unit that extracts a portion estimated as a human skin (skin) region by color discrimination of input image data and detects a motion of a subject from an image frame Based on the information from the above, an area in which movement between frames is detected is extracted as a feature area, and the rule matching unit matches the image mode definition information that matches the image mode of the skin area that is the feature area in the input image data The selection information is set as image mode definition information corresponding to an image frame, and the image identification processing unit is configured to select the feature region based on the image mode definition information associated with the image frame in the rule matching unit. It is the structure which performs the attitude | position or motion identification process of the person which uses as a component.

  According to this configuration, image aspect definition information in which a skin area such as a person's face or hand is selected and extracted from the input image as a specific area, and the display form of the skin area in the image and various feature areas are defined. Therefore, it is possible to easily and accurately determine a person's hand-raising operation, standing operation, seating operation, and the like.

  Furthermore, a second aspect of the present invention provides a feature extraction unit that extracts features from input image data indicating a subject, a storage unit that stores definition information that defines image feature modes, input image data, and the definition information And a collation identifying unit that identifies a subject based on the definition information.

  According to this configuration, the image feature is extracted from the input image data, and the image feature is compared with a plurality of image mode definition information related to the image feature to analyze the subject information of the image data. It is not necessary to perform matching of the entire image, and the posture and motion of the subject can be determined by simplified and efficient processing.

  Furthermore, in an embodiment of the image analysis apparatus of the present invention, the feature extraction unit is configured to extract a feature element indicating at least a part of the subject as the feature.

  According to this configuration, subject identification based on at least some of the features of the subject is executed, and subject posture and motion determination is reliably executed.

  Furthermore, in one embodiment of the image analysis apparatus of the present invention, the image feature mode includes a feature element indicating at least a part of the feature of the subject.

  According to this configuration, subject identification based on at least some of the features of the subject is executed, and subject posture and motion determination is reliably executed.

  Furthermore, in one embodiment of the image analysis apparatus of the present invention, the feature element indicates a region on the image.

  According to this configuration, for example, identification of a subject based on a region on an image such as a person's face or hand is executed, and the posture or motion determination of the subject is reliably executed.

  Furthermore, in an embodiment of the image analysis apparatus of the present invention, the definition information is configured as simple rule description information defining the image feature.

  According to this configuration, it is easy to add new image mode definition information, and it is possible to perform image analysis by setting definition information according to various subject postures and actions.

  Furthermore, in one embodiment of the image analysis apparatus of the present invention, the input image data is an image sequence according to a time series constituting a moving image.

  According to this configuration, it is possible to analyze the motion of the subject along the time axis, such as the motion of raising the subject and the motion of raising the hand, by analyzing the image sequence according to the time series constituting the moving image.

  Furthermore, in an embodiment of the image analysis device of the present invention, the image analysis device further includes a motion detection unit that detects a motion of a subject from an image frame constituting an image sequence according to a time series constituting a moving image. The feature extraction unit is configured to execute a process of extracting a region in which motion is detected between frames in the motion detection unit as a feature region, and the collation identification unit is a feature region in which motion is detected It is the structure which performs the matching process with the image mode definition information about the image frame containing this.

  According to this configuration, only the region in which the motion is detected is extracted as a feature region, and the subject information of the image is obtained by collating the display mode of the feature region with the image mode definition information defining various feature region modes. Therefore, the load corresponding to the processing of data other than the moving part can be reduced, and the movement of the subject can be determined efficiently.

  Furthermore, in one embodiment of the image analysis apparatus of the present invention, the image mode definition information is simple rule description information defining the feature, and includes position information of a specific area, aspect ratio information of a specific area, a specific area Size information, center-of-gravity position information of specific areas, distance information between specific areas, and definition data that defines at least one of these conditions.

  According to this configuration, since the configuration is such that the object information of the image frame is analyzed by collating with a simple rule that defines position information, aspect ratio information, size information, etc. as image mode definition information. It is possible to determine the movement of the subject.

  Furthermore, in an embodiment of the image analysis apparatus of the present invention, the image mode definition information is information configured by a conditional expression defining the feature mode.

  According to this configuration, since the image feature is compared with conditional expressions such as position information and size information to analyze subject information of the image, it is possible to efficiently determine the motion of the subject.

  Furthermore, in one embodiment of the image analysis apparatus of the present invention, the collation identifying unit is based on time-series sequence data of image aspect definition information associated with a plurality of image frames according to a time series constituting a moving image. In this case, the movement determination of the subject is executed.

  According to this configuration, since the operation determination is performed based on the transition of the display mode of the continuous specific areas of the plurality of image frames, it is possible to accurately analyze the motion of the subject information even in the moving image.

  Furthermore, in an embodiment of the image analysis apparatus of the present invention, the feature extraction unit is configured to execute feature extraction from input image data based on color discrimination processing of image data.

  According to this configuration, feature extraction from an input image is executed by color discrimination processing of image data, so that feature extraction with few errors can be performed without using a system such as a dedicated face analysis device.

  Furthermore, in the third aspect of the present invention, a feature extraction unit that extracts a feature element that indicates a feature of a subject from input image data that indicates the subject, and a condition condition of the feature element that indicates the feature of the subject on the image are defined. An image analysis apparatus comprising: a storage unit that stores definition information; and a collation identifying unit that collates input image data with the definition information and identifies a subject based on a condition of the definition information. .

  According to this configuration, the image feature is extracted from the input image data, and the image feature is compared with a plurality of image mode definition information related to the image feature to analyze the subject information of the image data. It is not necessary to perform matching of the entire image, and the posture and motion of the subject can be determined by simplified and efficient processing.

  Furthermore, in one embodiment of the image analysis device of the present invention, the definition information indicates at least one of a size of a feature element, a relationship between the feature elements, and a condition of the position of the feature element, and the verification identification The unit is characterized in that, when the condition is satisfied, the input image data is identified as a subject indicating the definition content of the image.

  According to this configuration, it is easy to add new image mode definition information, and it is possible to perform image analysis by setting definition information according to various subject postures and actions.

  Furthermore, the further side surface of this invention exists in the image analysis method and computer program which perform the process corresponding to the above-mentioned image analysis apparatus.

  According to this configuration, the image feature is extracted from the input image data, and the image feature is compared with a plurality of image mode definition information related to the image feature to analyze the subject information of the image data. There is provided a method and a computer program capable of determining the posture and motion of a subject by simplified and efficient processing without performing matching of the entire image.

  Note that the computer program of the present invention is a recording medium provided in a computer-readable format for a computer system capable of executing various program codes, for example, a recording medium such as a CD, FD, or MO. A computer program that can be provided by a medium or a communication medium such as a network. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the computer system.

  Other objects, features, and advantages of the present invention will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

  According to the configuration of the present invention, a specific image feature such as a human face or hand is extracted from input image data, for example, a condition of the image feature possessed by the input image data, and various image feature modes are defined. The image feature definition information consisting of rules described by formulas is collated, the rule that matches each image is selected and associated with the image data, and the image feature is determined based on the associated rule or its sequence. Since the posture and motion of a subject having a corresponding region as a constituent element, for example, a person with a face or hand, is determined, it is determined whether the image data satisfies the description data configured by the conditional expression. Thus, it is possible to accurately identify a subject such as a posture determination or a motion determination of a person by an efficient matching process.

  In the configuration of the present invention, the image mode definition information includes definition data that defines conditions such as position information of specific areas as image features, aspect ratio information, size information, barycentric position information, distance information between specific areas, For example, it is possible to perform rule association based on a process that is set as a conditional expression and determines whether or not the image mode satisfies these definition data, and is faster than the conventional method that performs matching between images. It becomes possible. In addition, since the image mode definition information is configured as descriptive data such as conditional expressions, it is easy to create new rules, and it is possible to efficiently add rules corresponding to various operations and apply them to operation determination. .

  Hereinafter, the details of an image analysis apparatus, an image analysis method, and a computer program according to the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing the configuration of the image analysis apparatus of the present invention. As shown in FIG. 1, the image analysis apparatus of the present invention inputs captured images of cameras 101 to 10n, for example, and performs motion detection based on the captured images. Note that the image data to be processed is not limited to an image directly input from the camera, but may be image data stored in advance in a storage unit.

  The image analysis apparatus according to the present invention includes, for example, a sampling unit 121 that inputs processing target image data that is captured images of the cameras 101 to 10n, an operation detection unit 122, a feature extraction unit 123, a definition information (rule) matching unit 124, and image identification. A processing unit 125, a definition information (rule) storage unit 126, and an action definition information storage unit 127 are included. The definition information (rule) verification unit 124 and the image identification processing unit 125 may be configured as an integrated verification identification unit.

  The sampling unit 121 inputs captured images of the cameras 101 to 10n or image data to be analyzed from a data storage unit (not shown). The image data to be processed is, for example, moving image data, and continuous frame image data. Here, the cameras 101 to 10n are, for example, cameras fixed in a certain meeting room, and are cameras that capture the participants of the meeting from different directions.

  The sampling unit executes a process of selecting an image for executing analysis in the subsequent motion detection unit 103 and the feature extraction unit 104 from the input image. In this sampling process, for example, a sampling process such as extracting data at equal intervals for selecting one frame every 20 frames is executed without executing an image analysis.

  Specifically, as shown in FIG. 1, when the image stream data input from the camera 101 is [S1], processing such as extracting an n1 frame every δ seconds is executed for the image stream [S1]. Similarly for the image stream data [S2] input from the camera 102, n2 frames are extracted every δ seconds. Thereafter, the same sampling process is executed for all the streams S1 to Sn.

  The sample frame image data extracted by the sampling unit 102 is input to the motion detection unit 103 and the feature extraction unit 104.

  Processing of the motion detection unit 103 will be described. Based on the sample frame image data extracted by the sampling unit 102, the motion detection unit 103 performs a motion extraction process for determining a subject area in motion by extracting a difference between a plurality of frames.

The motion detection unit 103 individually performs motion detection for each stream S _i . First, at time [t], the frame average [S _i avg _t ] of each stream is calculated according to the following equation (Equation 1) based on the sample frame data acquired up to time [t].
S _i avg _t = (f ₁ + f ₂ +... F _t−1 ) / (t−1) (Equation 1)

In the above formula, f _i + f _j represents the integrated value f _i (x, y) + f _j (x, y) of the corresponding pixel of each sample frame.

Next, the difference [ds _it ] between the sample frame data [f _t ] at time [t] and the frame average [S _i avg _t ] calculated based on the above formula (formula 1) is _expressed by the following formula (formula 2). calculate.
ds _it = S _i avg _t −f _t (Expression 2)

The above expression is an expression for calculating difference data between corresponding pixels of the image of the current frame [f _t ] at time t and the frame average [S _i avg _t ] calculated based on the above expression (Expression 1). Difference data for each pixel is calculated based on the above formula (Formula 2).

Next, edge extraction based on the difference data is executed to generate a change area mask. The edge extraction is performed by filtering processing, for example, applying a Gaussian smoothing filter to the difference data [ds _it ] calculated based on the above formula (Formula 2). A change area mask for discriminating the area is generated.

  In this change area mask, the set area of bit [1] is a pixel area in which a clear change has occurred between the average value of frames 1 to t-1 and the current frame (t). The set area of bit [0] is determined as an area that does not change, that is, an area that does not move.

  The binary image as the change area mask obtained in this way is subjected to Dilation processing for hole removal or the like to generate a binary mask. In the binary mask, the pixel value = black indicates the change area.

FIG. 2 schematically illustrates the processing of the motion detection unit 103 described above. For example, FIG. 2A shows images f _{1 to} f _t−1 of frames _{1 to t−1} , and when the image at the present time (t) is set to an image f _t shown in FIG. First, based on the images f _{1 to} f _t−1 shown in FIG. 2A, an image of the frame average [S _i avg _t ] is calculated based on the above equation (Equation 1).

Next, based on the frame average image, [S _i avg _t ], and the image f _t shown in FIG. 2B, the difference data [ds _it ] is calculated based on the above formula (formula 2), For the calculated difference data [ds _it ], for example, edge extraction is performed by a filtering process using a Gaussian smoothing filter, and a change area mask for determining a moving area is generated by threshold determination. By performing Dilation processing for hole removal or the like, a binary mask in which the black portion indicates the changed region as shown in FIG. 2C is generated.

  As shown in the figure, the binary mask is divided into n × m blocks, and a score for each block is set. The score is an integrated value of the change pixels included in each block, and the motion detection result is input to the feature extraction unit 123 and the definition information (rule) matching unit 124, and a specific area detection process in the feature extraction unit 123. The definition information (rule) matching unit 124 applies the matching process with the rule.

  Next, processing of the feature extraction unit 123 will be described. The feature extraction unit 123 extracts an image feature, for example, a specific region having a feature from the image frame. For example, if the intended subject posture or motion identification is intended to identify the posture or motion of a person, for example, the posture or motion of a person standing up, the sitting posture or motion, the posture or motion of a raised hand, etc. The face and hand skin (skin) regions are extracted as regions having image features. Note that what is extracted as an image feature can be set according to the purpose. Here, as an example, a process example of extracting a human face or a skin area of a hand as a specific area corresponding to an image feature will be described as an example for the purpose of detecting the posture or motion of a person.

  In this embodiment, color filtering is applied to feature extraction by the feature extraction unit 123. For example, human face extraction processing is not limited to such color filtering processing, but a special face detection device such as a document [K. Murai and S. Nakamura "Real Time Face Detection for Multimodal Speech Recognition", in proceedings. of ICME 2002, Vol.2, pp.373-376, 2002] can be applied. However, many face detection systems have problems such as a reduction in the accuracy of detection of faces that are not front-facing.

On the other hand, color-based skin detection based on simple color detection has the advantage that relatively accurate detection is possible even when there is a change in face direction or light environment, and detection errors are reduced. . The skin detection algorithm executed in this embodiment is, for example, the following document [A. Jaimes. Conceptual Structures and Computational Methods for Indexing and Organization of Visual Information, Ph.D. Thesis, Department of Electrical Engineering, Columbia University, February 2003]
Based on the algorithm described in.

An image feature extraction by the feature extraction unit 123, that is, a skin region extraction algorithm will be described. First, the pixel value data of each frame [f _i ] input from the sampling unit is developed into HSV color space coordinates. The HSV color space coordinates are three-dimensional coordinates of hue (H), saturation (S), and luminance (V). In the HSV space, a specific area corresponds to a skin color area.

Among the pixel value data of each frame [f _i ], a pixel corresponding to the color area of the skin (skin) in the HSV color space coordinates is determined as a skin image area, and other than the color area of the skin (skin) in the HSV color space coordinates. The pixel data to which it belongs is determined to be outside the skin area.

The above-described extraction of the skin region by color filtering is executed for each frame [f _i ] input from the sampling unit 121. However, even with this color filtering process, a region other than a skin (skin) region such as a wall or a table having a pixel value similar to a human skin (skin) may be determined as a skin (skin) region.

Therefore, the feature extraction unit 123 further executes the following processing. First, the skin regions extracted by the above color filtering are grouped. Set a set of adjacent pixels determined as the skin region as a group, detecting the bounding box S _bb as a boundary area set as the boundary. Note that areas smaller than a certain size are excluded.

Then, set the active bounding box A _bb comprising a contiguous area of the bounding box S _bb and bounding boxes S _bb set to the same center, or motion between frames is detected within this active bounding box A _bb not Determine whether.

  The presence / absence of motion can be determined based on difference information between the current frame and a frame that is several frames back in the past. Further, information in units of blocks of the binary mask described above with reference to FIG. 2, which is obtained based on the processing result of the motion detection unit 103, that is, score information corresponding to motion may be used. The non-moving area is excluded from the skin area. By this processing, it is determined that the region with movement is a specific region, that is, a skin region.

  FIG. 3 is a flowchart showing a specific area (skin area) extraction process sequence as an image feature extraction process executed by the feature extraction unit 123.

First, in step S101, image data of each frame input from the sampling unit 121 is developed in the HSV color space. In step S102, the bounding box _Sbb is set by grouping the partial areas determined as skin areas.

In step S103, an area smaller than a preset threshold is excluded. Next, in step S104, it sets the active bounding box _{A bb} including neighboring regions of the bounding box _{S bb.}

Next, in step S105, it is determined whether or not a motion is detected in the active bounding box _Abb , and the region where the motion is detected is finally extracted as a skin region, that is, a specific region.

  Through the above processing, the feature extraction unit 123 extracts image features based on color discrimination and motion discrimination, that is, extracts a specific region (skin region) from each frame image input from the sampling unit 121. This extraction result is input to the definition information (rule) matching unit 124.

  Next, details of processing executed in the definition information (rule) matching unit 124 will be described. In the image analysis apparatus of the present invention, image mode definition information such as a conditional expression describing layout information such as position information about a specific region as an image feature is set as domain knowledge, and an image is generated based on the image mode definition information. Identification of the posture or motion of the subject from the data, that is, action determination is executed. A subject to be an action identification target is a subject having a specific area as a constituent element, for example, a person having a skin area as a constituent element.

The items we observed in setting the image mode definition information as domain knowledge are, for example, the following items.
* Discrimination between a person and a non-person area: A non-person area not occupied by a person is a low activity area. The activity in such areas is mainly based on noise and changes in the light environment.
* Visual configuration: The physical configuration of, for example, the meeting room to be imaged is unchanged over all the captured images. For example, the ceiling is not an area occupied by a person, and the state of low activity continues.
* Person structure: Actions in meetings occur by person. A person also has a specific physical structure.

Based on such observation items, a rule including position information about a specific region indicating an image feature is generated. That is, a rule describing layout information as display mode information of a specific area in an image frame, for example, a conditional expression is generated. This rule is stored in the definition information (rule) storage unit 126 as image mode definition information. The definition information (rule) matching unit 124 uses the image mode definition information stored in the definition information (rule) storage unit 126 as a template to determine which rule (template) each sample image frame [f _i ] corresponds to. Do the attachment.

Note that the definition information (rule) matching unit 124 uses the sample image frame [f extracted by the sampling unit 121 as the image frame for matching with the image mode definition information stored in the definition information (rule) storage unit 126. It is possible to target only selected image frames including a specific region recognized as a skin region by processing of the motion detection unit 122 and the feature extraction unit 123, instead of all of _i ].

  The definition information (rule) matching unit 124 performs matching between an image frame including a specific region recognized as a skin region and the image mode definition information stored in the definition information (rule) storage unit 126 to obtain an image frame. Each of these is associated with image mode definition information.

  An example of the image mode definition information stored in the definition information (rule) accumulation unit 126 will be described with reference to FIG.

  The definition information (rule) storage unit 126 stores a large number of image mode definition information corresponding to templates that define various modes of image features. Each of them is set as condition data indicating image modes corresponding to various actions such as a predetermined action of a person, for example, a state where the hand is raised, a state where the person stands up.

  The example shown in FIG. 4 shows an example in which the image mode definition information (template) of [R00001] to [Rnnnn] is stored in the definition information (rule) storage unit 126. An example is shown in FIG. The image mode definition information has a configuration in which specific area information such as the position and size of the specific area as shown in FIG. 4A is set as a conditional expression of if and then.

  The conditions shown in FIG. 4 (a) correspond to the specific areas (area1, area2) shown in FIG. 4 (b).

The conditional expression shown in FIG.
if (7 <area1size <9)
and (1 <area2size <2)
and (1 <distance (area2, area1) <2)
and (1 <y-distance (area2, area1) <2)
then
Template [R00xxx] (Raising hand action)
The configuration of

  For example, when the image mode definition information stored in the definition information (rule) storage unit 126 is associated with the image frame shown in FIG. 4C, the definition information ( Rule) The image frame definition information [R00000] stored in the storage unit 126 is sequentially compared with an image frame, and a matching rule for the image frame definition information is selected as a matching rule for the image frame. Here, an example is shown in which image mode definition information [R00xxx] is selected.

  For example, in the image frame shown in FIG. 4C, the specific area indicating the image feature of the image frame, that is, the skin area, is the face partial area 201 and the hand partial area 202 shown in FIG. Corresponds to the area 1 (area1) and area 2 (area2) of the rule in FIG.

  In the image mode definition information shown in FIG. 4B, the size of the area 1 (area1) is 7 to 9, the size of the area 2 (area2) is 1 to 2, the area 1 (area1) and the area 2 (area2) The condition that the distance is 1 to 2 and the distance in the y direction between the area 1 (area1) and the area 2 (area2) is 1 to 2 is set. The image frame shown in FIG. Since the information condition is satisfied, the image frame shown in FIG. 4C is determined to be a frame corresponding to the image mode definition information, that is, the image mode definition information [R00201].

  As described above, the definition information (rule) collation unit 124 collates the image frame in which movement is recognized and determined as a skin area with the image mode definition information stored in the definition information (rule) storage unit 126. The image mode definition information that matches the display mode of the specific area identified in each of the image frames, that is, the skin (skin) area in this case, is determined as a rule corresponding to the image frame.

  The image mode definition information is not limited to the conditional expression in the if-then format as shown in FIG. 4A, for example, data that only describes information such as the size (size) and position of a specific area, or The definition information (rule) matching unit 124 may store the data as table data in a table format or the like, and may execute association processing based on these data.

The information defined as the image mode definition information is information indicating an image feature, for example, the following information regarding a specific region indicating the image feature.
Position information of specific area (eg coordinate data)
Aspect ratio information of specific area (aspect ratio)
Size information (area) of specific area
Center-of-gravity position information of a specific area Distance information between specific areas Image mode definition information is set as definition data that defines at least one of these conditions.

A specific processing example executed by the definition information (rule) matching unit 124 will be described. For example, when the image stream data acquired as the imaging data of the camera 101 is [S1] and the sample image frame extracted by the sampling unit is [f _i ], the motion detection unit 122 and the feature extraction unit 123 It is assumed that three image frames [P1], [P2], and [P3] are extracted by the processing, including a region in which movement is recognized and determined as a skin region.

The definition information (rule) collation unit 124 collates the image mode definition information stored in the definition information (rule) accumulation unit 126 for these three image frames, and performs image frames [P1] to [P1] to [ P3] is associated with image mode definition information that satisfies the display state of the specific area. For example,
Image frame [P1] → Image mode definition information [R00003]
Image frame [P2] → Image mode definition information [R00021]
Image frame [P3] → Image mode definition information [R00102]
Etc. are executed.

  Note that the image frames [P1], [P2], and [P3] are image frames arranged in accordance with the passage of time, that is, a sequence of image frames in which a moving skin area is detected. Note that the number of image frames to be associated with the rule can be any number from 1 to an arbitrary number.

  The image identification processing unit 125 executes an operation determination based on the image mode definition information sequence of the image frame associated with the definition information (rule) matching unit 124, that is, an action identification process. In the above example, the image mode definition information sequence is [R00003] → [R00021] → [R00102].

  For example, an execution example of motion identification based on two image frames [P1] and [P2] will be described with reference to FIG.

FIG. 5A shows an example identified as a person raising hand action. For example, in the definition information (rule) matching unit 124,
Image frame [P1] → Image mode definition information [R001]
Image frame [P2] → Image mode definition information [R002]
The rule sequence information [R001] → [R002] is input to the image identification processing unit 125.

The image identification processing unit 125 is based on the table data of the action definition information storage unit 127 that stores a table in which action information corresponding to various rule sequences is set.
Image mode definition information [R001] → Image mode definition information [R002]
Action information corresponding to the sequence is extracted.

  In the table data of the action definition information storage unit 127, the corresponding action of the image mode definition information [R001] → the image mode definition information [R002] is defined as a hand raising action, and the image identification processing unit 125 Conclude that the rule sequence is a raised hand action.

FIG. 5B shows an example identified as a person standing action. For example, in the definition information (rule) matching unit 124,
Image frame [P1] → Image mode definition information [R001]
Image frame [P2] → Image mode definition information [R003]
The rule sequence information [R001] → [R003] is input to the image identification processing unit 125.

The image identification processing unit 125 receives the action definition information storage unit 127 from
Image mode definition information [R001] → Image mode definition information [R003]
Action information corresponding to the sequence is extracted.

  In the table data of the action definition information storage unit 127, the corresponding action of the image mode definition information [R001] → the image mode definition information [R003] is defined as an upright action, and the image identification processing unit 125 Conclude that the rule sequence is a standing action.

  The table data of the action definition information storage unit 127 defines image mode definition information sequences corresponding to various actions. With reference to FIG. 6, the table data of the action definition information storage unit 127 will be described.

  FIG. 6 shows an example of an image mode definition information sequence corresponding to three actions of (a) raising hand action, (b) standing action, and (c) sitting action.

  (A) As the image mode definition information sequence corresponding to the raising hand action, there are, for example, FIGS. 6A1 to 6A6. These sequences are rule tables corresponding to the raising hand action, and the table of the action definition information storage unit 127 Set as data.

  The image mode definition information is described as data indicating conditions as described above with reference to FIG. 4A, and the sequence of identification information corresponding to these description data is the action definition information storage unit 127. Is set as table data. In FIG. 6, in order to facilitate understanding, the expression mode of the condition data described with reference to FIG.

For example, the sequence of FIG.
Image frame [P1]: Image mode definition information [R001] indicating a state in which the face area is located at the upper center and the hand area is located at the lower left and right.
Image frame [P2]: Image mode definition information [R002] indicating a state in which the face area is located at the upper center and the hand area is located at the upper left and lower right.
Image frame [P3]: Image mode definition information [R001] indicating a state in which the face area is located at the upper center and the hand area is located at the lower left and right
The sequence is shown.

That is, the table data of the action definition information storage unit 127 includes
The action corresponding to the sequence of image mode definition information [R001] → [R002] → [R001] is defined as a hand-raising action, and the image identification processing unit 125 performs image mode definition information [R001] → [R002. ] → [R001] is input from the definition information (rule) matching unit 124, it is determined that the action is a hand raising action.

  Similarly, all the image mode definition information sequences in FIGS. 6A1 to 6A6 are all defined as a hand raising action in the table data of the action definition information storage unit 127, and the image identification processing unit 125 performs rule matching. If the image mode definition information sequence input from 124 is one of these sequences, it is determined that a hand raising action has occurred.

FIG. 6B shows an image mode definition information sequence corresponding to the standing action.
For example, the sequence of FIG.
Image frame [P1]: Image mode definition information [R004] indicating a state where the face area is located at the lower center.
Image frame [P2]: Image mode definition information [R005] indicating a state where the face area is located at the upper center.
Image frame [P3]: Image mode definition information [R004] indicating a state in which the face area is located at the lower center.
The sequence is shown.

The table data of the action definition information storage unit 127 includes
The action corresponding to the sequence of image mode definition information [R004] → [R005] → [R004] is defined as an upright action, and the image identification processing unit 125 performs image mode definition information [R004] → [R005. ] → [R004] sequence information is input from the definition information (rule) matching unit 124, it is determined that an upright action has occurred.

FIG. 6C shows an image mode definition information sequence corresponding to the seating action.
For example, the sequence of FIG.
Image frame [P1]: Image mode definition information [R005] indicating that the face area is located at the upper center
Image frame [P2]: Image mode definition information [R004] indicating a state where the face area is located at the lower center.
Image frame [P3]: Image mode definition information [R005] indicating a state where the face area is located at the upper center.
The sequence is shown.

The table data of the action definition information storage unit 127 includes
The action corresponding to the sequence of image mode definition information [R005] → [R004] → [R005] is defined as a seating action, and the image identification processing unit 125 performs image mode definition information [R005] → [R004. ] → [R005] sequence information is input from the definition information (rule) matching unit 124, it is determined that a seating action has occurred.

  As described above, the definition information (rule) matching unit 124 stores the image modes stored in the definition information (rule) storage unit 126 for the image frames [P1] to [Pn] in which the moving skin region is detected. Collation with the definition information is executed to associate the image mode definition information [Rxxx] to [Ryyy] satisfying the display state of the specific area of the image frames [P1] to [Pn]. The image frames [P1] to [Pn] are arranged in time series, and the image mode definition information [Rxxx] to [Ryyy] corresponding to the image frames [P1] to [Pn] are also set as time series data. It is data.

  Next, the image identification processing unit 125 determines the action to be generated based on a table in which the image mode definition information sequence and the actions are associated with each other from the action definition information storage unit 127.

  That is, the input information from the definition information (rule) matching unit 124, that is, the rule sequence of the image mode definition information [Rxxx] to [Ryyy] corresponding to the image frames [P1] to [Pn] arranged in time series. The corresponding data entry is searched and extracted from the table acquired from the action definition information storage unit 127, and the action corresponding to the extracted entry is determined as the action identified from the image frames [P1] to [Pn].

  The processing of the definition information (rule) matching unit 124 and the processing procedure of the image identification processing unit 125 will be described with reference to FIGS.

  First, the processing procedure of the definition information (rule) matching unit 124 will be described with reference to FIG.

  First, in step S201, the definition information (rule) matching unit 124 selects, as the rule association processing target frame [Pi], one by one from the image frames [P1] to [Pn] set in time series.

  In step S202, image mode definition information [Rx] that matches the image frame [Pi] to be processed is selected, and a frame correspondence rule is determined.

  In step S203, it is determined whether or not the association of the image mode definition information with all the image frames [P1] to [Pn] set in time series is completed. If there is an unfinished image frame, step S201 is performed. The following process is repeated. When the association of the image mode definition information with all the image frames [P1] to [Pn] is completed, the process proceeds to step S204, and the rule sequence information [Rx] to [Ry] corresponding to the image frames [P1] to [Pn]. Is output to the image identification processing unit.

  Next, the processing procedure of the image identification processing unit 125 will be described with reference to FIG. In step S301, the image identification processing unit 125 inputs rule sequence information [Rx] to [Ry] corresponding to the image frames [P1] to [Pn] set in time series from the definition information (rule) matching unit 124. .

  Next, in step S302, a sequence that matches the rule sequence information [Rx] to [Ry] corresponding to the image frames [P1] to [Pn] is set based on the table stored in the action definition information storage unit 127. An entry is extracted, and in step S303, the operation information set in the extracted entry is determined as a generated action.

  Through the above processing, for example, the generation operation identification processing based on the image data photographed by the camera is executed. Based on the operation identified in this way, for example, indexing of video data is performed.

  As an actual process, an action identification process was performed using data obtained by photographing a meeting for one hour with four cameras from different directions. The number of frames taken by each camera was about 3000 frames, and a total of 12000 frames were acquired. The camera shooting data is MPEG-1 data and is composed of frame data of 352 × 240 pixels. First, one sample frame is extracted every second from the data stream photographed by the camera, the above-mentioned specific region, that is, the skin region detection and motion detection is performed on this sample frame, and an image frame having a moving skin region A process of associating the image with the image mode definition information was executed, and an operation determination based on the sequence was performed. As a result, efficient and accurate judgments about hand movements and general human movements were made.

In the image analysis apparatus and method of the present invention, for example, as shown in FIG. 4A, a rule defining a specific area is set, and a rule corresponding to each selected image is selected to determine an operation sequence. As shown in FIG. 4A, the rule defining the specific area is specific area information, that is,
Position information of specific area (eg coordinate data)
Aspect ratio information of specific area (aspect ratio)
Size information (area) of specific area
Center-of-gravity position information of a specific area Distance information between specific areas It is set as definition data that defines at least one of the conditions of these information, and its creation is extremely easy, and various new image mode definition information It is possible to generate image mode definition information that defines various operations such as handshake and nodding as well as the operations such as raising hand, standing up, and sitting as described in the embodiment. It becomes.

  Therefore, compared with the conventional method of performing template matching using, for example, image data itself as a template, creation of image mode definition information to be applied as a template is extremely simple, and various operation determinations can be made by generating image mode definition information. It becomes possible to execute efficiently. In addition, the image mode definition information is significantly reduced in data amount compared to the case where the image data itself is set as a template, and it is possible to achieve more efficient data processing, a reduced memory amount, and a smaller device. Become.

  Finally, a hardware configuration example of the above-described image analysis apparatus will be described with reference to FIG.

  A CPU (Central Processing Unit) 501 describes an execution sequence of each process of the OS (Operating System), the sampling process, the specific area detection process, the motion detection process, the rule matching process, and the action identification process described in the above embodiment. It is a control part which performs the process according to various computer programs.

  A ROM (Read Only Memory) 502 stores programs used by the CPU 501, calculation parameters, and the like. A RAM (Random Access Memory) 503 stores programs used in the execution of the CPU 501, parameters that change as appropriate during the execution, and the like. These are connected to each other by a host bus 504 including a CPU bus.

  The host bus 504 is connected to an external bus 506 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 505.

  A keyboard 508 and a pointing device 509 are input devices operated by the user. The display 510 includes a liquid crystal display device, a CRT (Cathode Ray Tube), or the like, and displays various types of information as text and images.

  An HDD (Hard Disk Drive) 511 includes a hard disk, drives the hard disk, and records or reproduces a program executed by the CPU 501 and information. The hard disk stores various computer programs such as a role as a storage unit of data stored in the definition information (rule) storage unit 126 and action definition information storage unit 127 shown in FIG. 1 and various data processing programs. .

  The drive 512 reads data or a program recorded in a removable recording medium 521 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and the data or program is read out from the interface 507 and the external bus 506. , And supplied to the RAM 503 connected via the bridge 505 and the host bus 504.

  The connection port 514 is a port for connecting the external connection device 522 and has a connection unit such as USB or IEEE1394. The connection port 514 is connected to the CPU 501 and the like via the interface 507, the external bus 506, the bridge 505, the host bus 504, and the like. A communication unit 515 is connected to a network and executes transmission of data supplied from the CPU 501 or the HDD 511 and data reception.

  Note that the hardware configuration example of the image analysis apparatus shown in FIG. 9 is an example of an apparatus configured by applying a PC, and the image analysis apparatus of the present invention is not limited to the configuration shown in FIG. Any configuration can be used as long as the processing described in the above item can be executed.

  The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims section described at the beginning should be considered.

  The series of processes described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run.

  For example, the program can be recorded in advance on a hard disk or ROM (Read Only Memory) as a recording medium. Alternatively, the program is temporarily or permanently stored on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, or a semiconductor memory. It can be stored (recorded). Such a removable recording medium can be provided as so-called package software.

  The program is installed on the computer from the removable recording medium as described above, or is wirelessly transferred from the download site to the computer, or is wired to the computer via a network such as a LAN (Local Area Network) or the Internet. The computer can receive the program transferred in this manner and install it on a recording medium such as a built-in hard disk.

  Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

  As described above, according to the configuration of the present invention, a specific image feature such as a human face or hand is extracted from input image data, and the feature mode of the input image data and various image feature modes are extracted. For example, the image mode definition information including rules described by a conditional expression or the like is defined, and a rule that matches each image is selected and associated with the image data. Based on the associated rule or sequence thereof Thus, the image data satisfies the description data constituted by the conditional expression because the posture and motion of a subject having a region corresponding to the image feature, for example, a person having a face or a hand, is determined. Thus, it is possible to accurately identify a subject such as a posture determination or a motion determination of a person by an efficient collation process for determining whether or not.

  In the configuration of the present invention, the image mode definition information includes definition data that defines conditions such as position information of specific areas as image features, aspect ratio information, size information, barycentric position information, distance information between specific areas, For example, it is possible to perform rule association based on a process that is set as a conditional expression and determines whether or not the image mode satisfies these definition data, and is faster than the conventional method that performs matching between images. It becomes possible. In addition, since the image mode definition information is configured as descriptive data such as conditional expressions, it is easy to create new rules, and it is possible to efficiently add rules corresponding to various operations and apply them to operation determination. .

It is a block diagram which shows the structure of the image analysis apparatus which concerns on this invention. It is a figure explaining the process which the motion detection part performs in the image analyzer which concerns on this invention. It is a figure which shows the flowchart explaining the process which the feature extraction part in the image analysis apparatus which concerns on this invention performs. It is a figure explaining the specific example of the process which the rule collation part performs in the image analysis apparatus which concerns on this invention, and the image mode definition information applied to this process. It is a figure explaining the process which the image identification process part in the image analysis apparatus which concerns on this invention performs. It is a figure explaining the operation | movement discrimination | determination process based on the information accumulate | stored in the action definition information storage part in the image analysis apparatus which concerns on this invention. It is a figure which shows the flowchart explaining the process which the rule collation part performs in the image analyzer which concerns on this invention. It is a figure which shows the flowchart explaining the process which the image identification process part in the image analyzer which concerns on this invention performs. It is a figure which shows the hardware structural example of the image analysis apparatus which concerns on this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101-10n Camera 121 Sampling part 122 Motion detection part 123 Feature extraction part 124 Rule collation part 125 Image identification process part 126 Rule storage part 127 Action definition information storage part 201 Face partial area 202 Hand partial area 501 CPU (Central processing Unit)
502 ROM (Read-Only-Memory)
503 RAM (Random Access Memory)
504 Host bus 505 Bridge 506 External bus 507 Interface 508 Keyboard 509 Pointing device 510 Display 511 HDD (Hard Disk Drive)
512 drive 514 connection port 515 communication unit 521 removable recording medium 522 external connection device

Claims (3)

An image analysis device,
Input multiple image frames that make up an image sequence according to time series, and set the skin area bounding box by grouping partial areas estimated as human skin (skin) areas by color discrimination of each image frame Then, an active bounding box including the skin area bounding box and its neighboring area is set , and motion between frames is detected based on information from the motion detection unit that detects the motion of the subject from the image frame. A feature extraction unit that extracts the skin region as a feature region when the region is in the active bounding box ;
A storage unit storing a plurality of image mode definition information characterized with respect to at least one of position information, aspect ratio information, size information, centroid position information, and distance information between the feature areas;
A definition information matching unit that selects image mode definition information that matches a feature region of each image frame extracted by the feature extraction unit from the storage unit, and sets the selection information as definition information of each image frame;
An operation mode identification processing unit for identifying an operation mode of a subject including the feature region based on definition information associated with each image frame in the definition information matching unit;
An image analysis apparatus comprising: The image mode definition information is:
The information configured by a conditional expression defined with respect to at least one of position information, aspect ratio information, size information, barycentric position information, and distance information between the characteristic areas. The image analysis apparatus according to 1. Computer
Input multiple image frames that make up an image sequence according to time series, and set the skin area bounding box by grouping partial areas estimated as human skin (skin) areas by color discrimination of each image frame Then, an active bounding box including the skin area bounding box and its neighboring area is set , and motion between frames is detected based on information from the motion detection unit that detects the motion of the subject from the image frame. A feature extraction unit that extracts the skin region as a feature region when the region is in the active bounding box ;
A storage unit storing a plurality of image mode definition information characterized with respect to at least one of position information, aspect ratio information, size information, centroid position information, and distance information between the feature areas;
A definition information matching unit that selects image mode definition information that matches a feature region of each image frame extracted by the feature extraction unit from the storage unit, and sets the selection information as definition information of each image frame;
A computer program for image analysis processing for causing the definition information matching unit to function as an operation mode identification processing unit that identifies an operation mode of a subject including the feature area based on definition information associated with each image frame.

Images