JP5718632B2 - Part recognition device, part recognition method, and part recognition program - Google Patents

Description

  The present invention relates to a part recognition device, a part recognition method, and a part recognition program, and in particular, a part recognition apparatus and part recognition for accurately recognizing a part of a person included in a video or an image obtained by photographing or the like. The present invention relates to a method and a site recognition program.

  Conventionally, cameras have been installed in surveillance areas such as banks, department stores, and convenience stores for reasons such as crime prevention. When a monitor or the like performs monitoring using a camera, the first method is to visually confirm a real-time or recorded image, and secondly to visually confirm only when a person is present by a person detection method or the like. By this visual confirmation, suspicious characteristics / suspicious behavior of the person appearing on the camera are found, and actions such as notifying or reporting to the police are performed.

  For example, many analog cameras using NTSC coaxial cables for image transmission are installed throughout the country. In many cases, a single camera is used to monitor one predetermined area.

  In recent years, there are methods for detecting parts such as a person and their hands from a camera image (see, for example, Patent Document 1 and Patent Document 2). In the technique disclosed in Patent Document 1, a compound eye camera is used to detect a face region and a region having a parallax close to the face and having a similar skin color value as an arm region, and then the detected arm region Recognize motion based on In the technique disclosed in Patent Document 1, a person (user) standing in front of a camera and a background area are separated using two cameras, and the head and the hand are recognized. Further, the technique disclosed in Patent Document 2 utilizes the fact that the hand region is pointed more than the head region in hand recognition.

JP 2010-123019 A JP 2009-211153 A

  Here, the method shown in Patent Document 1 described above solves the problem of frequent false detection due to skin color detection. For example, “when the hand is covered with gloves” or “the hand detects skin color” Cannot be detected with the skin color detection, and the hand cannot be detected with skin color detection and the face is covered. "If the color is close", "If the clothes are close to skin color", "If the hand cannot be detected by skin color detection and can be shaded on either the hand or the face, or if there is strong illumination", etc. If the head and hand colors are different, undetected / false positives may occur.

  Further, in the method shown in Patent Document 1, since two cameras or a compound eye camera is used, installation and processing costs are required. In the case of two cameras, the cameras are installed side by side. There are the above restrictions.

  Further, in the method shown in Patent Document 2, there are cases in which an operation such as forcing a marker to the subject, forcing to turn to the front, or forcing a hand forward or shaking is sometimes performed. Further, in the method shown in Patent Document 2, when detailed shape information (finger shape, etc.) is used, “when hand is held and round”, “when something is covered on head (hat etc.)” , “When a person is far away from the camera, etc., and the finger is not clearly visible”, “When the area that can be recognized may be limited, behavior recognition with movement is impossible”, etc. There is.

  When a monocular camera is used, a method of recognizing not the hand position but the arm position gently cannot be applied to behavior recognition. Also, it must be an ideal background, such as a white wall. Furthermore, even if a human body is mistakenly detected, the hand may be detected as it is.

  Furthermore, when only an infrared camera is used as a monocular camera, the type of camera is limited. Moreover, the case where there is a grasped object is not assumed.

  The present invention has been made in view of the above-described problems, and is a part recognition device, a part recognition method, and the like for accurately recognizing a person's part or the like included in a video or image obtained by photographing or the like. And a site recognition program.

  In order to solve the above problems, the present invention employs means for solving the problems having the following features.

Further, the present invention provides a human body region detecting means for detecting a human body region of at least one person included in the video or image, and the human body region detection in a part recognition device for recognizing a human part included in the video or image. Part recognition means for recognizing a predetermined part from the human body region obtained by the means, the part recognition means thins the human body region obtained by the human body region detection means, and the end point based on the thinned information And a branching point is generated, and the predetermined matrix is detected by comparing the generated matrix with a plurality of previously registered human matrices .

  Further, the present invention provides a human body region detecting means for detecting a human body region of at least one person included in the video or image, and the human body region detection in a part recognition device for recognizing a human part included in the video or image. Part recognition means for recognizing a predetermined part from the human body region obtained by the means, wherein the part recognition means thins the human body region obtained by the human body region detection means, and applies to the image of the human body region. Edge processing is performed, a portion where the edge of the human body region subjected to the edge processing has a predetermined shape is detected on the basis of the position of the line segment of the thinned human body region, and the region where the predetermined shape is detected is detected as the predetermined region. It is characterized by being a part of.

The present invention also provides a human body region detection step for detecting a human body region of at least one person included in the video or image in a part recognition method for recognizing a human part included in a video or image, and the human body A region recognition step for recognizing a predetermined region from the human body region obtained by the region detection step, wherein the region recognition step is based on the thinned information obtained by thinning the human body region obtained by the human body region detection step. Then , a matrix indicating a relationship between the end points and the branch points is generated, and the predetermined matrix is detected by comparing the generated matrix with a plurality of pre-registered human matrices .

  The present invention also provides a human body region detection step for detecting a human body region of at least one person included in the video or image in a part recognition method for recognizing a human part included in a video or image, and the human body A part recognition step for recognizing a predetermined part from the human body region obtained by the region detection step, wherein the part recognition step thins the human body region obtained by the human body region detection step, and images of the human body region Edge processing is performed on the basis of the position of the line segment of the thinned human body region, a portion where the edge of the human body region subjected to the edge processing has a predetermined shape is detected, and the region where the predetermined shape is detected is detected. The predetermined portion is used.

According to another aspect of the present invention, there is provided a part recognition program for causing a computer to function as the part recognition apparatus according to any one of claims 1 to 8 .

  ADVANTAGE OF THE INVENTION According to this invention, the site | part etc. of the person contained in the image | video and image obtained by imaging | photography etc. can be recognized with high precision.

It is a figure which shows an example of a function structure of the site | part recognition apparatus in this embodiment. It is a figure which shows an example of the hardware constitutions which can implement | achieve the site | part recognition process in this embodiment. It is a flowchart which shows an example of the outline process sequence of the site | part recognition in this embodiment. It is a figure for demonstrating the 1st Example of the site | part recognition in this embodiment. It is a figure which shows an example for demonstrating a circle detection process. It is a figure for demonstrating the 2nd Example of the site | part recognition in this embodiment. It is a figure for demonstrating the 3rd Example of the site | part recognition in this embodiment. It is a figure for demonstrating the 4th Example of the site | part recognition in this embodiment. It is a figure for demonstrating the matrixing process in this embodiment. It is a figure for demonstrating graph matching. It is a figure for demonstrating the production | generation method of the thinning graph to which the weight was provided. It is a figure for demonstrating a model graph database. It is a figure for demonstrating a response | compatibility when the target person's hand is raised. It is a figure for demonstrating the 6th Example of the site | part recognition in this embodiment. It is a figure for demonstrating the 7th Example of the site | part recognition in this embodiment. It is a figure for demonstrating the specific example of the circle | round | yen detection method using the human body edge in this embodiment. It is a flowchart which shows an example of the process sequence of (c). It is a figure for demonstrating the Example of hand detection. It is a figure which shows an example of the screen produced | generated by this embodiment. It is a figure which shows the other example of a screen produced | generated by this embodiment.

<About the present invention>
The present invention recognizes a person's part and recognizes a motion with high accuracy from a video (including an image) taken by an imaging means such as a camera or various videos acquired from the outside through a communication network such as the Internet. . Specifically, for example, the present invention recognizes a specific part (for example, a hand, arm, head, foot, etc.) of a person or the like included in an image by recognizing an image taken using, for example, an existing monocular camera. , Toes, etc.). In the present invention, accurate behavior recognition is performed by tracking the recognized specific part over time (in time series).

  Below, the form which carried out suitably the part recognition device, the part recognition method, and the part recognition program in the present invention is explained using a drawing. In the following processing, hand detection will be described as an example of human body region recognition. However, the present invention is not limited to this. For example, even a region such as a head (face) or a foot may be used. Good.

<Part recognition device: functional configuration example>
FIG. 1 is a diagram illustrating an example of a functional configuration of the part recognition apparatus according to the present embodiment. The part recognition apparatus 10 shown in FIG. 1 includes an input means 11, an output means 12, a storage means 13, a human body region detection means 14, a part recognition means 15, a behavior recognition means 16, a screen generation means 17, The notification means 18, the transmission / reception means 19, and the control means 20 are configured.

  The input unit 11 accepts various instructions for realizing the present embodiment such as a human body region detection instruction, a part recognition instruction, a behavior recognition instruction, a screen generation instruction, a notification instruction, and a transmission / reception instruction from a user or the like. Note that the input unit 11 includes, for example, a keyboard, a pointing device such as a mouse, or a voice input device such as a microphone.

  The output means 12 displays various information such as the progress or result executed in each configuration, or displays the sound based on the instruction content input by the input means 11 and the control data generated based on each instruction content. Output. The output unit 12 has a screen display function such as a display, a sound output function such as a speaker, and the like.

  Further, the output unit 12 outputs the results output by the respective functions, the information displayed on the screen generated by the screen generation unit 17 and the like to the external device. In other words, the output unit 12 outputs, for example, to a printer as an output to an external device, generates a file and outputs it to a storage unit 13, a storage device such as a preset database, or a recording medium, Switch on / off of sensors in the area (security target facility), turn on / off the light, and related information based on the part recognition result for mobile terminals held by security guards (location and contents where there was an abnormality) Has a print / output function and the like for outputting a control signal for displaying the image. Further, the output unit 12 can output simultaneously to one or more external devices described above.

  In addition, the input means 11 and the output means 12 mentioned above may be an integral structure as a touch panel.

  The storage unit 13 can store various information for realizing the above-described embodiment, and reading and writing are performed as necessary. Specifically, the accumulating unit 13 performs various feature amount data used for face authentication, gender and age estimation, a human body region detection result in the human body region detecting unit 14, and a hand in the part recognizing unit 15. Such as part detection results, behavior recognition results in the behavior recognition means 16, screen generation results in the screen generation means 17, notification results in the notification means 18, transmission / reception information in the transmission / reception means 19, information controlled by the control means 20, and when an error occurs Each information such as error information, log information, and a program for realizing the present invention is accumulated. Further, the storage means 13 stores a graph matching database, which will be described later, action pattern information corresponding to a human body region acquired in time series, a movement locus of a predetermined part, and the like.

  The human body region detection means 14 is, for example, a camera installed in the vicinity of a cash register of a convenience store, a department store, etc., in a predetermined monitoring area such as a bank reception, or a camera provided in a patrol surveillance robot. Real-time video captured by the imaging means, a huge amount of monitoring video accumulated after being captured, and stored in a remote image server via a communication network such as the Internet connected by the transmission / reception means 19 The human body region is detected from the images in the various videos such as the recorded video to determine whether or not a person is included.

  Specifically, the human body region detection unit 14 acquires, for example, a video captured by a camera or the like via the transmission / reception unit 19, and a predetermined image among the time-series images included in the acquired video. (Each frame image or an image with an interval of several frames or the like) is captured, and one or a plurality of persons are detected in the captured image.

  Further, the human body region detection means 14 compares, for example, successive image frames, and there is a place where the color information (luminance, chromaticity, etc.) changes within a predetermined time, and the region surrounded by the place is predetermined. More than this area, or those within the predetermined range of movement over time are detected as human body areas. It should be noted that the human body detection method is not limited to this in the present invention.

  Further, the human body region detection means 14 detects the center coordinates of the human body region and the size of the human body region on the image, and synthesizes the human body region with the original image with a predetermined shape so that the human body region can be clearly understood. Various information for screen display is acquired and stored in the storage means 13. Note that the shape of the human body region may be, for example, a rectangle, a circle, an ellipse, another polygon, a binary silhouette shape that is enlarged at a predetermined magnification from the outer shape of a person, or the like. That is, the human body region detection unit 14 can generate a silhouette-shaped image in which the human body region is painted white and the others are painted black, for example. Furthermore, the human body region detection means 14 may have a function of extracting color information such as hair, upper garment, and lower garment, and calculating position coordinates of a person in real space.

  The part recognizing means 15 detects, for example, a hand area as a predetermined part for the person area detected by the human body area detecting means 14. Specifically, the part recognizing means 15 performs thinning on the person area detected by the human body area detecting means 14. Further, the part recognizing means 15 converts the thinned image into a matrix (graph) representing the connection relationship between points such as end points and branch points.

  The part recognition means 15 performs edge detection on the entire original image, and extracts the edge of the human body region from the detected edge information of the entire screen. The part recognition means 15 can also perform matching (model graph matching) or the like using a person model graph (matrix model) in which the posture of the person is registered in advance based on the converted graph. Thus, it is possible to determine whether or not the obtained region is a person, to obtain one or more end points corresponding to the hand from the dictionary, and to recognize the end point as the hand.

  Further, if the recognition target is the hand, the part recognition means 15 may recognize the part of one hand or both hands, and in addition to the hand, one or more of a plurality of preset parts such as the head and feet may be recognized. A plurality of parts may be recognized simultaneously by performing a pattern matching process or the like using features such as preset shapes and colors. In detecting the hand, for example, first, the hand region of one hand of the target person included in the image is recognized, and then the other hand in the same image is determined based on the characteristics of the recognized hand region. By recognizing the area, both hands can be recognized.

  Further, as will be described later, the part recognizing means 15 determines each part such as the human head and the hand, and estimates the hand candidate before specifying the position of a predetermined part such as the hand. It can also be done. A specific example of the site recognition means 15 will be described later.

  The behavior recognition unit 16 chronologically identifies a human body region detected by the human body region detection unit 14 and / or a predetermined part (one or a plurality of parts) recognized by the part recognition unit 15 from a captured image. Acquired, and recognizes the behavior of the target person based on the continuous moving direction, moving speed, number of repetitions of a predetermined action, etc. of the acquired human body region or part. Note that the behavior recognition unit 16 can recognize the human behavior by, for example, collating time series information of the human body region and the predetermined part with a preset behavior pattern.

  Specifically, the behavior recognizing means 16 recognizes the presence or absence of the action of the hand, the behavior, etc. from the moving state of the hand, the position information of the hand, etc. . Also, the behavior recognition means 16 is used when, for example, the human body region makes a round trip around the cash register of a convenience store as a preset behavior pattern, or when it has been stopped near the cash register for a predetermined time or more. Can recognize the person as a suspicious person.

  Further, the behavior recognition means 16 extracts a face area from position information and a human body area such as the installation location and position of the installed camera, acquires a facial feature point from the face area, and behaves depending on the orientation of the face. Can also be recognized. For example, the feature points of the face are obtained by acquiring the feature amount of the face from position information such as eyes, nose, and mouth of the face included in the photographed image, and detected as a preset face. The face of a person is detected by performing a matching process using the matching pattern. Further, the present invention is not limited to the above-described face detection processing, and for example, face detection by edge detection or shape pattern detection, face detection by hue extraction or skin color extraction, or the like can be used.

  Furthermore, the behavior recognition means 16 detects the center coordinates (position information) of the face area and the size (size) of the area on the image, and synthesizes the face area with the original image with a predetermined shape to determine the face area. As can be clearly seen, various information for screen display can be acquired and stored in the storage means 13. In the present invention, the shape of the face region may be a rectangle, a circle, an ellipse, another polygon, a silhouette shape enlarged from the outer shape of a human face at a predetermined magnification, or the like.

  The screen generation means 17 includes an image captured by the camera, a human body area detected by the human body area detection means 14, a menu screen for performing part recognition in this embodiment, an input screen for performing part recognition, and a human body area result. Various screens necessary for realizing the part recognition processing in the present embodiment, such as a part recognition result, a behavior recognition result, and a notification result in the notification unit 18, are generated. At this time, the screen generation unit 17 generates not only a screen for displaying the results processed by the above-described configurations but also a screen for displaying various data set in advance in the storage unit 13 and the like. For example, digitized data (for example, coordinates, time information, and person information) related to position information corresponding to a photographed person's area can be displayed.

  Note that various information necessary for screen generation by the screen generation unit 17 can be used by appropriately reading out necessary information from information stored in the storage unit 13 in advance. Further, the screen generation means 17 can display the generated screen on a display or the like as the output means 12, and can output sound or the like through a speaker or the like.

  In the recognition result obtained by the behavior recognition means 16, the notification means 18 gives an emergency signal indicating that the person is a dangerous person, etc. The generated emergency signal is notified to a predetermined contact such as a security officer, a supervisor, a representative manager, a surveillance robot, or the like in charge of the building in a user, an administrator, a security company, or the like. Further, the notification means 18 causes the screen generation means 17 to generate and output information related to the image whose behavior has been recognized (shooting date and time, shooting location, video for a predetermined time before that) and information on the specific object. Displayed by means 12.

  For example, when notifying the monitoring robot or the like, the notifying unit 18 faces the target person or is close enough to the target person to be photographed by the imaging unit included in the monitoring robot. Therefore, it is also possible to notify a control signal to the monitoring robot that causes the monitoring robot to output a voice message to the target person, or to perform a process of calling attention to the surroundings by an alarm lamp or an emergency sound. .

  The transmission / reception means 19 receives a monitoring video from an imaging means included in one or a plurality of cameras or a monitoring robot via a communication network such as a LAN (Local Area Network) or the Internet. The transmission / reception means 19 can also select and acquire necessary information from various videos stored in an image server or the like at a remote location via the communication network described above.

  Here, it is preferable that the transmission / reception means 19 receives the monitoring video directly from the camera and processes it in real time to prevent crime in advance. For example, the transmission / reception means 19 temporarily stores the video previously acquired by the camera somewhere. In addition, the stored information may be collected and the various processes in the present embodiment described above may be performed.

  The transmission / reception means 19 can be used as a communication interface for transmitting various programs and various data stored in the storage means 13 in the apparatus to other terminals and receiving various data from other terminals. .

  The control means 20 controls the entire functional configuration of the part recognition device 10. Specifically, the control unit 20 performs various controls such as executing processing in each functional configuration described above based on instruction information from the user input by the input unit 11.

  In the above-described embodiment, the functions of the human body region detecting unit 14 and the site recognizing unit 15 are included in the site recognizing device 10, but the present invention is not limited to this, and the human body region detecting unit 14 and The function as the hand detection means (part recognition means) 15 may be a part recognition apparatus (not shown) and may be provided separately from the part recognition apparatus 10.

<Part recognition device: hardware configuration example>
Here, in the part recognition apparatus 10 described above, an execution program (part recognition program) that allows a computer to execute each function is generated, and the execution program is installed in, for example, a general-purpose personal computer (PC), a server, or the like. By doing so, the site | part recognition in this embodiment is realizable.

  Here, a hardware configuration example of a computer capable of realizing the part recognition process in the present embodiment will be described with reference to the drawings. FIG. 2 is a diagram illustrating an example of a hardware configuration capable of realizing the part recognition process in the present embodiment. 2 includes an input device 31, an output device 32, a drive device 33, an auxiliary storage device 34, a memory device 35, and a CPU (Central Processing Unit) 36 that performs various controls. And a network connection device 37, which are connected to each other via a system bus B.

  The input device 31 has a pointing device such as a keyboard and a mouse operated by the user, and inputs various operation signals such as execution of a program from the user. The output device 32 has a monitor for displaying various windows and data necessary for operating the computer main body for performing part recognition or the like in the present invention, and displays a program execution result or the like based on a control program in the CPU 36. Can be displayed.

  Here, in the present invention, the execution program installed in the computer main body is provided by, for example, the recording medium 38 such as a CD-ROM. The recording medium 38 on which the program is recorded can be set in the drive device 33, and the execution program included in the recording medium 38 is installed from the recording medium 38 to the auxiliary storage device 34 via the drive device 33.

  The auxiliary storage device 34 is a storage means such as a hard disk, and accumulates various data that can be read from the drive device 33 in addition to the execution program in the present invention and the control program provided in the computer. I / O can be performed. In addition, various data obtained by the above-described part recognition can be stored.

  The memory device 35 stores an execution program read from the auxiliary storage device 34 by the CPU 36. The memory device 35 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.

  The CPU 36 performs processing for the entire computer, such as various operations and input / output of data with each hardware component, based on a control program such as an OS (Operating System) and an execution program read and stored by the memory device 35. To control.

  Further, the CPU 36 can realize the part recognition process in the present embodiment by the execution program and the control program in the present invention. Various kinds of information necessary during the execution of the program can be acquired from the auxiliary storage device 34 and can also be stored.

  The network connection device 37 obtains an execution program from another terminal connected to the communication network by connecting to a communication network or the like, or various information obtained by executing the part recognition program or the program It can be provided to other terminals.

  The execution program in the present invention can be acquired and executed from the CD-ROM by an arbitrary terminal by storing it in a portable recording medium 38 such as a CD-ROM.

  In addition to the CD-ROM described above, the recording medium 38 is a recording medium that records information optically, electrically, or magnetically, such as a flexible disk or a magneto-optical disk, or information such as a ROM or a flash memory. Various types of recording media, such as a semiconductor memory that electrically records data, can be used.

  With the hardware configuration as described above, it is possible to perform the site recognition process in the present invention with high accuracy at low cost without requiring a special device configuration. Further, by installing the program, the part recognition process in the present invention can be easily realized by a general-purpose personal computer or the like.

<Example of part recognition processing>
Next, a part recognition processing procedure in the present embodiment using the part recognition apparatus 10 and the part recognition program described above will be described.

  FIG. 3 is a flowchart illustrating an example of a schematic processing procedure for part recognition in the present embodiment. In FIG. 3, first, a predetermined image included in a video image captured by an imaging means such as a camera attached at a predetermined position is captured (S01), and a human body region included in the captured image is detected. (S02). Next, it is determined whether or not there is a human body region as a result of the process of S02 (S03). If there is a human body region (YES in S03), the predetermined part of the human body (the hand) described above with respect to the human body region Etc.) (S04), the recognized part is traced in time series, and the behavior is recognized from the result (S05). Note that behavior recognition can be selectively performed as necessary. In the process of S04, if a plurality of people are photographed in the image, a predetermined part for each person is extracted.

  Thereafter, a screen for displaying the human body region detection result in the process of S02, the part recognition result in the process of S04, the behavior recognition result in the process of S05, etc. on the output means of the display is generated (S05), and the generated screen is displayed. (S07). Note that in the processing of S07, the comparison image is displayed by comparing human body region detection, part detection, behavior, and the like in a plurality of images selected from, for example, time-series images, not only from one image. You can also.

  Further, in the behavior recognition in the process of S05, when it is determined that the person is a suspicious person or a dangerous person, a notification is sent to the user, the management center, the guard, etc. (S08).

  Here, when there is no human body region in the process of S03 (NO in S03), or after the process of S08 is finished, it is determined whether or not the part recognition process is finished (S09), and the part recognition process is not finished (NO in S09), the process returns to S01 to capture the next target image and perform subsequent processing. In addition, when the part recognition process is ended by an end instruction from the user or the like (YES in S09), the process ends.

<Example of site recognition in this embodiment>
Next, an example of part recognition in the part recognition means 15 in the above-described embodiment will be described. In the following processing, processing contents in the human body region detection unit 14 will be described for convenience.

<First embodiment: circle detection>
First, a first example of site recognition in the present embodiment will be specifically described. FIG. 4 is a diagram for explaining a first example of site recognition in the present embodiment. In the first embodiment shown in FIG. 4, for example, a circle that seems to be a “hand” is searched for in the entire human body region detected from the image, and the hand is detected from the circle.

  In other words, in the circle detection, after detecting the edge of the human body, a circle search based on the size of the human body corresponding to the preset image size is performed in order from the top of the human body to recognize the position of the hand.

  As shown in FIG. 4, a predetermined image included in a video imaged by an imaging means such as a camera attached at a predetermined position is captured (acquired), and the captured original image (FIG. 4A) is obtained. Detect the human body area included. Specifically, the human body region is detected as a silhouette by the human body region detection means 14 (FIG. 4B). In the example of FIG. 4B, the human body region is painted white and the others are painted black. However, the present invention is not limited to this.

  Further, in the first embodiment, when a human body region is included in an image, an edge for the entire image is detected in parallel with the above-described human body region detection (FIG. 4C). This edge detection process can also be detected by the human body region detection means 14 described above.

  Next, the edge of the human body is detected from the edged image corresponding to the above-described human body region (FIG. 4D). Note that the edge in the present embodiment means a process for extracting an outline portion or the like of an object based on a color difference or luminance difference between adjacent pixels in an image.

  Next, in the first embodiment, circle detection for extracting a predetermined part (hand) part is performed. Specifically, the part recognizing means 15 searches the human body edge region by scanning a portion where the edge of the human body region is close to a circle from the end of the head or the like. The scanning direction is performed from the body direction to the hand. Then, for example, a portion close to a circle detected first is detected as a hand portion (FIG. 4E).

<About circle detection processing>
Here, the circle detection process described above will be specifically described. FIG. 5 is a diagram illustrating an example for explaining the circle detection processing. In the example of FIG. 5, there is a predetermined shape consisting of a circle or a shape close to a circle (an ellipse, a square, a regular hexagon, etc.) within a preset attention area including the human body edge detected by the human body edge detection process described above. Whether or not the circle is detected.

  Specifically, as shown in FIG. 5, all edge directions in the region are added to the histogram with respect to the angle of the edge direction from a certain point. At this time, if the shape of the edge is circular, the edge direction of all points is 90 ° with respect to the point of interest, and if it is almost circular, the value of the histogram within 80 to 100 degrees becomes large. Therefore, a circular portion can be detected by measuring these results by moving the point of interest on a thin line segment.

  When there is no circle in the attention area, the histogram results of the angles formed by the edges and the centers are different.

  Here, in this embodiment, for example, when the value of the added value is equal to or greater than a preset threshold value, it can be estimated that the portion is circular. Note that the threshold value can be changed as appropriate depending on the image quality, the size of the human body region in the image, and the like. When detecting a circle, the attention point is moved from the body part toward the hand based on the graphed information described above, and the attention point of the part where the circle is first detected is set as the hand position. . This is because the second and subsequent circle detections are likely to be gripped objects. Note that the above-described processing is performed on all hand candidates included in the image.

  As described above, by performing the circle detection process in the first embodiment, the position of a predetermined part (for example, a hand) with respect to the human body in the image can be obtained with high accuracy. In the present embodiment, the hand can be easily recognized using a predetermined shape, whether it is grasped or opened. The predetermined shape is set for each region to be recognized. For example, a foot shape is set for a foot, and a larger circular shape is set for a head.

<Second Embodiment: Thinned Image>
Next, a second example of site recognition in the present embodiment will be specifically described. FIG. 6 is a diagram for explaining a second example of site recognition in the present embodiment. In the second embodiment shown in FIG. 6, for example, the thinning process is performed without performing the circle detection in the first embodiment described above, and the position of a predetermined pixel among the end points of the thinned line segment is determined. From this, the hand position is determined.

  Specifically, as shown in FIG. 6, first, a human body region is extracted from the original image (FIG. 6A) (FIG. 6B). Up to this point, the same processing as in the first embodiment described above is performed, and a specific description thereof is omitted here.

  Next, thinning is performed on the human body region obtained in FIG. Here, the thinning process is obtained by compressing the silhouette of the human body region shown in FIG. 6B. Specifically, the thinning is performed by connecting the center points based on the outer peripheral shape of the silhouette. Is.

  Thereafter, based on the thinned information, the hand portion in the image is recognized from the shape of the line segment, the branch point, the end point, or a combination thereof, the relative positional relationship with other branch points and end points, and the like (FIG. 6). (C)).

  That is, in the second embodiment, the hand position is determined according to which pixel position the thinned end point is located.

<Third embodiment: circle detection + thinned image>
Next, a third example of part recognition in this embodiment will be specifically described. FIG. 7 is a diagram for explaining a third example of part recognition in the present embodiment. The third embodiment shown in FIG. 7 is a combination of the first and second embodiments described above. That is, when performing circle detection in the first embodiment described above, circle search is performed using the thinned line as a center candidate.

  Specifically, as shown in FIG. 7, the same processing as in FIGS. 7A to 7D is performed up to FIGS. 4A to 4D of the first embodiment described above. Thereafter, the silhouette of the human body region shown in FIG. 7B is compressed and thinned (FIG. 7E).

  After that, the thinned line segment is moved from the end of the line segment together with the center of the circle in the circle detection, thereby extracting the circular portion of the outer frame of the silhouette image of the human body region. Then, the detected circular part is recognized as the hand part, and the center position is acquired as the position coordinate of the hand part (FIG. 7F).

  As shown in the third embodiment, by performing a circle search using the thinned line as a circle center candidate, the processing cost can be reduced and the circle can be detected efficiently.

<Fourth Example: Thinned Graph>
Next, a fourth example of part recognition in this embodiment will be specifically described. FIG. 8 is a diagram for explaining a fourth example of site recognition in the present embodiment. In the fourth embodiment shown in FIG. 8, the thinned image in the second embodiment described above is matrixed, matched with a preset matrix model (model graph), and set to a matched matrix model in advance. The position of the hand is recognized based on the part information. In the following description, the above-described matching process with the matrix model is referred to as model graph matching.

That is, in the example of FIG. 8, first, the human body region is detected from the captured original image (FIG. 8A) in the same manner as in the second embodiment described above (FIG. 8B). Thereafter, thinning processing is performed based on the silhouette of the human body region (FIG. 8C), model graph matching is performed based on the thinned line segment (FIG. 8D), and the hand is detected ( FIG. 8 (e)).

<About matrix processing>
Here, the matrix processing of the thinned image described above will be specifically described with reference to the drawings. FIG. 9 is a diagram for explaining matrix processing in the present embodiment. In the matrix processing in the present embodiment, for example, for the thinned information, branch points and end points in the line segment are set, and the set branch points and end points are represented by a matrix. And graphing.

  Specifically, as shown in FIG. 9, the row of the matrix is the branch point number, the matrix column is the branch point and end point number, and when the matrix element is 0, “no connection”, When the matrix element is 1, graphing is performed using a matrix as “connected”.

  That is, in the example of FIG. 9, the branch point No. 0 is connected to the branch point No. 1 and the end points 0, 1, and 2, and the branch point No. 1 is the branch point No. 0 and the end point No. 3 This means that it is connected to No.4.

  The matrix generated as described above is matched with a preset matrix model, and the matrix model includes part information indicating what part each branch point or end point is in advance. And its position information are set. Therefore, by extracting the matched matrix model by model graph matching, the end point corresponding to the hand part can be easily and accurately acquired.

  In FIG. 8D described above, the thinned model is displayed for convenience, but actually, as shown in FIG. 9, it is stored in the storage means 13 as a graphed matrix model. ing.

<About model graph matching processing>
Here, the model graph matching process described above will be specifically described. FIG. 10 is a diagram for explaining the graph matching.

  As shown in FIG. 10, using a model graph database (matrix model database) in which a plurality of model graphs corresponding to a predetermined motion pattern of a person set in advance is stored, It is possible to accurately confirm whether or not it is a human body region.

  In the present embodiment, for example, accurate part information such as hands, heads, feet, etc. is set in advance in each model graph, and based on the part information, the hand part of the human body etc. is accurately obtained from the matched graph. Can be obtained.

  Therefore, as shown in FIG. 10, by using a model graph database, for example, graphed data obtained from an image is not matched when it is not a human body or when it is complicatedly branched due to noise or the like. Thus, it is possible to acquire information such as the detection of the human body and the position and orientation of the hand with higher accuracy. In this case, the state of the hand is not limited, and it may be grasped or opened.

  Also, for example, even if the target person is wearing a big hat, holding hands, wearing a cane, etc., even if it is different from the person itself, by setting the corresponding model graph in advance, Appropriate site identification can be performed.

  The model graph database described above may be stored in the storage unit 13 in advance, or may be acquired from an external device via a communication network such as the Internet using the transmission / reception unit 19.

  By performing the above-described matching, it is possible to recognize a specific posture (such as raising a hand), and it is possible to exclude an area that is erroneously detected at the time of human body detection without wasteful processing.

  The model graph shown in FIG. 10 is a conceptual diagram, and the data of the actual model graph is specifically registered based on a matrix or the like.

  By using the model graph, it is possible to perform a quick matching process by omitting useless information and to recognize a part with high accuracy.

<Fifth embodiment: weighted thinned graph>
Next, a fifth example of part recognition in this embodiment will be specifically described. In the fifth embodiment, substantially the same processing as that in the above-described fourth embodiment is performed (FIGS. 8A to 8E).

  However, in the fifth embodiment, compared with the fourth embodiment, the thinned image is made into a weighted matrix, and model graph matching is performed based on the matrix.

  Here, the above-mentioned content is concretely demonstrated using a figure. FIG. 11 is a diagram for explaining a method for generating a thinned graph with weights.

  In the graphing process in the fifth embodiment, a weight is assigned to a graphed value (matrix element).

  Specifically, as shown in FIG. 11, a predetermined frame image is captured from video captured by a camera or the like, a human body region is detected from the captured image, and the detected human body region is detected. Perform thinning. Further, based on the thinning information, graphing is performed using a matrix composed of branch points and end points as described above.

  At this time, the image area including the human body area is further subdivided into a plurality of areas (zones) according to the installation position of the camera, etc., and weighting is applied to the subdivided data to form a graph (matrix). Do.

  In the example of FIG. 11, for example, with respect to the image region corresponding to the human body region, a zone that is 1/5 from the top is defined as the head zone, the foot zone from the bottom, and the remaining regions are defined based on the overall vertical length. The hand zone is subdivided into three zones. Then, the weight of the subdivided head zone is set to 1, the weight of the hand zone is set to 2, and the weight of the foot zone is set to 3, and each is reflected as a graph. Furthermore, the difference can be clarified by using different numerical values (for example, the branch point 9, the end point 1, etc.) between the branch point portion and the end point portion.

  In this way, by assigning a different value for each type of weight or point set in advance for each zone, it is possible to easily grasp which part is the end point only by referring to the graph, The hand and other parts (for example, the head, feet, etc.) can be easily detected from that part. Further, by adding a high weight to the end points or branch points detected in the more important zones, the end points and branch points can be managed for each important zone. Note that the above-described weighting is also applied to the model graph set in advance and stored in the storage means 13 or the like.

  Therefore, according to the fifth embodiment, by matching the weighted matrix model with the model graph, it is possible to more reliably obtain the same model graph as the matrix, whereby the position of a predetermined part such as a hand Can be obtained with high accuracy.

<Regional information corresponding to the model graph>
Here, a specific example of the part information corresponding to the model graph described above will be described with reference to the drawings. FIG. 12 is a diagram for explaining the model graph database. 12A shows an example of data items of the model graph database, and FIG. 12B shows a specific example of data of the model graph database.

  The data items shown in FIG. 12A include, for example, “ModelID” as identification information for identifying a model by a model graph number, “branchnum” indicating the number of branch points of the model, and the number of nodes (branch) of the model. “Nodenum” indicating (point + number of endpoints), “graph” indicating its model graph (matrix model) (for example, including a weighted adjacency matrix as shown in FIG. 11), and the number of hands in the model “HandCount” and the like. Further, as shown in FIG. 12B, the data for the items shown in FIG. 12A has a plurality of models stored in the database. Note that the data arrangement, numerical conditions, and the like are not particularly limited in the present invention.

  For example, in model graph matching when recognizing a hand, among the components of the model graph database, “graph” represents hand, head, and foot information and is weighted to identify each part. . As described above, the weighting method is, for example, dividing the obtained human body region into five equal parts, the upper 1/5 is the head zone, the lower 1/5 is the foot zone, and the remaining central part is the hand zone. And weights 1 to 3 are assigned to the subdivided regions in order from the top. For example, 9 can be given to the connection between the branch points. In the present embodiment, for example, weighting may not be provided as in the above-described fourth example.

  As described above, for example, when a weighting method that divides a human body region into five parts is used, if the target person raises a hand, there are two head candidates, which makes it difficult to recognize the hand. . However, in this case, a specific posture (such as raising a hand) can be recognized by matching the thinned graph subjected to the weighting described above with a model graph prepared in advance, and erroneously detected when detecting a human body. Thus, the detected area can be excluded without wasteful processing.

  As described above, for example, when a weighting method that divides a human body region into five parts is used, if the target person raises a hand, there are two head candidates, which makes it difficult to recognize the hand. . However, in this case, a specific posture (such as raising a hand) can be recognized by matching the thinned graph subjected to the weighting described above with a model graph prepared in advance, and erroneously detected when detecting a human body. Thus, the detected area can be excluded without wasteful processing.

  Here, countermeasure processing when the target person's hand is raised will be described with reference to the drawings. FIG. 13 is a diagram for explaining the correspondence when the target person's hand is raised. FIG. 13A shows a thinned state when a hand is raised, and FIG. 13B shows an example of an expanded model graph database item. In the item example of FIG. 13B, compared to the data example of FIG. 12A, “head” that is a parameter for the head position determination process at the time of raising the hand and a model in which the hand is raised. “HandsUP” or the like indicating “” is added.

  That is, in the example of FIG. 13, whether or not the hand is raised is determined by “handsUp”. For example, when “handsUp” is 1 or more, processing is performed according to the value of “head”, and the hand and head are separated. To separate. That is, as shown in FIG. 13A, when there are two end points in the head zone, the x direction (horizontal direction) is compared, and the one located far from the branch point is set as the hand. To do.

  Also, as an example, when one hand is raised (handsUp: 1), the x coordinate value of the 0th branch point is compared with the head / hand candidate, and the one with the closest x coordinate value is defined as the head. By processing, the hand and head are separated. In “head”, information on which branch point should be compared is shown. In the countermeasure process when the hand is raised, the head / hand discrimination process described above is performed when a parameter indicating that the hand is raised is extracted as a result of the matching process. The head / hand discrimination process is a process performed by the above-described part detection means 15.

  As described above, by performing the model graph matching process, it is possible to recognize a specific posture (such as raising a hand), and to exclude a region that is erroneously detected at the time of human body detection without wasteful processing. it can.

<Sixth Embodiment: Circle Detection + Thinned Image Application>
Next, the sixth example of part recognition in this embodiment will be specifically described. FIG. 14 is a diagram for explaining a sixth example of site recognition in the present embodiment. In the sixth embodiment shown in FIG. 14, hand candidates are estimated when the circle detection and thinning process in the third embodiment described above are performed. In addition, estimation of a hand candidate is a process performed by the site | part detection means 15 mentioned above.

  In the sixth embodiment, as shown in FIG. 14, an arm region is estimated as a hand candidate (FIG. 14 (f)). Specifically, the processing from FIG. 14 (a) to FIG. 14 (e) is the same as that of the third embodiment (FIG. 7), but in the case of the sixth embodiment, the thin line is further reduced. The arm region is estimated from the digitized data. The arm area may be estimated by displaying a thin line segment on the screen and estimating the area designated by the user using the input means as the arm area. The arm region may be estimated on the basis of the positional relationship.

  Thereafter, in the sixth embodiment, circle detection is performed only on the estimated hand candidate area to identify the hand (FIG. 14G). Thereby, since the search range by circle detection can be reduced, site | part recognition can be performed more efficiently and rapidly.

<Seventh embodiment: Application of circle detection + thinning graph (including weighted)>
Next, a seventh example of part recognition in this embodiment will be specifically described. FIG. 15 is a diagram for explaining a seventh example of part recognition in the present embodiment. In the seventh embodiment shown in FIG. 15, the above-described sixth embodiment is further described in estimating the hand candidate from the thinned line segment (FIG. 15E) by the thinning process. Graph matching with the model graph is performed (FIG. 15 (f)).

  Specifically, one or a plurality of arm positions or regions as hand tips are set in the matrix model (model graph) described above, and the thinned line segments are matrixed (graphed) to form a matrix model. By matching with, the hand candidate area (FIG. 15G) can be easily obtained.

  Thereafter, in the seventh embodiment, circle detection is performed only on the position or region of the estimated hand candidate obtained by model graph matching, and the hand is specified (FIG. 15 (h)). Thereby, candidates can be estimated more accurately, and the search range by circle detection can be further reduced, so that site recognition can be performed more efficiently and quickly.

<About circle detection method of hand using human body edge and graph matching>
Here, for example, the hand circle detection method using the human body edge and the graph matching shown in the seventh embodiment will be specifically described with reference to the drawings. FIG. 16 is a diagram for explaining a specific example of the circle detection method using the human body edge in the present embodiment.

In the circle detection method using the human body edge in the present embodiment, for example, the following examples (a) to (d) can be considered.
(A) Edge detection → edge binarization → accumulation and discrimination in a histogram.
(A) Edge detection → edge binarization → edge thinning → accumulation and determination in a histogram.
(C) Edge detection → edge binarization → edge thinning → edge occupancy with respect to a circle is accumulated in a histogram for discrimination.
(D) The above methods (a) to (c) in which the circle center candidates are narrowed down using the thinned image are performed. Here, the specific processes (A) to (C) will be described below.

<About the method (a)>
First, as described above, after detecting the edges of the entire image, the edges are binarized by threshold processing or the like in order to leave only valid edges. Thereafter, the edge of the corresponding region is converted into a histogram, and circle determination is performed. When adding to the histogram, 1 is added to the corresponding angle for each pixel with an edge. The edge value before binarization may be added to the histogram.

  At this time, as shown in FIG. 16A, when edge detection is performed from the original image, it has edge strength information, and when binarization of the edge is performed thereafter, information on whether or not the edge is detected. Will have.

<About the method (a)>
In the method (a) described above, when the edge strength is high, the circle is composed of lines having a width of 2 pixels or more, so that the added value of the histogram increases, and a lot of circle-like parts may be detected. . Therefore, in the method (A), as shown in FIG. 16B, the edge is thinned to form a line of one pixel, thereby suppressing useless detection of a portion that is not a circle. Here, for edge binarization and thinning, a Canny edge detection process, which is a general process, is used. However, the present invention is not limited to this. In the case of the method (A), when adding to the histogram, 1 is added to the corresponding angle for each pixel having an edge.

<About method (c)>
In (a) described above, as the size of the circle increases, the number of pixels constituting the circle increases and the added value of the histogram increases. On the other hand, a small circle has a small addition value, and it may be difficult to detect a small circle when performing circle detection by giving a threshold to the number of votes. Further, in order to detect a small circle, it is necessary to lower the threshold value. As a result, there is a possibility that an edge lump that is not a circle is erroneously detected as a circle.

  Therefore, in the method (c), using the distance of the pixel of interest from the center of the region of interest, assuming that the pixel of interest constitutes a perfect circle, the ratio of the pixel of interest to the circumference (hereinafter referred to as the circle) , “Circle occupancy”) is added to the histogram. Thus, large and small circles can be handled in an integrated manner, and circle detection can be performed regardless of the size of the circle.

  Here, the method (c) will be described more specifically with reference to FIG. First, let r be the distance from the center of the region of interest to the pixel of interest as a constant. If the pixel of interest is an edge, it is assumed that a true circle is formed with the center of the region of interest as the center, and the pixel of interest occupies the circumference of the length of “1” that is the size of the pixel. Suppose that At this time, the circumference of a circle having a radius r is “2πr”. Therefore, the circle occupancy of the target pixel is “½πr”. The circle occupancy calculated in this way is added to the histogram, and the same processing is performed for all the pixels in the attention area. At this time, if the figure in the attention area is a perfect circle with the center of the attention area as it is and the edge is not interrupted, there are ideally 2πr edges, as shown in FIG. It can be considered that the value of the histogram is 1 (that is, the edges are aligned 100% without interruption). Thereby, for example, a circle can be determined using a threshold value such as “circle if the degree of circle occupation is 50% or more”.

  In this method, even if the figure in the attention area is an ellipse, it is a useful index for grasping how much the circumference occupies the edge and comparing with other areas. In the case of an ellipse, an edge closer to the center (closer to the short axis) has a higher occupancy (large weight of one vote), whereas an edge far from the center (close to the longer axis) has a lower occupancy (one vote). Small weight). When the ellipse is not interrupted, the circle occupancy is considered to be a value that can be approximated to 1.

  However, since it is assumed that the circumference occupies “1” in any position of the circle, the circle occupancy may exceed 1 when the circle is not interrupted. That is, when using a histogram using the degree of circle occupancy, more effective processing is possible by binarizing and thinning the edges.

  Here, the processing procedure of the above (c) will be described using a flowchart. FIG. 17 is a flowchart illustrating an example of the processing procedure of (c). In FIG. 17, first, an edge is detected as described above (S11), the detected edge is binarized (S12), and thinning is performed (S13).

  Next, a loop process for the region of interest is performed. As the loop processing for the attention area, as the loop processing for the pixel of the attention area, it is first determined whether or not the pixel of interest is an edge (S14), and if it is an edge (YES in S14), as described above. The circle occupancy is calculated (S15) and added to the histogram (S16). After the process of S16 is completed or when the pixel of interest is not an edge in the process of S14 (NO in S16), the same process is performed on the next pixel.

  After the loop processing for pixels is completed, it is determined whether or not the attention area is a circle from the result of the histogram (S17). If it is not a circle (NO in S17), another attention area is set. The same processing is performed on the attention area. Further, in the process of S17, when the attention area is a circle (YES in S17), the process ends.

  In addition, as the loop processing for the region of interest described above, for example, it is possible to perform loop processing from the trunk direction to the hand direction, with the upper arm of the thinned image set in advance as a circle center candidate, The present invention is not limited to this.

  In the above-described processing, the process ends when a circle is first detected or when a circle is not detected in the entire region. However, the present invention is not limited to this. For example, a circle is detected. The position of the attention area may be stored, and the loop process may be continued.

<Application example of hand detection>
Next, an application example of hand detection in the present embodiment will be described. Regarding the hand detection method described above, each hand can be detected by performing the same processing on both hands included in the image, but in the present embodiment, the present invention is limited to this. Instead, the other undetected hand can be detected based on, for example, information acquired from the detected one hand. Specific examples thereof will be described below. FIG. 18 is a diagram for explaining an example of hand detection.

  In the example of FIG. 18, for both hands of the same person in one image, the color information and shape characteristics of one hand and the other hand are the same or similar. A template is generated using features such as the shape pattern of the region and the color information of the image, and matching processing is performed on the entire human body region based on the generated template, and the most matching part is used as a hand.

  More specifically, first, filter information is created from one detected hand. This filter information includes hand information such as the hand size, shape and position based on the human body photographed in the target image. In the example of FIG. 18, a circle generated from the size and shape of the hand estimated from the height of the photographed human body is used as a filter. Further, the content of the filter is left as it is as the detected hand pixel value.

  Next, template matching is performed using this filter information. In FIG. 18, when the filter is moved sequentially from the top of the obtained human body region to the right side, and the degree of coincidence exceeds a preset threshold value, that portion is used as a hand.

  Here, the shape included in the filter information can be freely set such as a circle, a point, a rectangle, or any other shape (an ellipse or a rectangle). In addition to the detected hand pixel value, the value in the filter distributes the value defined as the skin color over the entire area, distributes the pixel values on the hand coordinates over the entire area, or averages the detected hand area For example, pixel values can be distributed in a region.

  The template matching location can be, for example, the entire image, a human body region, a hand zone in the human body region (a range weighted when there is a hand), and the like, based on a preset order and position. Processing can be performed.

Further, in the present embodiment, the similarity can be calculated by a mathematical formula. In this case, the formula to be calculated is generally an R _SSD such as the following formula (1), and the calculated R _SSD is When the value is equal to or greater than a preset value, it can be determined that the values match. Note that “R” of R _SSD indicates similarity, and “SSD” is an abbreviation of “Sum of Squared Difference”, which means that the distance calculation method uses the sum of squares of differences.

Here, M in Expression (1) indicates the number of horizontal pixels of the template image, and N indicates the number of vertical pixels of the template image.

  Note that the above-described mathematical expressions are not limited to this for the present invention. For example, the distance shown in the equation (1) is called “Euclidean distance”, and the Euclidean distance is a linear distance between pixels. In addition to the Euclidean distance, there are ways of thinking such as “chessboard distance” and “city distance”. They can also be used in the present invention.

  As another method, there is a method in which a pixel having a highest matching degree is used as a hand, in addition to a case where a pixel exceeding a threshold is used as a hand.

  Furthermore, as another embodiment, for example, when a cuff position is acquired based on a preset cuff shape such as a rectangle, and a hand and a cuff position are paired, or when many hand candidates appear Using the above-described circle detection method, it is possible to perform processing such as using a point with the highest circularity (appearing to be a circle) as a tip.

<Example of screen generated by screen generation means>
Next, an example of a screen generated by the screen generation unit 17 in the present embodiment will be described with reference to the drawings. FIG. 19 is a diagram illustrating an example of a screen generated according to the present embodiment.

  On the screen 40 shown in FIG. 19, an original image 41, a thinned image 42, a human body edge image 43, a part recognition result image 44, and the like can be displayed at a time. Further, by selecting each image, the target image can be enlarged and displayed in time series. In addition, when displaying an image, it is preferable that the original image 41, the thinned image 42, the human body edge image 43, and the part recognition result image 44 are displayed in synchronization with each other at the same time. One image may be displayed at different times.

  In the present embodiment, predetermined character information (for example, hand coordinates, part information such as the head, imaging time, etc.) is displayed on the image, for example, as a part recognition result image 44 shown in FIG. Can do.

  The contents and layout of the screen generated by the screen generation means 17 are not limited to this in the present invention. For example, the behavior result obtained by the behavior recognition means 16 and the notification means 18 are notified. In addition, various error messages at the time of execution of other processes can be displayed. Furthermore, the screen generation means 17 can generate a screen in which the part recognition results at different times are displayed side by side or combined and displayed.

  Here, FIG. 20 is a diagram illustrating another screen example generated by the present embodiment. For example, as shown in FIG. 20 (a), when the behavior of a person is recognized as a walking action, a screen that displays the moving predetermined frames side by side is generated.

  In the case of recognizing a walking action, the part recognizing means 15 first recognizes the parts of both hands and both feet, and then the behavior recognizing means 16 determines the distance between both hands and both feet (for example, large, medium, The behavior can be recognized from the movement distance of the human body region (movement of the x coordinate of the rectangle center, movement of the head coordinate, etc.). That is, in the example of FIG. 20A, the interval between the parts changes in the walking state, and the periodicity is recognized, so that it can be determined that the hand is waving and the foot is opened and closed.

  In the present embodiment, the screen generation means 17 can display a composite image obtained by combining a plurality of frames as shown in FIG. At this time, even if the human body region is moving, there is an overlapping portion depending on the speed. Therefore, in the present embodiment, as shown in FIG. 20B, the luminance of the human body region obtained by the human body region detecting means 14 can be changed and displayed through the human body. In addition, when the composite image is displayed, the screen generation unit 17 may generate a screen that displays the moving image of the head coordinates using dots, arrows, or the like as shown in FIG. 20B, for example. it can. In the present embodiment, the behavior recognition unit 16 can also determine a behavior (such as a moving action) only from the displacement of the head coordinates as described above.

  20C shows a continuous frame image when the raising hand action is recognized as the behavior recognition, and FIG. 20D shows the continuous frame of FIG. 20C overlapped similarly to FIG. 20B. A combined example is shown. In this case, since only the arm portion is moved, as shown in FIG. 20D, the region whose luminance is changed can be displayed as a partial region (for example, only the arm portion).

  FIG. 20 (e) shows a continuous frame image when recognizing a behavior to be recognized as behavior recognition, and FIG. 20 (f) shows the continuous frame of FIG. 20 (e) overlapped as in FIG. 20 (b). A combined example is shown. In this case, since only the arm portion is moved, as shown in FIG. 20 (f), the region whose luminance is changed can be displayed as a partial region (for example, only the arm portion).

  In this embodiment, as shown in FIGS. 20A to 20F, the part recognized by the part recognition unit 15 can be highlighted with a circle or the like, and further obtained by the behavior recognition unit 16. The trajectory of movement can also be displayed with an arrow or the like.

  Furthermore, the screen generation means 17 can also enlarge the position where the hand has been recognized, whereby it is possible to easily confirm information on the grasped object useful for crime prevention and the action of grasping the object. When a real-time video from the camera is displayed, the behavior recognition unit 16 can display a zoomed-up image of the part using the function of the photographed camera.

  As described above, according to the present invention, it is possible to recognize a human part included in a video or image obtained by photographing or the like with high accuracy. Specifically, according to the present invention, since color information is not used, the present invention can be carried out even if there is something that covers the face or hand. Further, according to the present invention, since it is resistant to changes in illumination, it is possible to recognize a part if the edge is visible even when it is dark, as well as when there is a color difference between the face and the hand. At the same time, the illumination may be strong, and even a black and white image may have poor color adjustment.

  Further, the region recognition can be performed even if the person does not face the front of the camera. The present invention is not limited to an analog camera, and may be a network camera, a USB camera, an infrared camera, or the like, and has a wide range of applications. Furthermore, when the present invention is carried out, it is only necessary to have a video or an image acquired from one camera, so a plurality of cameras are not required and the cost is low. In addition, video and images from analog cameras that are often installed throughout the country can be used as they are.

  Further, according to the present invention, the present invention is not limited to the background of the image, and does not force the target person to perform a predetermined action or marker. A predetermined part can be acquired.

  In addition, as an application example of the present invention, for example, by recognizing a specific behavior, it is possible to automatically report to a supervisor or the police, and to reduce the burden on the supervisor. Specifically, according to the present invention, an action that moves the hand forward and backward is “spoken” and immediately notified to the police, or a person who moves (passes through) the hand is excluded from monitoring. It can be used for processing.

  Further, in the present invention, if it is used for detecting the tip of a foot, it is possible to detect walking using information on the position of the foot. Moreover, since only a predetermined action needs to be confirmed, the number of confirmation objects is reduced, and it is possible to allow a monitor to confirm a plurality of monitors simultaneously. Therefore, cost reduction can be achieved.

  Furthermore, in the patrol of the security guard, for example, when confirming the position of the door lock or the fire extinguisher, confirmation of pointing called “pointing designation” is performed, and it is recognized whether or not such pointing designation is performed. Therefore, it can be applied to attendance management of security guards.

  In addition, the behavior recognition method according to the present invention can be used as a function of a user interface such as “turn off the light when the hand is waved”. In addition, by recognizing a specific behavior in the present invention, for example, by recognizing a behavior such as picking up a product and returning it, a detailed user operation result can be reflected in marketing.

  In addition, the present invention enables recognition of a person who performs a motion of stopping a taxi by raising his hand or a motion of a traffic controller from the processing of an image by a vehicle-mounted camera. In addition, the present invention can also create a privacy image that expresses only a person's actions by utilizing the existing security camera by generating an animation image by applying the above-described human body region body detection and part recognition. .

  Further, the present invention can be applied to a system that can automatically support rehabilitation even in the absence of a doctor or a physical therapist. For example, taking a picture of a person performing rehabilitation, measuring the state of exercise function recovery training of the limbs, or applying it to a system that directs the position of the limbs to an appropriate position, such as "more hands up" Can do.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications, within the scope of the gist of the present invention described in the claims, It can be changed.

DESCRIPTION OF SYMBOLS 10 Part recognition apparatus 11 Input means 12 Output means 13 Accumulation means 14 Human body area | region detection means 15 Part recognition means 16 Behavior recognition means 17 Screen generation means 18 Notification means 19 Transmission / reception means 20 Control means 31 Input apparatus 32 Output apparatus 33 Drive apparatus 34 Auxiliary Storage device 35 Memory device 36 CPU (Central Processing Unit)
37 Network connection device 38 Recording medium 40 Screen 41 Original image 42 Thinned image 43 Human body edge image 44 Region recognition result image

Claims (11)

In a part recognition device for recognizing a part of a person included in a video or an image,
Human body region detecting means for detecting a human body region of at least one person included in the video or image;
Recognizing a predetermined part from the human body region obtained by the human body region detecting means,
The part recognition means includes
The human body region obtained by the human body region detection means is thinned, a matrix indicating the relationship between the end points and the branch points is generated based on the thinned information, and the generated matrix and a plurality of pre-registered persons A part recognition apparatus that detects the predetermined part by comparing with a matrix . The part recognition means includes
Site according to claim 1, wherein an image including a human body region is subdivided into a plurality of regions, and generates a granular said branch point and said end point for each region and the matrix by applying a weight to Recognition device. In a part recognition device for recognizing a part of a person included in a video or an image,
Human body region detecting means for detecting a human body region of at least one person included in the video or image;
Recognizing a predetermined part from the human body region obtained by the human body region detecting means,
The part recognition means includes
The human body region obtained by the human body region detection means is thinned, edge processing is performed on the human body region image, and the edge-processed human body is used with reference to the position of the line segment of the thinned human body region. A part recognizing apparatus, wherein a part where an edge of a region has a predetermined shape is detected, and the region where the predetermined shape is detected is set as the predetermined portion. The part recognition means includes
Of the thinned human body region, the region where the predetermined part is present is estimated, and the edge-processed human body is used with reference to the position of the line segment of the thinned human body region included in the estimated region The part recognition apparatus according to claim 3 , wherein a part where an edge of the area has a predetermined shape is detected, and the area where the predetermined shape is detected is set as the predetermined part. The part recognition means includes
Based on the thinned information, a matrix indicating the relationship between the end points and the branch points is generated, and the predetermined matrix is detected by comparing the generated matrix with a plurality of previously registered human matrices. The part recognition apparatus according to claim 3 or 4 , wherein The part recognition means includes
6. The matrix according to claim 5 , wherein the image including the human body region is subdivided into a plurality of regions, and the matrix is generated by assigning weights to the end points and the branch points for each subdivided region. Site recognition device. The part recognition means includes
Based on the characteristics of the hand area of one hand of the person, the part recognition apparatus according to any one of claims 1 to 6, characterized in that recognize other hand areas in the same image. Claims 1 to 7 wherein the track in time series site obtained by the body region and / or the region recognition unit obtained by the body region detection unit, characterized by having a behavior recognizing means for recognizing the behavior of the person The site | part recognition apparatus of any one of these. In a part recognition method for recognizing a part of a person included in a video or an image,
A human body region detecting step of detecting a human body region of at least one person included in the video or image;
Recognizing a predetermined part from the human body region obtained by the human body region detection step,
The site recognition step includes
The human body region obtained by the human body region detection step is thinned, a matrix indicating the relationship between the end points and the branch points is generated based on the thinned information, and the generated matrix and a plurality of pre-registered persons A part recognition method , wherein the predetermined part is detected by comparing with a matrix . In a part recognition method for recognizing a part of a person included in a video or an image,
A human body region detecting step of detecting a human body region of at least one person included in the video or image;
Recognizing a predetermined part from the human body region obtained by the human body region detection step,
The site recognition step includes
The human body region obtained by the human body region detection step is thinned, edge processing is performed on the image of the human body region, and the edge-processed human body is used with reference to the position of the line segment of the thinned human body region. A part recognizing method, wherein a part where an edge of a region has a predetermined shape is detected, and the region where the predetermined shape is detected is set as the predetermined portion. Computer
The part recognition program for functioning as a part recognition apparatus as described in any one of Claims 1 thru | or 8 .