JP2010123019A - Device and method for recognizing motion - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To recognize motion of a person even when it is difficult to discriminate a face area from the other areas with color information when motion of the person is recognized using video. <P>SOLUTION: An action estimation device 205 divides each image obtained by high-resolution cameras 211 and 213 into a plurality of small areas, and associates the small areas of images with each other, which are recognized to include the same portion of a person. Then, a distance between correspondence points of the associated two small areas is calculated as parallax. Next, parallax of a face area obtained by imaging the face of the person is compared with parallax of a flesh color area obtained by imaging a flesh color portion other than the face of the person, and a flesh color area having parallax close to the parallax of the face area around the face area is detected as an arm area. The motion of an arm of the person is recognized on the basis of a relative position of the arm area with respect to the face area, and a difference between the parallax of the face area and the parallax of the arm area. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a motion recognition apparatus and method for recognizing a motion of a person.

  In recent years, robots have been proposed that support people's activities by managing their actions and schedules. For example, a robot that recognizes a person's action, points out an inappropriate action, or prompts an appropriate action.

  As a human action recognition technology based on video, a method is known in which motion elements are extracted based on a person's movement pattern and posture, and actions such as “eat” and “drink” are recognized based on the frequency of appearance of the motion elements. Yes. In this case, for example, skin color (face, hand, foot) and clothing color are extracted based on color information included in the image, and posture / movement information is extracted based on the extracted color.

There is also known a face image determination method for determining whether a certain area in an image is a face area.
Furthermore, a method for extracting a high-resolution video of a moving region from a time-series low-resolution image and a special lens having a high central resolution are also known.
JP 2005-215927 A JP 2005-284348 A JP 2007-000205 A Active Stereo Vision System with Foveated Wide Angle Lenses, Y. Kuniyoshi, N. Kita, S. Rougeaux and T. Suehiro, Proc. Of Asian Conf. On Computer Vision (1995), pp. 359-363.

The conventional action recognition technology described above has the following problems.
In the method of extracting posture / motion information based on color information, it may be difficult to distinguish a face area from other areas by color information if conditions such as clothing color and lighting conditions are bad. In this case, a person's arm region or the like cannot be specified in the image, and posture / motion information cannot be accurately extracted.

  It is an object of the present invention to make it possible to recognize a person's action even when it is difficult to distinguish a face area from other areas by color information in human action recognition by video.

The disclosed motion recognition apparatus includes first and second imaging means, parallax calculation means, detection means, and recognition means.
The first and second imaging means image a person. The parallax calculation means divides the image obtained by each of the first and second imaging means into a plurality of small areas, and associates the small areas of each image recognized as having captured the same part of the person . Then, the distance between the corresponding points of the two associated small areas is calculated as the parallax.

  The detection unit compares the parallax of the face area recognized as an image of a person's face with the parallax of the recognized skin color area as an image of a skin color part other than the person's face. Then, a skin color area whose distance from the face area is smaller than the first threshold and whose parallax difference from the face area is smaller than the second threshold is detected as an arm area.

The recognizing means recognizes the movement of the person's arm based on the relative position of the arm area with respect to the face area and the difference between the parallax of the face area and the parallax of the arm area.
The first and second imaging means output a human image to the parallax calculation means, and the parallax calculation means outputs the distance between the corresponding points of the associated small areas as parallax to the detection means and the recognition means. The detection means outputs information on the face area and the arm area to the recognition means, and the recognition means outputs information on the recognized motion.

  By using the images of the first and second imaging means together, the parallax of each small area can be obtained. Further, if the difference in parallax between the face area and the skin color area is small, it is considered that the distance between the face and the skin color part in the depth direction of the image is small. Therefore, by comparing the parallax of the face area and the parallax of the skin color area, the skin color area corresponding to the arm area of the same person can be detected by excluding the skin color area of another person or the like who is far away from the depth position.

  According to the disclosed motion recognition device, even when it is difficult to distinguish a face region from other regions using color information in human action recognition by video, the arm region is identified and the motion of the person's arm is recognized. can do.

Hereinafter, the best embodiment will be described in detail with reference to the drawings.
FIG. 1 shows a configuration example of the motion recognition apparatus of the embodiment. 1 includes a wide-field imaging unit 101, a high-resolution imaging unit 102, a motion / face detection unit 103, a gazing point control unit 104, a posture estimation unit 105, and an action recognition unit 106.

  The wide-field imaging unit 101 captures a wide area image including objects such as a person and an object, and the motion / face detection unit 103 detects the movement and face of the person from the captured image. The gazing point control unit 104 performs control to adjust the gazing point of the high resolution imaging unit 102 to the target object, and the high resolution imaging unit 102 shoots the video around the gazing point with high resolution. The posture estimation unit 105 estimates the posture of the person from the video shot by the high-resolution photographing unit 102, and the action recognition unit 106 specifies the action of the person from the estimated posture.

The video captured by the wide field imaging unit 101 and the high resolution imaging unit 102 includes time-series images. As configurations of the wide-field imaging unit 101 and the high-resolution imaging unit 102, for example, the following is used.
(1) Two types of cameras are provided: a wide-field camera and a high-resolution camera.
(2) A camera capable of simultaneously shooting a wide field of view and a high-resolution image by using a special lens having a high central resolution.
(3) A wide-field image and a high-resolution image are taken with the same camera using a camera having a zoom function. However, in this case, the wide-field video and the high-resolution video cannot be referenced simultaneously. (4) Even when using a camera that can shoot wide-field images and high-resolution images, if those images cannot be processed in real time, a wide-area image with reduced resolution and a partial image that has not been reduced Use.
(5) Utilizing a wide-field camera and a high-resolution video of the point of interest by super-resolution technology. As a super-resolution technique, for example, a method of extracting a high-resolution video of a moving area from a time-series low-resolution image can be used.

  By using a wide-field image by the wide-field imaging unit 101, it is possible to grasp a phenomenon occurring in a wide area, and by using a high-resolution image by the high-resolution imaging unit 102, a person's action and its object Can be accurately identified.

  The movement / face detection unit 103 detects rough movement of the object by extracting the color information of the object from the image, and detects the face area of the person from the image by methods such as skin color extraction and face image determination. Extract.

  The point-of-gaze control unit 104 acquires information indicating the rough movement of the target object to be watched from the movement / face detection unit 103, and includes color information of the target object, optical flow that is vector information indicating the movement of the target object, and the like. Is used to finally capture the object in the field of view of the high-resolution imaging unit 102.

  For example, when the person who is gazing is out of the field of view of the high-resolution imaging unit 102, the wide-field imaging unit 101 acquires motion information (for example, an optical flow) of an object around the gazing point. The relative position information between the high-resolution image of the person and the surrounding objects can be continuously obtained by moving the gazing point of the high-resolution imaging unit 102 in accordance with the movement.

  The posture estimation unit 105 estimates, for example, the direction of the person and the positions of the hands and arms as the posture of the person. When a stereo camera is used as the high-resolution imaging unit 102, the posture of the person is estimated by three-dimensional measurement using stereo vision. When shooting at high resolution, the field of view of each of the two cameras is narrowed, and the overlap of the images of the stereo camera capable of three-dimensional measurement is limited. Therefore, the congestion control of each camera is performed as necessary.

  Since the field of view of the high-resolution camera is narrow, when performing position calculation based on normal parallax in three-dimensional measurement, the influence of the accuracy of congestion control is large, and calibration is difficult. On the other hand, what is necessary when recognizing a person's action is the posture of the person, the position of the arm, and the relative positional relationship with surrounding objects, not the absolute position of the person or object.

  Therefore, instead of calculating the three-dimensional absolute position, the posture estimation unit 105 pays attention to the relative parallax of the stereo feature points of each camera in order to measure the relative distance in the vicinity of the gazing point. Get the relative positional relationship.

  As a result, as long as each camera captures the target in the field of view, the posture of the person can be estimated without being affected by the accuracy of the congestion control or calibration. In addition, the calibration itself that is necessary for calculating the absolute position becomes unnecessary.

  The action recognition unit 106 recognizes the action of the person based on the pattern of the time series information of the posture of the person. Depending on the action, it is necessary to identify the action object. Therefore, in addition to the time-series information of the posture, the detailed action is recognized by using the surrounding situation and the detailed video of the object.

  For example, in the case of recognizing an action of “drinking medicine”, if the pattern of the “drinking” action is recognized, the object touched by the hand immediately before can be photographed with high resolution and recognized as a medicine container. Good. In addition, when recognizing the action of “opening the door”, it is only necessary to recognize that the door has been opened by an optical flow pattern or the like in a wide area image simultaneously with the movement pattern of the arms and hands.

  According to such a motion recognition device, a wide-field video and a partial high-resolution video can be used at the same time, and flexible action recognition can be performed according to a person's posture history and surrounding conditions. Further, by measuring the relative positional relationship between a person and a surrounding object, highly accurate congestion control and calibration are not necessary. Furthermore, by adopting congestion control, the range in which three-dimensional measurement is possible is expanded, and recognition processing that is resistant to movement is realized.

FIG. 2 shows a more specific configuration example of the motion recognition apparatus of FIG. 2 includes a camera head 201, a pan / tilt control motor 202, a camera control device 203, an image processing device 204, an action estimation device 205, and a host device 206. Camera head 2
01 includes high-resolution cameras 211 and 213, congestion control motors 212 and 214, and a wide-field camera 215.

  The wide-field camera 215 corresponds to the wide-field imaging unit 101 in FIG. 1, and the high-resolution cameras 211 and 213 correspond to the high-resolution imaging unit 102. The image processing device 204 and the action estimation device 205 correspond to the motion / face detection unit 103, the posture estimation unit 105, and the action recognition unit 106, and the camera control device 203 and the host device 206 correspond to the gaze point control unit 104. .

  The pan / tilt control motor 202 is driven by the camera control device 203 to cause the camera head 201 to perform a pan / tilt operation. When the entire camera head 201 performs a pan / tilt operation, the shooting direction changes. If the pan / tilt operation is unnecessary, the pan / tilt control motor 202 may be omitted.

  The high-resolution cameras 211 and 213 constitute, for example, one stereo camera, and the congestion control motors 212 and 214 are driven by the camera control device 203 to cause the high-resolution cameras 211 and 213 to perform a convergence motion.

  The image processing device 204 extracts image information such as the color of the object and the optical flow from the images included in the images captured by the wide-field camera 215 and the high-resolution cameras 211 and 213, and outputs the image information to the action estimation device 205. .

  The action estimation device 205 outputs the image information from the image processing device 204 to the host device 206, detects rough movement of the object using the image information, and extracts a human face area, arm area, and the like. . Then, the posture of the person is estimated using the extracted face region, arm region, and the like, an action is recognized from the estimated posture, and the recognition result is output to the host device 206.

  The host device 206 instructs the camera control device 203 to perform pan / tilt operation and congestion control based on the image information from the action estimation device 205, and the camera control device 203 follows the instructions, and the congestion control motors 212, 214, and The pan / tilt control motor 202 is driven. Further, the host device 206 instructs the action estimation device 205 to start action recognition, and receives a recognition result from the action estimation device 205.

  For example, an information processing device (computer) is used as the camera control device 203, the image processing device 204, the action estimation device 205, and the host device 206. It is also possible to realize all of these apparatuses using one information processing apparatus.

  In this case, video data acquired by the wide-field camera 215 and the high-resolution cameras 211 and 213 and image information such as the color of the object and the optical flow are stored in the memory as data to be processed.

  FIG. 3 shows an arrangement example of the high resolution cameras 211 and 213. In FIG. 3, the left camera 312 and the right camera 322 correspond to the high resolution cameras 211 and 213, respectively. Congestion control is control of the movement of each of the left camera 312 and the right camera 322 rotating around a vertical axis. When photographing the target person 301 at high resolution, the congestion control of the left camera 321 and the right camera 322 is performed individually in order to ensure the overlap of the images of the left camera 321 and the right camera 322 necessary for three-dimensional measurement. Is called.

FIG. 4 shows the concept of relative parallax in the arrangement of FIG. When the left camera 321 and the right camera 322 capture a person 301, the left camera 321 acquires an image 401 and the right camera 322 acquires an image 402. Images 401 and 402 include images of the face 311, the right arm 312, and the left arm 313 of the person 301.

When the horizontal direction of the images 401 and 402 is the x-axis and the vertical direction is the y-axis, when one point in the three-dimensional space is photographed by both cameras, the relative parallax at that point is defined by the following equation, for example: The

Relative parallax = x coordinate of left camera-x coordinate of right camera (1)

In the convergence control, only a rotational movement around the vertical axis is performed, so that no parallax occurs with respect to the y coordinate.

  As shown in FIG. 4, if the x-coordinates of the images 401 and 402 are defined as 0, 1, 2, and 3, the relative parallax of the points 411, 412, 413, and 414 is 0, 1, 2, and 3, respectively. It becomes. Further, the relative parallaxes of the points 421, 422, 423, 424, 431, 432, 433, 441, and 442 are -1, 0, 1, 2, -1, 0, 1, -1, and 0, respectively. Therefore, it can be seen that the relative parallax increases as the point is closer to the left camera 321 and the right camera 322, and the relative parallax decreases as the point is farther from the left camera 321 and the right camera 322.

  In addition, the difference in relative parallax between the points close to each other in the depth direction of the images 401 and 402 like the points 412, 423, and 433 is small or 0, and the points in the depth direction like the points 412 and 431 The difference in relative parallax between points that are separated is large. Therefore, by comparing the relative parallaxes of the face 311, the right arm 312, and the left arm 313, the relative positional relationship of these portions in the depth direction can be determined.

  FIG. 5 is a flowchart illustrating an example of gaze point control processing by the camera control device 203 and the host device 206. In a state where the left camera 321 and the right camera 322 are capturing an object such as a person 301 (step 501), the host device 206 has detected whether the left camera 321 has captured the object based on the color information of the object. It is checked whether or not (step 502).

  If the left camera 321 has not captured the object (NO in step 502), the image processing apparatus 204 is instructed to calculate the optical flow via the action estimation apparatus 205. The image processing device 204 calculates the optical flow of the object using the wide-area video of the wide-field camera 215, and outputs it to the host device 206 via the action estimation device 205 (step 503).

  The host device 206 performs pan / tilt control of the camera head 201 via the camera control device 203 to search for an object with the left camera 321 while moving the line of sight in the direction indicated by the optical flow (step 504). Then, the processing after step 502 is repeated.

  On the other hand, if the left camera 321 has captured the object (step 502, YES), the host device 206 checks whether the right camera 322 has captured the object based on the color information of the object. (Step 505).

  If the right camera 322 has not captured the object (step 505, NO), the right camera 322 is controlled by the camera control device 203 to control the right camera 322 while moving the line of sight of the right camera 322 in the horizontal direction. The camera 322 searches for an object (step 506). Then, the processing after step 505 is repeated.

On the other hand, if the right camera 322 has captured the object (step 505, YES), the processing after step 501 is repeated.
By continuing such a gazing point control process, the object can always be captured by the left and right cameras. In the gazing point control process of FIG. 5, the video of the left camera 321 is used for pan / tilt control and the video of the right camera 322 is used for congestion control, but the left camera 321 and the right camera 322 are used. The control order may be changed.

  FIG. 6 is a flowchart illustrating an example of motion recognition processing by the motion recognition apparatus of FIG. The motion recognition apparatus performs the gazing point control process shown in FIG. 5 to capture the target person in the field of view of the left camera 321 and the right camera 322 (step 601). Next, the action estimation apparatus 205 performs the processing of steps 601 to 606.

  The action estimation device 205 first searches for corresponding points between the image of the left camera 321 and the image of the right camera 322 by stereo matching (step 602). Here, for example, each image of the left camera 321 and the right camera 322 is divided into a plurality of small areas, and the correlation between the small areas of the respective images is obtained. Then, the small areas with high correlation are recognized as images of the same part, and the small areas are associated with each other. Then, the relative parallax is calculated by the equation (1) for each corresponding point (corresponding pixel) in the two associated small regions, and is stored in the memory. As the shape of the small region, for example, a rectangle including a certain number of pixels is used.

  Next, using either the left camera 321 or the right camera 322, the face area and arm area (or hand area) of the person are specified (step 603). In the following, it is described as an arm region including a hand region.

  Here, for example, an area of the skin color portion (skin color area) of the person is extracted by skin color extraction, and the face area of the person is extracted by face image determination. Then, the average value of the relative parallax of the points included in the extracted face area (average parallax) is compared with the average parallax of each skin color area, and the average parallax close to the average parallax of the face area around the face area is obtained. The skin color area to have is detected as an arm area.

  For example, the center of gravity coordinates of the face area and each skin color area are calculated, the distance between the center of the face area and the center of the skin color area is calculated, and if the distance is smaller than the threshold, it is determined that the skin color area is around the face area Is done. Further, the difference between the average parallax of the face area and the average parallax of the skin color area is calculated, and if the difference is smaller than the threshold value, it is determined that the parallax of the skin color area is close to the parallax of the face area. Information of the face area and arm area extracted in this way is stored in the memory.

Next, the width, height, and range of variation in parallax of the face area are calculated (step 604), and the relative positional relationship of each arm area with respect to the face area is estimated (step 605). Here, for example, the positional relationship is estimated by the following procedure.
1. The center of gravity coordinates (fx, fy) of the face area, the width fw and height fh of the face area, the average parallax fmd of the face area, and the parallax variation range fdd are calculated. The range of the parallax variation is obtained as a difference between the maximum value and the minimum value of the parallax in the face area. At this time, error may be taken into consideration, such as by removing outliers.
2. Select one unprocessed arm region.
3. The center-of-gravity coordinates (ax, ay) of the arm region and the average parallax amd of the arm region are calculated.
4). Judgment of left-right positional relationship (α is a positive constant)
If fx−ax <−α × fw, it is determined that the arm region is to the right of the face region.
If fx−ax> α × fw, it is determined that the arm region is to the left of the face region.
If −α × fw ≦ fx−ax ≦ α × fw, it is determined that the arm region is in the center of the face region.
5). Judgment of the vertical relationship (β is a positive constant)
If fy−ay <−β × fh, it is determined that the arm region is above the face region.
If fy-ay> β × fh, it is determined that the arm region is below the face region.
If −β × fh ≦ fy−ay ≦ β × fh, it is determined that the arm region is in the center of the face region.
6). Determining the positional relationship of depth (γ is a positive constant)
If fmd−amd <−γ × fdd, it is determined that the arm region is in front of the face region.
If fmd−amd> γ × fdd, it is determined that the arm region is behind the face region.
If −γ × fdd ≦ fmd−amd ≦ γ × fdd, it is determined that the arm region is in the center of the face region.
7). You may perform the said process of 3-6 about another arm area | region (or hand area | region).

  According to such positional relationship estimation processing, it is possible to estimate the relative positions of the face and the arm only by calculating simple parameters. Further, by setting appropriate values of α, β, and γ based on simulation or the like, threshold values −α × fw, α × fw, −β × fh, β × fh, −γ × fdd, and γ × It is possible to adjust fdd and improve the estimation accuracy.

Note that, instead of the average parallax of the face area, the skin color area, and the arm area, a parallax obtained by another statistical calculation such as a weighted average may be used.
Once the relative positional relationship between the arm region and the face region is estimated, the person's action is estimated from the time-series information of the arm position and the situation of objects around the person (step 606). Here, for example, the relative position of the arm region with respect to the face region is recorded in memory in time series, and the action of a person is recognized by collating the recorded time series information with a pre-registered operation pattern. .

  For example, as an arm movement pattern in the act of taking medicine, a pattern in which the hand moves in the order of medicine → mouth → cup → mouth can be considered. In this case, in the storage device of the action estimation device 205, a recognition table as shown in FIG. 7 is registered as an operation pattern, and an object table as shown in FIG. 8 is registered as the position of the object.

  The stage of action shown in FIG. 7 represents the temporal context of each action included in the action, and the position of the arm is the position defined as the relative position described in step 605 of FIG. To express. For example, “center / center / center” indicates that the arm is in the face position (mouth position) in the three directions of right and left, up and down, and depth.

  Further, “medicine” and “cup” indicate the positions of the objects defined by the object table of FIG. In this example, the medicine is in the “right / bottom / near” position, that is, to the right of the face, below the face, and in front of the face, and the cup is in the “center / bottom / near” position, that is, left and right. In direction, it matches the face and is below the face and in front of the face. Assume that the positions of these objects have been acquired in advance from the video of the wide-field camera 215, prior knowledge, past video history, or other sensor information. The table in FIG. 8 can be updated at any time according to the position of the object.

The required time in FIG. 7 represents the time during which the arm position continues at each stage. For example, the time for which the arm is at the position of the medicine is set to 2 seconds, and the time for moving from the medicine to the cup is set to 3 seconds.
The relative position estimated by the action recognition process of FIG. 6 from the video taken as an action of taking medicine is recorded in time series, and the recorded position is associated with each stage of the action, so that the action of FIG. It is also possible to generate pattern information.

FIG. 9 is a flowchart illustrating an example of an action recognition process using a recognition table and an object table. The action estimation device 205 first initializes the stage of the action to be recognized to 0 (step 901), and acquires the arm position at the current stage from the recognition table (step 902). If the acquired position is an object, the position of the object is acquired from the object table (step 903).

  Next, the posture of the person is estimated by the processing of steps 601 to 605 in FIG. 6 (step 905), and it is checked whether or not the estimated arm position matches the acquired arm position (step 906).

  If the estimated arm position matches the acquired arm position (step 906, YES), the current stage is incremented by 1 (step 907), and whether or not the current stage has reached the end stage is determined. Check (step 908). In the case of FIG. 7, stage “5” corresponds to the end stage. If the current stage has not reached the end stage (step 908, NO), the current time is recorded (step 909), and the processes after step 902 are repeated.

  On the other hand, if the estimated arm position does not match the acquired arm position (step 906, NO), the required time for the current stage is acquired from the recognition table, and the required time has elapsed since the current stage. It is checked whether or not (step 910).

  If the required time has not elapsed (step 910, NO), the processing after step 903 is repeated, and if the required time has elapsed (step 910, YES), the processing after step 901 is repeated.

  When the current stage reaches the end stage (step 908, YES), the process is terminated. At this time, the action pattern action registered in the recognition table is output to the host device 206 as a recognition result. In the case of FIG. 7, the action of taking medicine is output as a recognition result.

  The motion recognition apparatus of FIGS. 1 and 2 can be applied to a life support robot that recognizes a person's action and supports life. As a result, it becomes possible to provide services such as recognizing the act of the subject person taking medicine, and providing support for preventing duplicate medications and forgetting to take medication.

  Note that the act of taking medicine is merely an example of a recognition target, and the action patterns of various actions can be defined by changing the contents of the recognition table and the object table. In the motion recognition apparatus, one or a plurality of motion patterns are registered in advance according to the application.

  Furthermore, in addition to the movement pattern of the arm, it is also possible to expand the range of the recognition target by detecting the movement of the object around the person. For example, when recognizing the act of opening or closing a door or furniture, the optical flow of the door or furniture is calculated using a wide-area image, and the opening or closing is detected by collating with a pre-registered optical flow pattern. be able to.

  The camera control device 203, the image processing device 204, the action estimation device 205, and the host device 206 in FIG. 2 can be realized using, for example, an information processing device as shown in FIG. 10 includes a central processing unit (CPU) 1001, a memory 1002, an input device 1003, an output device 1004, an external storage device 1005, a medium drive device 1006, and a network connection device 1007, which are connected via a bus 1008. Are connected to each other.

The memory 1002 includes, for example, a read only memory (ROM), a random access memory (RAM), and the like, and stores a program and data used for operation recognition processing. For example, the CPU 1001 performs a motion recognition process by executing a program using the memory 1002.

  The input device 1003 is, for example, a keyboard, a pointing device, or the like, and is used for inputting instructions and information from an operator. The output device 1004 is, for example, a display, a printer, a speaker, and the like, and is used to output an inquiry to the operator and a processing result.

  The external storage device 1005 is, for example, a magnetic disk device, an optical disk device, a magneto-optical disk device, a tape device, or the like. The information processing apparatus stores programs and data in the external storage device 1005, and loads them into the memory 1002 for use as necessary. The external storage device 1005 is also used as a database that stores a recognition table and an object table.

  The medium driving device 1006 drives a portable recording medium 1009 and accesses the recorded contents. The portable recording medium 1009 is an arbitrary computer-readable recording medium such as a memory card, a flexible disk, an optical disk, and a magneto-optical disk. The operator stores programs and data in the portable recording medium 1009 and loads them into the memory 1002 for use as necessary.

  The network connection device 1007 is connected to a communication network and performs data conversion accompanying communication. The information processing apparatus receives programs and data from an external apparatus via the network connection apparatus 1007 as necessary, and loads them into the memory 1002 for use.

  FIG. 11 shows a method for providing a program and data to the information processing apparatus of FIG. Programs and data stored in the portable recording medium 1009 and the database 1111 of the external device 1101 are loaded into the memory 1002 of the information processing device 1102. The external device 1101 generates a carrier signal for carrying the program and data, and transmits the carrier signal to the information processing device 1102 via a transmission medium on the communication network. The CPU 1001 executes the program using the data and performs the above-described processing.

  Although the disclosed embodiments and their advantages have been described in detail, those skilled in the art will be able to make various changes, additions and omissions without departing from the scope of the invention as explicitly set forth in the claims. .

It is a lineblock diagram of the 1st operation recognition device. It is a block diagram of the 2nd operation | movement recognition apparatus. It is a figure which shows arrangement | positioning of the left camera and the right camera. It is a figure which shows a relative parallax. It is a flowchart of a gaze point control process. It is a flowchart of an action recognition process. It is a figure which shows a recognition table. It is a figure which shows an object table. It is a flowchart of an action recognition process. It is a block diagram of information processing apparatus. It is a figure which shows the method of providing a program and data.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Wide-field imaging | photography part 102 High-resolution imaging | photography part 103 Motion / face detection part 104 Gaze point control part 105 Posture estimation part 106 Action recognition part 201 Camera head 202 Pan / tilt control motor 203 Camera control apparatus 204 Image processing apparatus 205 Action estimation apparatus 206 Host device 211, 213 High resolution camera 212, 214 Convergence control motor 215 Wide field of view camera 301 Person 311 Face 312 Right arm 313 Left arm 321 Left camera 322 Right camera 401, 402 Image 411, 412, 413, 414, 421, 422, 423 424, 431, 432, 433, 441, 442 points 1001 CPU
1002 Memory 1003 Input device 1004 Output device 1005 External storage device 1006 Medium drive device 1007 Network connection device 1008 Bus 1009 Portable recording medium 1101 External device 1102 Information processing device 1111 Database

Claims (5)

First imaging means for imaging a person;
Second imaging means for imaging the person;
The image obtained by each of the first and second imaging means is divided into a plurality of small areas, and the small areas of each image recognized as having captured the same part of the person are associated with each other. Parallax calculation means for calculating the distance between corresponding points of the two attached small areas as parallax;
The parallax of the face area recognized as an image of the person's face is compared with the parallax of the skin color area recognized as an image of the skin color part other than the person's face, and the distance from the face area is first. Detecting means for detecting, as an arm region, a skin color region that is smaller than the threshold value and the difference in parallax from the face region is smaller than the second threshold value;
Recognizing means for recognizing the movement of the person's arm based on the relative position of the arm area with respect to the face area and the difference between the parallax of the face area and the parallax of the arm area; Motion recognition device.   The recognizing unit compares the difference between the coordinate value of the face region and the coordinate value of the arm region with a third threshold value in the image obtained by the first or second imaging unit, so that the face Determining the relative position of the arm region relative to the region and comparing the difference between the parallax of the face region and the parallax of the arm region with a fourth threshold value, so that The motion recognition apparatus according to claim 1, wherein a depth position is obtained, and the posture of the person is obtained from the obtained in-image position and depth position.   The image processing apparatus further comprises storage means for storing a time series pattern of a relative position in the image of the arm area with respect to the face area and a relative depth position of the arm with respect to the face. The position is recorded in time series, and the movement of the person is recognized by collating the time series pattern of the recorded in-image position and depth position with the time series pattern stored in the storage means. The motion recognition apparatus according to claim 2. Controlling the first and second imaging means to image a person,
Dividing an image obtained by each of the first and second imaging means into a plurality of small regions;
Recognizing that the same part of the person was imaged, the small areas of each image are associated with each other,
Calculating the distance between corresponding points of the two associated small areas as parallax,
Comparing the parallax of the face area recognized as an image of the person's face with the parallax of the skin color area recognized as an image of the skin color part other than the person's face;
Detecting a skin color area whose distance from the face area is smaller than a first threshold and whose parallax difference from the face area is smaller than a second threshold as an arm area;
A program for causing a computer to execute processing for recognizing the movement of the person's arm based on the relative position of the arm region with respect to the face region and the difference between the parallax of the face region and the parallax of the arm region . The first and second imaging means image a person,
Computer
Dividing an image obtained by each of the first and second imaging means into a plurality of small regions;
Recognizing that the same part of the person was imaged, the small areas of each image are associated with each other,
Calculating the distance between corresponding points of the two associated small areas as parallax,
Comparing the parallax of the face area recognized as an image of the person's face with the parallax of the skin color area recognized as an image of the skin color part other than the person's face;
Detecting a skin color area whose distance from the face area is smaller than a first threshold and whose parallax difference from the face area is smaller than a second threshold as an arm area;
A motion recognition method comprising recognizing a motion of the person's arm based on a relative position of the arm region with respect to the face region and a difference between the parallax of the face region and the parallax of the arm region.