JP2019053647A - Behavior estimation apparatus and behavior estimation program - Google Patents

Abstract

To provide a behavior estimation apparatus which can estimate behavior of a subject in a predetermined scene, and a behavior estimation program.SOLUTION: A behavior estimation includes: an acquisition unit which acquires depth information of an infrared camera (100); and a behavior state estimation unit which analyzes the depth information to detect a pose and an amount of displacement (108), and generates state tables for upper body and lower body on the basis of the above information to estimate a state (110). A behavior classifying unit classifies behaviors defined in advance in two axes, upper-body state of the subject and lower-body state of the subject, on the basis of the state tables, to estimate behavior of the subject from the estimated state of the subject (112).SELECTED DRAWING: Figure 11

Description

  The present invention relates to a behavior estimation device and a behavior estimation program.

  Patent Document 1 derives feature information that represents a feature of each behavior of a plurality of subjects based on behavior information that appears as behavior in each of a plurality of subjects, and an emotion element of a designated subject based on the feature information There has been proposed an information providing apparatus that obtains element information that represents and obtains and presents psychological information that represents the psychological state of the designated person based on the element information.

JP 2011-201121 A

  Although it is possible to present psychological information using behavior information that appears as behavior and to detect human movements, it is not possible to estimate the behavior of the subject in a predetermined scene such as a meeting, class, lecture, etc. could not. Then, an object of this invention is to provide the action estimation apparatus and action estimation program which can estimate the action of the subject in a predetermined scene.

  The behavior estimation apparatus according to claim 1 estimates a behavior of a target person in a predetermined scene using an acquisition unit that acquires motion information of one or more subjects and the motion information acquired by the acquisition unit. An estimation unit.

  According to a second aspect of the present invention, in the first aspect of the invention, the estimation unit estimates the state of the subject using the motion information acquired by the acquisition unit, and the subject from the estimated state The behavior in a predetermined scene is estimated.

  According to a third aspect of the present invention, in the second aspect of the present invention, the estimating unit classifies a predetermined action corresponding to the state of the subject in advance, and the behavior of the subject from the estimated state of the subject. Is estimated.

  The invention according to claim 4 is the invention according to claim 3, wherein the estimation unit classifies actions in advance in a biaxial region of a state of the upper half of the subject and a state of the lower half of the subject, The behavior of the subject is estimated by estimating each of the upper body state and the lower body state from the motion information.

  The invention according to claim 5 is the invention according to claim 4, wherein the region into which the behavior is classified has two or more overlaps, and the presence / absence of a predetermined specific element is defined as the classification condition in the overlap. To do.

  The invention according to claim 6 is the invention according to any one of claims 1 to 5, wherein the estimation unit is configured to determine the frequency of motion of the target person and the target person for each time interval determined in advance from the motion information. At least one subject-specific motion information of the operation maintenance time is obtained, and the subject-specific motion information is further used to estimate the behavior of the subject in a predetermined scene.

  The invention according to claim 7 is the invention according to any one of claims 1 to 6, wherein the estimation unit estimates an action of the subject for each predetermined time interval, and is estimated by the estimation unit. A storage unit is further provided for storing the behavior of the subject for each time interval.

  The invention according to claim 8 is the invention according to claim 7, wherein the storage unit further stores detection information in the predetermined scene other than the motion information for each time interval.

  The invention according to claim 9 is the invention according to claim 8, wherein the detection information is information including at least one of audio information for each time interval and shooting information for each time interval.

  The invention according to claim 10 is the invention according to any one of claims 6 to 9, wherein the time interval is changed to a predetermined time according to the operation speed of the subject.

  An action estimation program according to an eleventh aspect causes a computer to function as each unit of the behavior estimation apparatus according to any one of the first to tenth aspects.

  According to the behavior estimation device of the first aspect, it is possible to provide a behavior estimation device capable of estimating the behavior of the subject in a predetermined scene.

  According to the invention described in claim 2, it is possible to estimate the behavior of the subject in a predetermined scene.

  According to invention of Claim 3, it becomes possible to estimate action from a subject's state.

  According to the fourth aspect of the present invention, it is possible to easily estimate the behavior as compared with the case where the behavior is estimated from the state without being divided into two axes of the upper body state and the lower body state.

  According to the fifth aspect of the present invention, it is possible to estimate the action from the state of the target person even when the areas where the actions are classified overlap.

  According to the sixth aspect of the present invention, it is possible to accurately estimate the action compared to the case where the subject-specific action is not taken into consideration.

  According to the seventh aspect of the present invention, it is possible to search for each time interval using behavior as a keyword.

  According to the eighth aspect of the invention, it is possible to search for not only behavior but also information related to behavior for each time interval.

  According to the ninth aspect of the present invention, it is possible to search for at least one of audio information and shooting information in a time interval corresponding to the action using the action as a keyword.

  According to the tenth aspect of the present invention, it is possible to accurately estimate the action compared to the case where the time interval is a fixed time.

  According to the invention described in claim 11, it is possible to provide a behavior estimation program capable of estimating the behavior of the subject in a predetermined scene.

It is a block diagram which shows schematic structure of the action estimation apparatus which concerns on this embodiment. It is a figure which shows an example of the conference room provided with the action estimation apparatus which concerns on this embodiment. It is a functional block diagram which shows the function of the control apparatus of the action estimation apparatus which concerns on this embodiment. (A) is a figure which shows an example of the diffraction part 25 points | pieces of a human body, (B) is a figure which shows an example of the position and displacement of a diffraction part. It is a figure which shows an example of a motion of each person's upper body and a lower body when the person A talks to the person B with a gesture, and the person C is recording. (A) is a figure which shows an example of the state table of an upper body, (B) is a figure which shows an example of the state table of a lower body. It is a figure which shows the example which classified beforehand the action predetermined in the biaxial area | region of a subject's upper body state and a subject's lower body state. It is a figure which shows an example of the specific element for every action. It is a figure which shows an example of the relationship between a specific element and an overlap part. It is a figure for demonstrating an example of the operation | movement analysis using the information of an infrared camera, a microphone, and a camera in the case of explaining while writing on a board after standing up and moving to a whiteboard device. It is a flowchart which shows an example of the flow of the action estimation process performed with the control apparatus of the action estimation apparatus which concerns on this embodiment. It is a flowchart which shows an example of the flow of a depth information analysis process. It is a flowchart which shows an example of the flow of an overlap determination process. It is a flowchart which shows the flow of the overlap determination process of overlap area | region R1. It is a flowchart which shows the flow of the overlap determination process of overlap area | region R2. It is a flowchart which shows the flow of the overlap determination process of overlap area | region R3. It is a flowchart which shows the flow of the overlap determination process of overlap area | region R4. It is a flowchart which shows the flow of the overlap determination process of overlap area | region R5. It is a flowchart which shows the flow of the overlap determination process of overlap area | region R6. It is a flowchart which shows the flow of the overlap determination process of the overlap area | region R7.

  Hereinafter, an example of an embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram illustrating a schematic configuration of a behavior estimation apparatus 10 according to the present embodiment. Moreover, FIG. 2 is a figure which shows an example of the conference room provided with the action estimation apparatus 10 which concerns on this embodiment.

  The behavior estimation apparatus 10 according to the present embodiment is a target person who participates in a predetermined scene such as a meeting, a meeting, a lecture, or a class held in a predetermined space such as the conference room, lecture hall, or classroom shown in FIG. Estimate behavior. Here, the action of the subject in the present embodiment is not a simple movement of a person (such as movement of a line of sight, movement of a head, movement of a hand, movement of a leg, or the like), but includes a combination of a plurality of movements. This is an action in a scene. As an example, in this embodiment, a person's action during a meeting is estimated. Specifically, in the present embodiment, the estimated behavior of the person during the meeting is “explanation”, “question / opinion”, “presentation”, “thought / contemplation”, “note / employment”, and “appeal”. It is classified into these six types, and it is estimated which action it is.

  The behavior estimation device 10 includes a control device 20 as an estimation unit that acquires various types of information and estimates a subject's behavior.

  The control device 20 includes a microcomputer in which a CPU (Central Processing Unit) 20A, a ROM (Read Only Memory) 20B, a RAM (Random Access Memory) 20C, and an input / output port 20D are connected to a bus 20E.

  The ROM 20B stores various programs such as a behavior estimation program for estimating the behavior of the target person who participated in the conference. The program stored in the ROM 20B is expanded in the RAM 20C and executed by the CPU 20A, whereby the behavior of the subject is estimated. Alternatively, a program stored in advance in an HDD (Hard Disc Drive) 20F may be expanded in the RAM 20C and executed by the CPU 20A to estimate the target person's action.

  The input / output port 20D is connected to the infrared camera 12, the camera 14, the microphone 16, the database DB18 as a storage unit, the display device 22, the whiteboard device 24, and the hand operating device 26.

  For example, as shown in FIG. 2, the infrared camera 12, the camera 14, and the microphone 16 are provided in a conference room.

  The infrared camera 12 is provided, for example, on the ceiling or wall of a conference room, and outputs depth information obtained by photographing the state during the conference to the control device 20 by photographing light in the infrared region.

  Similarly to the infrared camera 12, the camera 14 is provided on, for example, a ceiling or a wall of a conference room, and outputs captured image information obtained by photographing a state during the conference to the control device 20.

  The microphone 16 is provided, for example, on a desk or a wall surface of a conference room, acquires voice during a conference in the conference room as voice information, and outputs it to the control device 20.

  The database DB 18 stores various information such as depth information of the infrared camera 12, captured image information of the camera 14, and audio information of the microphone 16 together with time information.

  For example, a display device such as a projector device is applied to the display device 22, displays a document during a meeting, and outputs the display state of the display device 22 to the control device 20 as material presentation information together with time information.

  The whiteboard device 24 detects the blackboard information during the meeting, and outputs the detection result to the control device 20 as the blackboard information together with the time information.

  For example, a device such as an electronic pen, a user-registered computer, or a tablet is applied to the hand operation device 26, and an operation state is detected and the detection result is output to the control device 20 as hand operation information together with time information.

  And the control apparatus 20 estimates the action of the subject who participated in the meeting based on the acquired information, stores the estimated action in the database DB 18 together with the time information, and later captures images taken during the meeting. Searchable.

  FIG. 3 is a functional block diagram illustrating functions of the control device 20 of the behavior estimation device 10 according to the present embodiment.

  As shown in FIG. 3, the behavior estimation device 10 includes an acquisition unit 30 and a behavior estimation unit 40 as an estimation unit as functions.

  The acquisition unit 30 acquires depth information obtained from the infrared camera 12 as operation information of one or more subjects, and acquires captured image information obtained from the camera 14 and audio information obtained from the microphone 16. The acquisition unit 30 also acquires information from the display device 22, the whiteboard device 24, the hand operating device 26, and the like.

  The acquisition unit 30 includes a material presentation detection unit 32, a board writing detection unit 34, an utterance detection unit 36, and a hand operation detection unit 38. The material presentation detection unit 32 detects display of material during the meeting by acquiring display information from the display device 22. Further, the board writing detection unit 34 detects a board writing during a meeting from information of the whiteboard device 24, a photographed image of the camera 14, or the like. Further, the utterance detection unit 36 detects the utterance (voice) during the conference from the voice information of the microphone 16. Further, the hand operation detection unit 38 detects the hand operation during the meeting based on the information of the hand operation device 26. And the acquisition part 30 stores the detection result of each part in database DBa18A.

  The behavior estimation unit 40 includes functions of a media recording unit 42, an operation state estimation unit 44, a behavior classification unit 46, and an overlap determination unit 48. The behavior estimation unit 40 estimates the behavior of the target person who participated in the conference, and stores it in the database DBb 18B. Stores estimated behavior information.

  The media recording unit 42 stores each of depth information, captured image information, and audio information acquired by the acquisition unit 30 in the database DBb 18B together with time information.

  The motion state estimation unit 44 detects the motions of one or more subjects using the depth information acquired by the acquisition unit 30, and estimates the status of the subjects from the detected motions. The motion of the subject is detected using a well-known technique such as OpenCV (Open Source Computer Vision) that measures the motion (motion, posture, movement, etc.) of a person using depth information obtained from the infrared camera 12. For example, in the product of Microsoft Corporation, Kinect (trademark) for Windows (registered trademark), as shown in FIG. 4 (A), 25 diffracting parts of the human body are specified, and as shown in FIG. Since the position and displacement (for example, θ in FIG. 4B) are measured, the movement of the person is detected using this. More specifically, an infinite number of infrared patterns are irradiated, and the pattern irradiated to the subject is observed with an infrared camera. The depth from the pattern projection position to the subject is measured to detect the person area. After that, the enormous human posture pattern learned in advance is compared with the observed human region, and the three-dimensional position of each body part is accurately measured. In addition, for example, when the person A talks to the person B together with the gesture and the person C records, the person in the meeting, as shown in FIG. b, and movements of arms, hands c) and lower body (posture change d, walking e in FIG. 5). Therefore, the motion state estimation unit 44 estimates the state of the upper body and the state of the lower body from the motion detected based on the depth information. As the state to be estimated, for example, as shown in FIG. 6A, the upper body state is estimated as an example of a state of resting, moving hand, moving head, moving arm, walking or the like. Further, as shown in FIG. 6B, as the state of the lower body, states such as stillness, posture change, middle waist, and walking are estimated as an example. Specifically, the upper body state table shown in FIG. 6 (A) and the lower body state table shown in FIG. 6 (B) using the poses and displacement amounts recognized based on the movements of the joints obtained from the depth information. The state is estimated by generating. A value in a predetermined range is input to VuX and VlX in the state table shown in FIG. In the present embodiment, a plurality of states such as nodding are not estimated while moving the hand, but a single state such as moving or nodding is estimated, and any one of the state tables in FIG. A case where a motion (numerical value) enters only one state will be described.

  The behavior classification unit 46 classifies the behavior of the target person using the estimation result of the motion state estimation unit 44. For example, as shown in FIG. 7, predetermined actions are classified in advance in a biaxial region of the subject's upper body state and the subject's lower body state, and the subject's behavior is estimated from the estimated subject's state. Is estimated. In the example of FIG. 7, the upper body state is the vertical axis, and the lower body state is the horizontal axis. Then, categorize the areas into six types of actions: “explanation”, “question / opinion”, “presentation”, “thought and meditation”, “note / internal employment”, and “appeal”. Estimate behavior.

  When the behavior classification unit 46 estimates the behavior, the overlap determination unit 48 determines the behavior for the overlapping portion (R1 to R7 in FIG. 7) of each behavior. In the present embodiment, the behavior of the overlapping portion is determined on the condition of the presence or absence of a predetermined specific element. As a specific element, in the present embodiment, the overlapping portion is determined on the condition that the material presentation, the board writing, the voice, and the presence / absence of the hand operation detected from the information acquired by the acquisition unit 30 are used. For example, as illustrated in FIG. 8, the “description” action includes a blackboard and a voice as specific elements. In the “question / opinion” action, sound is a specific element. In the action of “presentation”, specific elements include voice, presentation of material, and board writing. There is no specific element in the action of “thought and meditation”. In the action of “note / inside job”, there is a hand operation as a specific element. In the action of “appeal”, there is a voice as a specific element. Therefore, based on the relationship between the specific element and the overlapping portion shown in FIG. 9, the action is determined for the overlapping portion of R1 to R7 on the condition of the presence or absence of each element (C1 to C4).

  Further, the behavior estimation unit 40 stores the estimated behavior together with time information in the database DBb 18B so that a photographed image can be searched based on the behavior during the meeting. Here, the behavior estimation unit 40 uses the infrared camera 12, the microphone 16, and the camera 14 to detect information related to the state of the target person, so that these are used and determined in advance by the behavior classification unit 46 and the overlap determination unit 48. The behavior of the target person may be estimated by performing motion analysis for each time interval. For example, in the case of standing up (standing up), moving to the whiteboard device 24 (walking movement), and explaining while writing on the board (board writing), as shown in FIG. 16 and the information of the camera 14 are obtained, and the behavior of the subject person is estimated by performing an operation analysis for each time interval using each of the information. And the estimated action is stored in database DBb18B with time information. In the example of FIG. 10, Δt1, Δt3, and Δt4 are noises, Δt2 is “question / opinion”, and Δt5 and Δt6 are “explanation”, respectively.

  Further, the behavior estimation unit 40 obtains at least one subject-specific motion of the subject's motion frequency and the subject's motion maintenance time for each predetermined time interval from the motion information, and further performs the subject-specific motion. It may be used to estimate the behavior of the subject in a predetermined scene. For example, in order to take into account the habit of the subject, the frequency of the subject's motion and the duration of the motion may be obtained as the subject-specific behavior and used for behavior estimation. Specifically, people who frequently nod may be nodding when estimating their status because they may be in a nodding state that is different from the behavior of `` question / opinion '' or the behavior of `` note / internal employment ''. You may utilize for the estimation of the state at the time of estimating action like not judging.

  Note that the predetermined time interval for estimating the behavior may be a predetermined fixed time interval or a time interval that varies according to a predetermined condition. For example, when applying a time interval of fluctuation, the time may be changed according to the operation speed of the subject. Specifically, since the longer the time is required to estimate the behavior as the operation speed of the subject is lower, the longer the time interval is, the lower the operation speed of the subject is, and the shorter the operation speed of the subject is, the shorter Time interval.

  Then, the specific process performed with the control apparatus 20 of the action estimation apparatus 10 which concerns on this embodiment comprised as mentioned above is demonstrated. FIG. 11 is a flowchart illustrating an example of a flow of behavior estimation processing performed by the control device 20 of the behavior estimation device 10 according to the present embodiment.

  First, in step 100, the acquisition unit 30 acquires various information and proceeds to step 102. That is, the depth information obtained from the infrared camera 12, the captured image of the camera 14, the sound information of the microphone 16, the display state of the display device 22, the writing information of the whiteboard device 24, and the operation state of the hand operating device 26 are acquired. To do.

  In step 102, the acquisition unit 30 determines whether the acquired information is depth information of the infrared camera 12. If the determination is negative, the process proceeds to step 104, and if the determination is affirmative, the process proceeds to step 108.

  In step 104, the board writing detection unit 34 detects the board writing information from the photographed image of the camera 14, the board writing information of the whiteboard device 24, and the like, and the speech detection unit 36 detects the voice information in the photographed image of the camera 14 and the microphone 16. The speech information is detected from the voice information, and the process proceeds to step 106.

  In step 106, the acquisition unit 30 records the detected detection information together with the corresponding time information in the database DBa 18A, returns to step 100, and repeats the above-described processing.

  On the other hand, in step 108, the motion state estimation unit 44 performs depth information analysis processing and proceeds to step 110. Here, the depth information analysis processing will be described. FIG. 12 is a flowchart illustrating an example of the flow of depth information analysis processing.

  When the process proceeds to the depth information analysis process, in step 200, the motion state estimation unit 44 extracts a joint coordinate group having a displacement and proceeds to step 202.

  In step 202, the motion state estimation unit 44 calculates a scalar and an inner product using the coordinates of any two adjacent joints as a vector, and proceeds to step 204.

  In step 204, the motion state estimation unit 44 calculates an angle θ as the displacement amount of the two joints having the displacement, and proceeds to step 206.

  In step 206, the operation state estimation unit 44 updates the counter and timer, and proceeds to step 208.

  In step 208, the operation state estimation unit 44 determines whether a predetermined number of times or a predetermined time has elapsed. If the determination is negative, the process returns to step 200 and the above-described processing is repeated. If the determination is affirmative, the process proceeds to step 210. The predetermined time interval described above is applied as the predetermined time. That is, a predetermined time may be applied as the predetermined time, or a time that varies according to a predetermined condition may be applied. Similarly, a predetermined fixed number of times may be applied to the predetermined number of times, or a number of times that fluctuates according to a predetermined condition may be applied. For example, the number of times may be increased as the operation speed of the subject is lower, and the number may be decreased as the operation speed of the subject is increased.

  In step 210, the motion state estimation unit 44 recognizes a person's pose (state) from the calculated displacement of θ, and proceeds to step 212.

  In step 212, the motion state estimation unit 44 outputs the recognized pose and the calculated displacement amount θ, and proceeds to step 110 in FIG.

  Subsequently, in step 110, the motion state estimation unit 44 creates a state table for each of the upper body and lower body based on the pose and displacement obtained by the depth information analysis process, estimates the state, and proceeds to step 112. Transition. That is, the state table shown in FIGS. 6A and 6B is created from the movement of the person detected from the depth information. In the present embodiment, the state table is created when there is movement only in a portion corresponding to a single state. If there is movement for multiple states, the state table shall be recreated.

  In step 112, the behavior classification unit 46 estimates the behavior of the subject person and proceeds to step 114. Specifically, based on the created state table, as shown in FIG. 7, pre-classified and estimated behaviors predetermined in two axes, the state of the subject's upper body and the state of the subject's lower body The behavior of the subject is estimated from the state of the subject.

  In step 114, the overlap determination unit 48 determines whether or not the action is an overlap region. In this determination, it is determined whether or not the upper and lower body states correspond to the overlapping regions (R1 to R7) in FIG. If the determination is negative, the process proceeds to step 118, and if the determination is affirmative, the process proceeds to step 116.

  In step 116, the overlap determination unit 48 performs an overlap determination process and proceeds to step 118. Here, the overlap determination process will be described in detail. FIG. 13 is a flowchart illustrating an example of the flow of the overlap determination process.

  In step 300, the overlap determination unit 48 acquires the detection information of the overlap occurrence time from the DBa 18A. That is, detection information (information acquired by the acquisition unit 30 in step 106 and recorded in the DBa 18A) at the time when the upper and lower body states of the overlapping region are detected is acquired from the database DBa 18A. The acquired detection information acquires each detection information of presentation of material during the meeting, board writing during the meeting, utterance during the meeting, and hand operation during the meeting.

  In step 302, the overlap determination unit 48 classifies which of the overlap regions R1 to R7, and proceeds to step 304.

  In step 304, the overlap determination unit 48 performs an overlap determination process for each overlap region, and proceeds to step 118 in FIG. Here, the overlap determination process for each overlap region will be described in detail. The overlap determination for each overlap region is performed according to the overlap region, and the action of the overlap region is determined on the condition that there is a predetermined specific element shown in FIGS. 14 to 20 are flowcharts showing the flow of the overlap determination process in each of the overlap regions R1 to R7.

  First, when the overlap region is R1, in step 310, the overlap determination unit 48 determines whether or not the specific element is the board C2 and the sound C3 based on the information acquired in step 300. If the determination is affirmative, the process proceeds to step 312, and if the determination is negative, the process proceeds to step 314.

  In step 312, the overlap determination unit 48 determines that the action of the overlap region is “explanation”.

  On the other hand, in step 314, the overlap determination unit 48 determines whether or not the specific element is only the voice C3 based on the information acquired in step 300. If the determination is affirmative, the process proceeds to step 316, and if the determination is negative, the process proceeds to step 318.

  In step 316, the overlap determination unit 48 determines that the action of the overlap region is “question / opinion”.

  In step 318, the overlap determination unit 48 determines that the behavior of the overlap region is not specified.

  Next, when the overlapping area is R2, in step 320, based on the information acquired in step 300, whether or not the specific element is the material presentation C1, the board writing C2, and the voice C3 is determined. judge. If the determination is affirmative, the process proceeds to step 322, and if the determination is negative, the process proceeds to step 324.

  In step 322, the overlap determination unit 48 determines that the action of the overlap region is “presentation”.

  On the other hand, in step 324, the overlap determination unit 48 determines whether or not the specific element is only the voice C3 based on the information acquired in step 300. If the determination is affirmative, the process proceeds to step 326. If the determination is negative, the process proceeds to step 328.

  In step 326, the overlap determination unit 48 determines that the behavior of the overlap region is “question / opinion”.

  In step 328, the overlap determination unit 48 determines that the behavior of the overlap region is not specified.

  Next, when the overlap region is R3, in step 330, the overlap determination unit 48 determines whether or not the specific element is the board C2 and the sound C3 based on the information acquired in step 300. If the determination is affirmative, the process proceeds to step 332; if the determination is negative, the process proceeds to step 334.

  In step 332, the overlap determination unit 48 determines that the action of the overlap region is “explanation”.

  On the other hand, in step 334, the overlap determination unit 48 determines whether or not the specific element has no detection information based on the information acquired in step 300. If the determination is affirmative, the process proceeds to step 336, and if the determination is negative, the process proceeds to step 338.

  In step 336, the overlap determination unit 48 determines that the action of the overlap region is “thought / contemplation”.

  In step 338, the overlap determination unit 48 determines that the behavior of the overlap region is not specified.

  Next, when the overlap region is R4, in step 340, the overlap determination unit 48 determines whether the specific element is only the voice C3 based on the information acquired in step 300. If the determination is affirmative, the process proceeds to step 342. If the determination is negative, the process proceeds to step 344.

  In step 342, the overlap determination unit 48 determines that the behavior of the overlap region is “question / opinion”.

  On the other hand, in step 344, the overlap determination unit 48 determines whether the specific element is only the hand operation C4 based on the information acquired in step 300. If the determination is affirmative, the process proceeds to step 346, and if the determination is negative, the process proceeds to step 348.

  In step 346, the overlap determination unit 48 determines that the action of the overlap region is “note / internal work”.

  In step 348, the overlap determination unit 48 determines that the behavior of the overlap region is not specified.

  Next, when the overlap region is R5, whether or not the specific elements are the material presentation C1, the board writing C2, and the voice C3 based on the information acquired in step 300 by the overlap determination unit 48 in step 350. judge. If the determination is affirmative, the process proceeds to step 352, and if the determination is negative, the process proceeds to step 354.

  In step 352, the overlap determination unit 48 determines that the action of the overlap region is “presentation”.

  On the other hand, in step 354, the overlap determination unit 48 determines whether or not the specific element is only the voice C3 based on the information acquired in step 300. If the determination is affirmative, the process proceeds to step 356, and if the determination is negative, the process proceeds to step 358.

  In step 356, the overlap determination unit 48 determines that the action of the overlap region is “appeal”.

  In step 358, the overlap determination unit 48 determines that the behavior of the overlap region is not specified.

  Next, when the overlap region is R6, in step 360, the overlap determination unit 48 determines whether or not the specific element has no detection information based on the information acquired in step 300. If the determination is affirmative, the process proceeds to step 362. If the determination is negative, the process proceeds to step 364.

  In step 362, the overlap determination unit 48 determines that the action of the overlap region is “thought / contemplation”.

  On the other hand, in step 364, the overlap determination unit 48 determines whether the specific element is only the hand operation C4 based on the information acquired in step 300. If the determination is affirmative, the process proceeds to step 366, and if the determination is negative, the process proceeds to step 368.

  In step 366, the overlap determination unit 48 determines that the action of the overlap region is “note / internal work”.

  In step 368, the overlap determination unit 48 determines that the behavior of the overlap region is not specified.

  Next, when the overlap region is R7, in step 370, the overlap determination unit 48 determines whether the specific element is only the hand operation C4 based on the information acquired in step 300. If the determination is affirmative, the process proceeds to step 372. If the determination is negative, the process proceeds to step 374.

  In step 372, the overlap determination unit 48 determines that the action of the overlap region is “note / internal work”.

  On the other hand, in step 374, the overlap determination unit 48 determines whether or not the specific element is only the voice C3 based on the information acquired in step 300. If the determination is affirmative, the process proceeds to step 376, and if the determination is negative, the process proceeds to step 378.

  In step 376, the overlap determination unit 48 determines that the action of the overlap region is “appeal”.

  In step 378, the overlap determination unit 48 determines that the behavior of the overlap region is not specified.

  Then, when the overlap determination process for each overlap region is performed by any of the above, the process of FIG. 13 is returned and the process proceeds to step 118 of FIG.

  In step 118, the behavior estimation unit 40 records the estimated behavior together with time information in the database DBb 18B, returns to step 100, and repeats the above processing. As a result, the database DB 18 records the estimated behavior along with the time information, and stores information about the conference such as the captured image information and the audio information.

  The behavior of the target person during the conference estimated in this way is analyzed in real time during the conference to detect the stagnation of the discussion (for example, detection of a relatively long noise section). When the discussion is stagnant, the analysis result up to that point may be presented to prompt the next action.

  Further, the behavior of the conference participants may be analyzed after the entire recording time of the conference, and may be used to visualize and present the communication structure by removing the noise section and the behavior classification.

  In addition, in order to search the conference information recorded in the database DB 18 using a keyword representing the behavior such as “board writing”, for example, by analyzing the conference participants' behavior after the entire recording time of the conference. May be used. For example, it searches for a board at that meeting and uses it to search for records of discussions that led to this board.

  In the above embodiment, the state table in FIG. 6 has been described as having movement (numerical values) in only one of the states. However, the present invention is not limited to this. For example, when the state table shown in FIG. 6 is generated from the motion information, a case where a motion (numerical value) enters each of a plurality of states may be applied. In this case, for example, the state may be estimated by adopting the state with the largest movement.

  In the above embodiment, as illustrated in FIG. 7, an example in which two actions are overlapped in a biaxial region has been described. However, three or more actions may be classified.

  Moreover, the process (FIGS. 11-20) performed by the control apparatus 20 of the behavior estimation apparatus 10 according to the above embodiment may be a process performed by software, or may be a process performed by hardware. It is good also as processing which combined both. Further, the processing performed by each unit of the control device 20 may be stored and distributed as a program in a storage medium.

  Further, the present invention is not limited to the above, and it is needless to say that various modifications can be made without departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 10 Action estimation apparatus 12 Infrared camera 14 Camera 16 Microphone 18 Database DB
18A Database DBa
18B Database DBb
DESCRIPTION OF SYMBOLS 20 Control apparatus 30 Acquisition part 32 Material presentation detection part 34 Board writing detection part 36 Speech detection part 38 Hand operation detection part 40 Action estimation part 42 Media recording part 44 Motion state estimation part 46 Action classification part 48 Overlap determination part

Claims (11)

An acquisition unit for acquiring operation information of one or more subjects,
An estimation unit that estimates an action in a predetermined scene of the subject using the operation information acquired by the acquisition unit;
A behavior estimation device comprising:   The behavior estimation according to claim 1, wherein the estimation unit estimates the state of the target person using the motion information acquired by the acquisition unit, and estimates a behavior of the target person in a predetermined scene from the estimated state. apparatus.   The behavior estimation apparatus according to claim 2, wherein the estimation unit classifies a predetermined behavior corresponding to the state of the subject in advance and estimates the behavior of the subject from the estimated state of the subject.   The estimation unit pre-classifies actions in a biaxial region of the subject's upper body state and the subject's lower body state, and estimates each of the upper body state and the lower body state from the motion information The behavior estimation apparatus according to claim 3, wherein the behavior of the subject is estimated.   The behavior estimation apparatus according to claim 4, wherein the region in which the behavior is classified has two or more overlaps, and the presence / absence of a predetermined specific element is a condition for classification in the overlap.   The estimation unit obtains target person-specific motion information of at least one of a target person's motion frequency and a target person's motion maintenance time for each predetermined time interval from the motion information, and further includes the target person-specific motion information. The behavior estimation apparatus according to any one of claims 1 to 5, wherein the behavior estimation apparatus uses the subject to estimate a behavior in a predetermined scene.   The said estimation part is further provided with the memory | storage part which estimates a subject's action for every predetermined time interval, and memorize | stores the subject's action for every said time interval estimated by the said estimation part. The behavior estimation apparatus according to any one of the above.   The behavior estimation apparatus according to claim 7, wherein the storage unit further stores detection information in the predetermined scene other than the motion information for each time interval.   The behavior estimation apparatus according to claim 8, wherein the detection information is information including at least one of audio information for each time interval and shooting information for each time interval.   The behavior estimation apparatus according to any one of claims 6 to 9, wherein the time interval is changed to a predetermined time according to the operation speed of the subject.   The action estimation program for functioning a computer as each part of the action estimation apparatus of any one of Claims 1-10.