JP7224569B2 - Information processing device, estimation method, and estimation program - Google Patents

Description

The present disclosure relates to an information processing device, an estimation method, and an estimation program.

In recent years, it has been desired to control devices according to human conditions. For example, human behavior or emotions are indicative of human situations. For example, Patent Literature 1 describes that emotions, intentions of speech, etc. can be estimated from movements of the faces and hands obtained by tracking the positions of the hands in chronological order.

JP 2007-058615 A

As described above, when a body part such as a hand or a leg is largely moved, it is considered that an emotion is estimated. On the other hand, when the motion is small, estimation cannot be performed according to the above considerations.

The purpose of this disclosure is to estimate emotion based on small movements.

An information processing device according to one aspect of the present disclosure is provided. An information processing apparatus includes an acquisition unit that acquires a plurality of images that include a user and that are captured at different times, and extracts a plurality of pieces of skeletal information representing skeletal feature points of the user based on the plurality of images. an extraction unit that calculates the amount of movement of the user in a preset period based on the plurality of pieces of skeleton information; and a determination unit that determines whether the amount of movement is equal to or less than a preset threshold. and an estimation unit for estimating that the user's feeling is unpleasant when the amount of motion is equal to or less than the threshold.

According to the present disclosure, emotions can be estimated based on small movements.

It is a figure which shows the control system of embodiment. It is a figure which shows the specific example of control of the control object apparatus of embodiment. 3 is a block diagram showing functions of the information processing apparatus according to the embodiment; FIG. FIG. 4 is a diagram showing an example of skeleton information according to the embodiment; FIG. It is a figure which shows the example of the graph of embodiment. 4 is a flow chart showing an example of processing executed by the information processing apparatus according to the embodiment;

Embodiments will be described below with reference to the drawings. The following embodiments are merely examples, and various modifications are possible within the scope of the present disclosure.

Embodiment.
FIG. 1 is a diagram showing a control system according to an embodiment. The control system includes an information processing device 100 , an imaging device 200 and a controlled device 300 . The information processing device 100, imaging device 200, and controlled device 300 communicate via a network.
The information processing device 100 is a device that executes an estimation method. The information processing device 100 may be called a control device or an emotion estimation device.

First, hardware included in the information processing apparatus 100 will be described. The information processing device 100 has a processor 101 , a volatile memory device 102 , a nonvolatile memory device 103 and a communication interface 104 .

The processor 101 controls the information processing apparatus 100 as a whole. For example, the processor 101 is a CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), or the like. Processor 101 may be a multiprocessor. Further, the information processing device 100 may have a processing circuit. The processing circuit may be a single circuit or multiple circuits.

The volatile memory device 102 is the main memory device of the information processing device 100 . For example, the volatile memory device 102 is RAM (Random Access Memory). The nonvolatile storage device 103 is an auxiliary storage device of the information processing device 100 . For example, the nonvolatile memory device 103 is a HDD (Hard Disk Drive) or an SSD (Solid State Drive).
The communication interface 104 communicates with the imaging device 200 and the control target device 300 .

For example, the imaging device 200 is a surveillance camera, a DMS (Driver Monitor System), or an infrared camera. The imaging device 200 images the user.
The information processing device 100 acquires a plurality of images (that is, videos) from the imaging device 200 . Information processing apparatus 100 estimates a user's emotion based on a plurality of images. The information processing device 100 can control the control target device 300 based on the estimated emotion information. Here, an example of control of the controlled device 300 is shown.

FIG. 2 is a diagram illustrating a specific example of control of a device to be controlled according to the embodiment. FIG. 2 shows an elevator hoist 301 as an example of the device 300 to be controlled. Also shown in FIG. 2 is a top view of the user making small movements.
The imaging device 200 is installed on the ceiling. The imaging device 200 images the user. The information processing device 100 acquires multiple images from the imaging device 200 . Information processing apparatus 100 estimates a user's emotion based on a plurality of images. The information processing device 100 controls the elevator 301 based on the estimated emotion information.

Next, functions of the information processing apparatus 100 will be described.
FIG. 3 is a block diagram illustrating functions of the information processing apparatus according to the embodiment. The information processing apparatus 100 has a storage unit 110 , an acquisition unit 120 , an extraction unit 130 , an operation amount calculation unit 140 , a determination unit 150 and an estimation unit 160 .

The storage unit 110 may be implemented as a storage area secured in the volatile storage device 102 or the nonvolatile storage device 103 .
A part or all of the acquisition unit 120, the extraction unit 130, the motion amount calculation unit 140, the determination unit 150, and the estimation unit 160 may be realized by a processing circuit. Part or all of the acquisition unit 120, the extraction unit 130, the motion amount calculation unit 140, the determination unit 150, and the estimation unit 160 may be implemented as modules of a program executed by the processor 101. FIG. For example, the program executed by processor 101 is also called an estimation program. For example, the estimation program is recorded on a recording medium.

The acquisition unit 120 acquires a plurality of images captured at different times from the imaging device 200 . Acquisition unit 120 may acquire a plurality of images from an external device (for example, a cloud server). Each of the plurality of images is an image obtained by imaging the user at different times. Therefore, each of the multiple images includes the user.

The extraction unit 130 extracts multiple pieces of skeleton information based on multiple images. Each of the pieces of skeleton information is information indicating a skeleton feature point of the user. For example, the information indicating the user's skeletal feature points is coordinates. Hereinafter, the information indicating the skeleton feature points of the user is assumed to be coordinates.

First, a case will be described in which one piece of skeleton information is extracted based on one image. Here, an example of skeleton information is shown.

FIG. 4 is a diagram showing an example of skeleton information according to the embodiment. FIG. 4 shows an image 400, which is one of a plurality of images. Extraction unit 130 extracts skeleton information 500 using a known technique. For example, known technologies are OpenPose or PoseNet. The skeletal information 500 is coordinates indicating skeletal feature points of the user. For example, coordinates 501 in FIG. 4 indicate skeletal feature points of the user's ear. Note that the coordinates are represented by an X coordinate and a Y coordinate.

FIG. 4 illustrates the case where one user is included in image 400 . When multiple users are included in the image, the extraction unit 130 extracts skeleton information of each of the multiple users.

In this way, the extraction unit 130 extracts one piece of skeleton information based on one image. Similarly, the extraction unit 130 performs skeleton information extraction processing on each of the plurality of images. As a result, a plurality of pieces of skeletal information are obtained.

Here, the control system may include a plurality of imaging devices. Each of the plurality of imaging devices is installed at a different position. Each of the plurality of imaging devices images the user from different directions. When a plurality of imaging devices are included in the control system, the acquisition unit 120 acquires a plurality of images obtained by imaging the user at the same time by the plurality of imaging devices. For example, the acquisition unit 120 acquires the plurality of images from a plurality of imaging devices. Also, the acquisition unit 120 acquires position information of each of the plurality of imaging devices. For example, the acquisition unit 120 acquires the position information of each of the multiple imaging devices from the storage unit 110 . The extraction unit 130 extracts a piece of skeleton information based on the plurality of images and the position information of each of the plurality of imaging devices. Here, for example, when an imaging device captures an image of a user from a certain direction, part of the user may not be captured. Therefore, another image capturing device captures an image of the user from a different direction, thereby capturing an image of the part. In this manner, the extracting unit 130 extracts skeleton information based on multiple images obtained by capturing images of the user from different directions using multiple imaging devices. Accordingly, the information processing apparatus 100 can obtain highly accurate skeleton information.

The movement amount calculator 140 calculates the user's movement amount FP in a preset period T based on a plurality of pieces of skeleton information. The motion amount FP is calculated by averaging the amount of movement of the coordinates (that is, the skeleton feature points) indicated by the skeleton information in the period T. FIG. When the i-th (1<i<N) coordinate at time t (0<t<T) is (Xi(t), Yi(t)), the amount of motion FP is expressed by equation (1). .

The determination unit 150 determines whether or not the amount of motion FP is equal to or less than a preset threshold. If the motion amount FP is less than or equal to a preset threshold, it can be said that the user's motion is small.
The estimation unit 160 estimates that the user's emotion is unpleasant when the amount of motion FP is equal to or less than the threshold.

Here, for example, discomfort occurs when the user is waiting. When the discomfort increases, the user performs motions such as swaying, changing the supporting leg, and rearranging the arm. Therefore, the estimation unit 160 may perform the following processing in order to estimate the type of uncomfortable motion. Estimating section 160 calculates a trigonometric function “A sin(2πft+φ)” using a plurality of skeleton information. The estimator 160 approximates the trigonometric function “A sin(2πft+φ)”. As a result, "A sin(2πft+φ)+B" is obtained. Here, a graph showing "A sin(2πft+φ)+B" is illustrated.

FIG. 5 is a diagram illustrating an example of a graph according to the embodiment; The vertical axis indicates the positions of skeleton feature points. For example, the vertical axis indicates the positions of the skeletal feature points of the foot. The position of the skeleton feature point may be considered as the X coordinate of the coordinates indicating the skeleton feature point, or may be considered as the Y coordinate of the coordinate. The horizontal axis indicates time t.
As shown in the graph, the estimating unit 160 can calculate "A sin(2πft+φ)+B" based on a plurality of pieces of skeleton information.

Estimating section 160 calculates the frequency based on “A sin(2πft+φ)+B”. When the amplitude A is equal to or greater than a preset threshold, the estimating unit 160 estimates that the uncomfortable motion is switching the pivot leg or rearranging the arm. If the frequency is equal to or greater than a preset threshold, the estimating unit 160 estimates that the motion of discomfort is extortion.
In this way, estimation section 160 estimates the type of uncomfortable motion based on a plurality of pieces of skeleton information. Accordingly, the information processing apparatus 100 can estimate the type of uncomfortable motion.

The estimation unit 160 may calculate the degree of discomfort based on the length of time the user stays and the length of time estimated to be uncomfortable. For example, the estimation unit 160 calculates the length of time the user stays at the same position based on multiple images. The estimating unit 160 calculates the estimated discomfort time based on a plurality of images out of the plurality of images. The estimating unit 160 calculates the degree of discomfort based on the length of time the user stays and the length of time estimated to be uncomfortable. Accordingly, the information processing apparatus 100 can detect the user's discomfort level. The information processing apparatus 100 may control the control target device 300 to reduce the degree of discomfort.

Next, processing executed by the information processing apparatus 100 will be described using a flowchart.
FIG. 6 is a flowchart illustrating an example of processing executed by the information processing apparatus according to the embodiment;
(Step S<b>11 ) The acquisition unit 120 acquires a plurality of images captured at different times from the imaging device 200 .
(Step S12) Extraction unit 130 extracts a plurality of skeleton information based on a plurality of images.
(Step S13) The motion amount calculator 140 calculates the user's motion amount FP based on a plurality of pieces of skeleton information.

(Step S14) The determination unit 150 determines whether the motion amount FP is equal to or less than a preset threshold. If the motion amount FP is equal to or less than the threshold, the process proceeds to step S15. If the motion amount FP is greater than the threshold, the process ends.
(Step S15) The estimation unit 160 estimates that the user's emotion is unpleasant.

As described above, according to the embodiment, the information processing device 100 can estimate emotion based on small movements.

100 information processing device, 101 processor, 102 volatile storage device, 103 nonvolatile storage device, 104 communication interface, 110 storage unit, 120 acquisition unit, 130 extraction unit, 140 operation amount calculation unit, 150 determination unit, 160 estimation unit, 200 imaging device, 300 controlled device, 301 elevator, 400 image, 500 skeleton information, 501 coordinates.

Claims (6)

an acquisition unit that acquires a plurality of images that include a user and are captured at different times;
an extraction unit that extracts a plurality of pieces of skeletal information, which are information indicating skeletal feature points of the user, based on the plurality of images;
an action amount calculation unit that calculates an action amount of the user in a preset period based on the plurality of pieces of skeleton information;
a determination unit that determines whether the amount of movement is equal to or less than a preset threshold;
an estimation unit for estimating that the user's emotion is unpleasant when the amount of motion is equal to or less than the threshold;
Information processing device having The acquisition unit acquires a plurality of images obtained by imaging the user at the same time by a plurality of image capturing devices that capture images of the user from different directions, and position information of each of the plurality of image capturing devices. death,
The extraction unit extracts the plurality of skeleton information based on a plurality of images obtained by imaging the user at the same time by the plurality of imaging devices and position information of each of the plurality of imaging devices. extract the skeletal information of one of the
The information processing device according to claim 1 . The estimation unit estimates the type of the unpleasant motion based on the plurality of pieces of skeleton information.
The information processing apparatus according to claim 1 or 2. The estimation unit calculates the discomfort level based on the time the user stays and the time estimated to be uncomfortable.
The information processing apparatus according to any one of claims 1 to 3. The information processing device
Acquiring a plurality of images including a user and having different imaging times,
extracting, based on the plurality of images, a plurality of pieces of skeletal information representing skeletal feature points of the user;
calculating an amount of movement of the user in a preset period based on the plurality of pieces of skeleton information;
determining whether the amount of movement is equal to or less than a preset threshold;
estimating that the user's feeling is unpleasant when the amount of movement is equal to or less than the threshold;
estimation method. information processing equipment,
Acquiring a plurality of images including a user and having different imaging times,
extracting, based on the plurality of images, a plurality of pieces of skeletal information representing skeletal feature points of the user;
calculating an amount of movement of the user in a preset period based on the plurality of pieces of skeleton information;
determining whether the amount of movement is equal to or less than a preset threshold;
estimating that the user's feeling is unpleasant when the amount of movement is equal to or less than the threshold;
An inferred program that causes the process to take place.

Images