TW202232375A - Information processing device, inference method, and inference program - Google Patents

Abstract

An information items processing device (100) has: an acquisition unit (120) that acquires a plurality of images including a user and having different imaging times from each other; an extraction unit (130) for extracting, on the basis of the plurality of images, a plurality of skeleton information items that are information items indicating skeleton feature points of the user; a movement amount calculation unit (140) that, on the basis of the plurality of skeleton information items, calculates a movement amount of the user in a preset period; a determination unit (150) that determines whether or not the movement amount is equal to or less than a preset threshold value; and an inference unit (160) for, when the movement amount is equal to or less than the threshold value, inferring that the emotion of the user is discomfort.

Description

Information processing device, estimation method, and recording medium

The present disclosure relates to an information processing device, an estimation method, and a recording medium on which the estimation program is recorded.

In recent years, it has been desired to control machines in accordance with human conditions. For example, to indicate the human condition, there are human actions or moods. For example, Patent Document 1 describes that it is possible to estimate the mood, the intention of speaking, and the like from the actions obtained by tracking the position of the face and the hand through time series. [Prior Art Literature] [Patent Literature]

[Patent Document 1] Japanese Patent Laid-Open No. 2007-058615

[Problems to be solved by the invention]

As mentioned above, if you swing your body parts such as your hands and feet a lot, you can estimate your mood. On the other hand, if the motion is small, the above-mentioned thinking cannot be used for estimation.

For the purpose of this disclosure, it is possible to estimate the mood based on small actions. [means to solve the problem]

An information processing apparatus related to an aspect of the present disclosure is provided. The information processing device includes: an acquisition unit that acquires a plurality of images; the plurality of images includes a user, and the shooting times are different; an extraction unit, based on the plurality of images, retrieves a plurality of skeleton information; the plurality of skeleton information indicates the The skeletal feature points of the user; the motion amount calculation unit, based on the plurality of skeleton information, calculates the user's motion amount within a preset period; the determination unit determines whether the motion amount is below a set threshold value; The measuring unit estimates that the user's mood is unhappy when the movement amount is below the threshold value. [Effect of invention]

According to the present disclosure, it is possible to estimate a mood based on small actions.

Hereinafter, embodiments will be described with reference to the drawings. The following embodiments are merely examples, and various modifications can be made within the scope of the present disclosure.

Implementation form. Fig. 1 is a schematic diagram showing the control system of the embodiment. The control system includes an information processing device 100 , a photographing device 200 , and a control object device 300 . The information processing device 100 , the photographing device 200 , and the control target device 300 communicate through a network. The information processing apparatus 100 is an apparatus for executing the estimation method. The information processing device 100 may also be referred to as a control device or a mood estimation device.

First, the hardware included in the information processing apparatus 100 will be described. The information processing device 100 includes a processor 101 , a volatile memory device 102 , a non-volatile memory device 103 , and a communication interface 104 .

The processor 101 controls the entire information processing device 100 . For example, the processor 101 is a central processing unit (Central Processing Unit, CPU), a field programmable gate array (Field Programmable Gate Array, FPGA) and the like. The processor 101 may also be a multiprocessor. In addition, the information processing apparatus 100 may also include a processing circuit. The processing circuit may also be a single circuit or a composite circuit.

The volatile memory device 102 is the main memory device of the information processing device 100 . For example, the volatile memory device 102 is a random access memory (Random Access Memory, RAM). The non-volatile memory device 103 is an auxiliary memory device of the information processing device 100 . For example, the non-volatile memory device 103 is a Hard Disk Drive (HDD) or a Solid State Drive (SSD). The communication interface 104 communicates with the photographing device 200 and the control target device 300 .

For example, the photographing device 200 is a surveillance camera, a driver monitor system (DMS) or an infrared camera. The photographing device 200 photographs the user. The information processing device 100 acquires a plurality of images (in other words, images) from the imaging device 200 . The information processing apparatus 100 estimates the user's mood based on the plurality of images. The information processing device 100 can control the control target device 300 based on the estimated mood information. Here, an example of the control of the control target apparatus 300 is shown.

FIG. 2 is a schematic diagram showing a specific example of the control of the apparatus to be controlled according to the embodiment. FIG. 2 shows an elevator 301 of a motor as an example of the apparatus 300 to be controlled. In addition, in FIG. 2, the top view which shows the user who performs small movements is shown. The photographing device 200 is installed on the ceiling. The photographing device 200 photographs the user. The information processing device 100 acquires a plurality of images from the photographing device 200 . The information processing apparatus 100 estimates the user's mood based on the plurality of images. The information processing device 100 controls the elevator 301 based on the estimated mood information.

Next, the functions of the information processing apparatus 100 will be described. FIG. 3 is a block diagram showing the function of the information processing apparatus according to the embodiment. The information processing apparatus 100 includes a storage unit 110 , an acquisition unit 120 , a retrieval unit 130 , an operation amount calculation unit 140 , a determination unit 150 , and an estimation unit 160 .

The memory unit 110 can also be implemented as a memory area reserved in the volatile memory device 102 or the non-volatile memory device 103 . Some or all of the acquisition unit 120 , the extraction unit 130 , the motion amount calculation unit 140 , the determination unit 150 , and the estimation unit 160 may be implemented by a processing circuit. In addition, some or all of the acquisition unit 120 , the extraction unit 130 , the motion amount calculation unit 140 , the determination unit 150 , and the estimation unit 160 may be implemented as modules of programs executed by the processor 101 . For example, the program executed by the processor 101 is also referred to as an estimation program. For example, the estimation program is recorded on a recording medium.

The acquisition unit 120 acquires, from the imaging device 200 , a plurality of images whose shooting times are different from each other. The acquisition unit 120 may acquire a plurality of images from an external device (eg, a cloud server). Each of the plurality of images is an image captured by the user at different times. Therefore, the user is included in each of the plurality of images.

The extraction unit 130 extracts a plurality of skeleton information based on a plurality of images. Each of the plurality of skeleton information is information indicating a skeleton feature point of the user. For example, the information indicating the skeleton feature points of the user is the coordinates. Hereinafter, it is assumed that the information indicating the skeletal feature points of the user is the coordinates.

First, the case of extracting one piece of skeleton information based on one image will be described. Here, an example of bone information is shown.

FIG. 4 is a schematic diagram showing an example of bone information in an embodiment. FIG. 4 illustrates an image 400, that is, one image among a plurality of images. The extracting part 130 extracts the bone information 500 using conventional techniques. Examples of conventional technologies are OpenPose or OpenNet. The skeleton information 500 is the coordinates indicating the skeleton feature points of the user. For example, the coordinate 501 in FIG. 4 represents the skeletal feature point of the user's ear. In addition, the coordinates are represented by an X coordinate and a Y coordinate.

FIG. 4 illustrates a case where one user is included in the image 400 . If the image includes a plurality of users, the extracting unit 130 extracts the skeleton information of each of the plurality of users.

In this way, the extraction unit 130 extracts one piece of bone information based on one video. Similarly, the extraction unit 130 performs extraction processing of skeleton information for each of the plurality of images. In this way, a plurality of bone information can be obtained.

Here, a plurality of imaging devices may be included in the control system. Each of the plurality of photographing devices is installed at a different position. Each of the plurality of photographing devices photograph the user from different directions. If the control system includes a plurality of photographing devices, the obtaining unit 120 obtains a plurality of images obtained by the plurality of photographing devices photographing the user at the same time. For example, the obtaining unit 120 obtains the plurality of images from the plurality of photographing devices. In addition, the acquisition unit 120 acquires the position information of each of the plurality of imaging devices. For example, the obtaining unit 120 obtains the respective position information of the plurality of photographing devices from the memory unit 110 . The extraction unit 130 extracts one piece of bone information based on the plurality of images and the respective position information of the plurality of imaging devices. Here, for example, when a certain photographing device photographs the user from a certain direction, it is possible that a part of the user cannot be photographed. Then, the part is photographed by photographing the user from a different direction by another photographing device. Thereby, the extracting unit 130 extracts the skeleton information based on the plurality of images obtained by each of the plurality of imaging devices photographing the user from different directions. In this way, the information processing apparatus 100 can obtain high-precision bone information.

The motion amount calculation unit 140 calculates the user's motion amount FP during a predetermined period T based on the plurality of pieces of skeleton information. The movement amount FP is calculated by averaging the movement amounts of the coordinates indicated by the skeleton information (in other words, the skeleton feature points) during the period T. Assuming that the ith (1<i<N) coordinate at time t (0<t<T) is (Xi(t), Yi(t)), the action amount FP can be expressed by the formula (1).

[Equation 1]

The determination unit 150 determines whether or not the motion amount FP is equal to or less than a predetermined threshold value. When the motion amount FP is equal to or less than a predetermined threshold value, it can be said that the user's motion is small. When the motion amount FP is below the threshold value, the estimation unit 160 estimates that the user's mood is unpleasant.

Here, for example, the user is displeased while on standby. When the user's unpleasantness increases, he/she will perform actions such as shaking his feet, tilting his feet, and crossing his arms. Then, the estimation unit 160 may perform the following processing in order to estimate the type of unpleasant action. The estimation unit 160 calculates the trigonometric function "Asin(2πft+ϕ)" using the plurality of skeleton information. The estimation unit 160 approximates the trigonometric function "Asin(2πft+ϕ)". This results in "Asin(2πft+ϕ)+B". Here, a graph showing "Asin(2πft+ϕ)+B" is used as an example.

Fig. 5 is a schematic diagram showing an example of a graph of the embodiment. The vertical axis indicates the position of the bone feature points. For example, the vertical axis represents the bone feature points of the foot. The position of the skeleton feature point can be considered as the X coordinate indicating the coordinates of the skeleton feature point, and can also be considered as the Y coordinate of the coordinate. The horizontal axis indicates time t. As shown in the graph, the estimation unit 160 can calculate “Asin(2πft+ϕ)+B” based on a plurality of skeleton information.

The estimation unit 160 calculates the frequency based on "Asin(2πft+φ)+B". If the amplitude A is greater than or equal to a preset threshold value, the estimation unit 160 estimates that the unpleasant action is lifting the feet or crossing the arms. If the frequency is above a preset threshold, the estimating unit 160 estimates that the unpleasant action is foot shaking. Thereby, the estimation part 160 estimates the kind of unpleasant motion based on a plurality of skeleton information. Thereby, the information processing apparatus 100 can estimate the type of unpleasant action.

The estimation unit 160 may calculate the degree of discomfort based on the time that the user stayed and the time when the user was estimated to be unhappy. For example, the estimation unit 160 calculates the time that the user stays at the same location based on the plurality of images. The estimation part 160 calculates the time estimated to be unpleasant based on the plurality of videos among the plurality of videos. The estimation unit 160 calculates the degree of discomfort based on the time spent by the user and the time estimated to be unpleasant. Thereby, the information processing apparatus 100 can detect the user's dissatisfaction level. The information processing device 100 may also perform control for reducing the degree of discomfort to the control target device 300 .

Next, the processing performed by the information processing apparatus 100 will be described using a flowchart. FIG. 6 is a flowchart showing an example of processing executed by the information processing apparatus according to the embodiment. (Step S11 ) The acquisition unit 120 acquires, from the imaging device 200 , a plurality of video images whose shooting times are different from each other. (Step S12 ) The extraction unit 130 extracts a plurality of skeleton information based on a plurality of images. (Step S13 ) The motion amount calculation unit 140 calculates the motion amount FP of the user based on the plurality of pieces of skeleton information.

(Step S14 ) The determination unit 150 determines whether or not the motion amount FP is equal to or less than a predetermined threshold value. When the motion amount FP is equal to or less than the threshold value, the process proceeds to step S15. When the motion amount FP is larger than the threshold value, the process ends. (Step S15 ) The estimation unit 160 estimates that the user's mood is unpleasant.

As described above, according to the embodiment, the information processing apparatus 100 can estimate the mood based on small actions.

100: Information processing device 101: Processor 102: Volatile memory device 103: Non-volatile memory device 104: Communication interface 110: Memory Department 120: Acquire Department 130: Take out part 140: Motion amount calculation section 150: Judgment Department 160: Estimation Department 200: Photographic Installation 300: Control the target machine 301: Lift 400: Video 500: Bone Information 501: Coordinates S11~S15: Steps

Fig. 1 is a schematic diagram showing the control system of the embodiment. FIG. 2 is a schematic diagram showing a specific example of the control of the apparatus to be controlled according to the embodiment. FIG. 3 is a block diagram showing the function of the information processing apparatus according to the embodiment. FIG. 4 is a schematic diagram illustrating bone information of an embodiment. Fig. 5 is a schematic diagram showing a diagram of an embodiment. FIG. 6 is a flowchart showing an example of processing executed by the information processing apparatus of the embodiment.

100: Information processing device

110: Memory Department

120: Acquire Department

130: Take out part

140: Motion amount calculation section

150: Judgment Department

160: Estimation Department

Claims (6)

An information processing device, comprising: the obtaining part obtains a plurality of images; the plurality of images include the user, and the shooting times are different from each other; The extraction part, based on the plurality of images, extracts a plurality of skeleton information; the plurality of skeleton information indicates the skeleton feature points of the user; an amount of motion calculation unit for calculating the amount of motion of the user within a preset period based on the plurality of skeleton information; a determination unit that determines whether the motion amount is below a set threshold; and The estimation unit estimates that the user's mood is unhappy when the movement amount is below the threshold value. If the information processing device of claim 1, Wherein, the obtaining unit obtains a plurality of images obtained by photographing the user from different directions at the same time by a plurality of photographing devices of the user, and the respective position information of the plurality of photographing devices; Wherein, the extracting part extracts one piece of bone information from the plurality of bone information based on the plurality of images obtained by the plurality of photographing devices at the same time shooting the user and the respective position information of the plurality of photographing devices. If the information processing device of claim 1 or 2, The estimating unit estimates the type of the unpleasant action based on the skeleton information of the skeleton. If the information processing device of claim 1 or 2, The estimation unit calculates the degree of discomfort based on the time the user stayed and the time estimated to be the discomfort. An estimation method, executed by an information processing device: Acquiring a plurality of images; the plurality of images include users, and the shooting times are different from each other; Based on the plurality of images, extracting a plurality of skeleton information; the plurality of skeleton information indicates the skeleton feature points of the user; based on the plurality of skeleton information, calculate the motion amount of the user within a preset period; determine whether the movement amount is below a set threshold; and When the motion amount is below the threshold value, it is estimated that the user's mood is unhappy. A recording medium recording an estimation program, the estimation program causes an information processing device to perform the following processing: Acquiring a plurality of images; the plurality of images include users, and the shooting times are different from each other; Based on the plurality of images, extracting a plurality of skeleton information; the plurality of skeleton information indicates the skeleton feature points of the user; based on the plurality of skeleton information, calculate the motion amount of the user within a preset period; determine whether the movement amount is below a set threshold; and When the motion amount is below the threshold value, it is estimated that the user's mood is unhappy.

Images