JP2022189456A - Action recognition program, action recognition method, and information processing apparatus - Google Patents

Abstract

To recognize an action of a target person at a low cost with high accuracy.SOLUTION: An information processing apparatus detects skeleton information of a person from an image obtained by an imaging apparatus. The information processing apparatus acquires region information on a position where the person was imaged, and positions of a target region and an object. The information processing apparatus estimates an action performed by the person with respect to the object, based on the skeleton information of the person and the region information. The information processing apparatus recognizes an action of the person based on a distance between the person and the object, and the estimated action.SELECTED DRAWING: Figure 7

Description

The present invention relates to an action recognition program, an action recognition method, and an information processing apparatus.

With the development of AI (Artificial Intelligence) technology, technology has been developed and used that recognizes people and objects from images and automatically detects human movements, postures, states, and actions from the skeletal information of the recognized people. ing. For example, technology that automatically detects elderly people and physically handicapped people and ascertains whether they are in a dangerous state, recognizes the posture and processes of workers, and determines whether they are entering dangerous places or not. There is a technology to grasp whether the worker is working in a proper posture and whether the procedure is followed.

In this way, the use of AI technology makes it possible to automatically analyze human behavior and conditions, so it can be applied to various fields such as purchase analysis, work analysis, on-site monitoring and monitoring, and suspicious person detection. desired. For example, there is known a technology that combines a camera and a sensor and recognizes the movement of a subject, particularly the movement of reaching out and the work.

JP 2019-139321 A JP 2015-176227 A

However, with the above technology, it is difficult to recognize the behavior of the target person at low cost and with high accuracy. For example, recognition technology that uses sensors generally uses special sensors such as lasers and wireless sensors, which results in complex configurations and high costs. Although a recognition technique using only a camera is also known, it cannot accurately detect up, down, left, right, and depth directions.

An object of one aspect is to provide an action recognition program, an action recognition method, and an information processing apparatus capable of recognizing a target person's action at low cost and with high accuracy.

In the first plan, the action recognition program causes the computer to detect the skeleton information of the person from the image acquired by the imaging device, and the position where the person was imaged, the target area, and the area information regarding the position of the target object. is obtained, based on the skeletal information of the person and the region information, an action performed by the person on the object is estimated, and based on the distance between the person and the object and the estimated action , recognizing the action of the person, and executing a process.

According to one embodiment, the behavior of a subject can be recognized at low cost and with high accuracy.

FIG. 1 is a diagram for explaining an information processing apparatus according to a first embodiment; FIG. 2 is a functional block diagram of the functional configuration of the information processing apparatus according to the first embodiment; FIG. 3 is a diagram showing an example of information stored in the target location information DB. FIG. 4 is a diagram for explaining person detection and skeleton detection. FIG. 5 is a diagram illustrating an example of skeleton information. FIG. 6 is a diagram for explaining motion estimation of a person. FIG. 7 is a diagram for explaining estimation of a hand stretching range. FIG. 8 is a diagram for explaining recognition of a person's hand stretching action and position calculation. FIG. 9 is a flowchart illustrating an overall flow of action recognition processing according to the first embodiment; FIG. 10 is a flowchart illustrating the flow of human detection processing according to the first embodiment. FIG. 11 is a flowchart illustrating the flow of skeleton detection processing according to the first embodiment. FIG. 12 is a flowchart illustrating the flow of motion estimation processing according to the first embodiment. 13 is a flowchart illustrating the flow of range estimation processing according to the first embodiment; FIG. FIG. 14 is a flowchart illustrating a flow of recognition processing for a hand-stretching action according to the first embodiment. FIG. 15 is a flowchart of a process for calculating a stretched-hand position according to the first embodiment. FIG. 16 is a flowchart illustrating an overall flow of action recognition processing according to the second embodiment; FIG. 17 is a diagram for explaining an example of the output result screen. FIG. 18 is a diagram for explaining motion estimation processing using motion transitions. FIG. 19 is a diagram illustrating a hardware configuration example.

An action recognition program, an action recognition method, and an information processing apparatus disclosed in the present application will be described in detail below with reference to the drawings. In addition, this invention is not limited by this Example. Moreover, each embodiment can be appropriately combined within a range without contradiction.

[overall structure]
FIG. 1 is a diagram illustrating an information processing apparatus 10 according to the first embodiment. As shown in FIG. 1, in this system, an imaging device such as a camera installed in a retail store that sells clothing, food, etc. to consumers, and an information processing device 10 are connected by wire or wirelessly. This is a system for grasping the purchase situation in a retail store by means of a processing device 10. FIG. The information processing device 10 acquires and analyzes video data inside a retail store, and can be used to identify a consumer group that purchases a certain product, a consumer group that picks up a certain product, and the like. It is an example of a computer device that generates and outputs information.

Specifically, the information processing device 10 identifies skeleton information of a person from image data acquired by an imaging device. Then, the information processing apparatus 10 acquires area information about the position where the person is imaged, the target area, and the position of the object, and based on the skeleton information and the area information of the person, the motion performed by the person on the object. to estimate After that, the information processing device 10 recognizes the action of the person based on the distance between the person and the target object and the estimated action.

For example, the information processing apparatus 10 recognizes skeleton information of a person from image data acquired from a camera, and the skeleton information is based on conditions such as "looking at (saw) a shelf, facing a shelf, standing or crouching, holding hands." Including actions such as stretching, the distance between the shelf and the person is within the reach of the hand", the person present in the image is "reaching for the product on the shelf". Recognize. Then, when the information processing apparatus 10 recognizes that "the person is reaching for the product on the shelf", the information processing apparatus 10 identifies the position where the hand is reaching from the skeleton information.

By doing so, the information processing apparatus 10 can recognize the behavior of the target person with high accuracy by a simple and low-cost method using the image of the camera. As a result, the information processing apparatus 10 can perform a work analysis such as a motion analysis of a target person's hand stretching and a shelf removal work with high accuracy.

[Function configuration]
FIG. 2 is a functional block diagram of the functional configuration of the information processing apparatus 10 according to the first embodiment. As shown in FIG. 2 , the information processing device 10 has a communication section 11 , an output section 12 , a storage section 13 and a control section 20 .

The communication unit 11 is a processing unit that controls communication between other devices, and is realized by, for example, a communication interface. For example, the communication unit 11 receives data such as videos, images, or moving images from imaging devices such as cameras.

The output unit 12 is a processing unit that displays various types of information, and is realized by, for example, a display or a touch panel. For example, the output unit 12 outputs a result of behavior recognition recognized by the control unit 20, which will be described later.

The storage unit 13 is a processing unit that stores various data, programs executed by the control unit 20, and the like, and is realized by, for example, a memory or a hard disk. This storage unit 13 stores a machine learning model DB 14 and a target location information DB 15 .

The machine learning model DB 14 is a database that stores trained machine learning models. For example, the machine learning model DB 14 identifies a person appearing in the image data according to input of image data such as each frame of video data, and outputs area information of the person appearing in the first machine learning model. memorize The machine learning model DB 14 also stores a second machine learning model that outputs skeleton information of a person in the image in response to input of area information of the person and image data.

The target location information DB 15 is a database that stores target location information regarding an area imaged by the imaging device. Specifically, the target location information DB 15 stores, for each installed imaging device, target location information indicating the areas and positions of product shelves and products in the imaging area of each imaging device.

FIG. 3 is a diagram showing an example of information stored in the target location information DB 15. As shown in FIG. As shown in FIG. 3, the target place information DB 15 stores a shelf ROI 30 that is a ROI (Region of Interest) of a product shelf and an aisle ROI 40 that is a ROI of an aisle. In addition to these, the target location information DB 15 stores the coordinates of the product shelf, the coordinates of each product placed on the product shelf (shelf ROI 30), the coordinates of the passage, and the like.

Note that the target location information may be information in which the positions of the left, right, upper and lower ends of the shelf area are recorded in advance. Alternatively, the target location information may be obtained by learning the shelf area in advance and using the recognized shelf area (semantic segmentation). For the target location information, it is also possible to use an area group in which products on shelves are learned in advance as object models and objects are detected using image analysis technology.

The control unit 20 is a processing unit that controls the entire information processing apparatus 10, and is realized by, for example, a processor. The control unit 20 has an image acquisition unit 21 , a person detection unit 22 , a skeleton detection unit 23 , a motion estimation unit 24 , a range estimation unit 25 , a hand stretching action recognition unit 26 and a hand stretching position calculation unit 27 . Note that the image acquisition unit 21, the person detection unit 22, the skeleton detection unit 23, the motion estimation unit 24, the range estimation unit 25, the hand stretching behavior recognition unit 26, and the hand stretching position calculation unit 27 are electronic circuits and processors possessed by the processor. It is realized by a process to execute.

The video acquisition unit 21 is a processing unit that acquires video data. For example, the video acquisition unit 21 acquires video data of a target area from a camera connected via USB (Universal Serial Bus), LAN (Local Area Network), wireless, or the like, and outputs the video data to the person detection unit 22 . Note that the video acquisition unit 21 can acquire not only real-time video data from the camera but also moving image data captured in advance.

The person detection unit 22 is a processing unit that detects a person appearing in the video data from the video data. Specifically, the person detection unit 22 executes person detection for each frame of the video data acquired by the image acquisition unit 21 and outputs information about the detected person to the skeleton detection unit 23 . For example, the human detection unit 22 performs human detection on the acquired video using video analysis technology such as deep learning. It is also possible to use a parameter such as a threshold value to determine the likeness of a person, and then proceed to the next frame video acquisition process.

The skeleton detection unit 23 is a processing unit that detects skeleton information of a person from an image acquired by an imaging device. For example, the skeleton detection unit 23 performs skeleton detection on the person region detected by the person detection unit 22 using the same video analysis technology. It is also possible to use a parameter such as a threshold value to determine the skeleton-likeness, and then proceed to the next frame video acquisition process.

Here, person detection and skeleton detection will be specifically described. FIG. 4 is a diagram for explaining person detection and skeleton detection. As shown in FIG. 4, the person detection unit 22 inputs one frame of video data to the first machine learning model and acquires position information (area information) including the area of the detected person. Here, when the score (probability) of the output value of the first machine learning model is equal to or greater than the threshold, the person detection unit 22 can also adopt the output position information of the person.

Next, the skeleton detection unit 23 inputs the image data in which the person is detected and the position information of the person obtained from the first machine learning model to the second machine learning model, and acquires the skeleton information of the detected person. . Here, if the score (probability) of the output value of the second machine learning model is equal to or greater than the threshold, the skeleton detection unit 23 can also adopt the output skeleton information of the person.

FIG. 5 is a diagram illustrating an example of skeleton information. FIG. 5 shows skeleton definition information acquired as skeleton information. As shown in FIG. 5, the skeletal information stores 18 pieces (0 to 17) of definition information numbered for each joint specified in a known skeletal model. For example, the right shoulder joint (SHOULDER_RIGHT) is assigned number 7, the left elbow joint (ELBOW_LEFT) is assigned number 5, the left knee joint (KNEE_LEFT) is assigned number 11, and the right hip joint (HIP_RIGHT) is assigned number 11. is given the number 15.

Therefore, the skeleton detection unit 23 acquires the coordinate information of the 18 skeletons shown in FIG. 5 from the second machine learning model. For example, the skeleton detection unit 23 acquires “X coordinate=X7, Y coordinate=Y7, Z coordinate=Z7” as the position of the seventh right shoulder joint. Note that, for example, the Z-axis can be defined as the distance direction from the imaging device to the object, the Y-axis as the height direction perpendicular to the Z-axis, and the X-axis as the horizontal direction.

Returning to FIG. 2, the motion estimating unit 24 acquires region information about the region of the object in the position where the person was imaged, and estimates the motion of the person to the object based on the region information and the skeleton information of the person. This is the processing unit for estimation. Specifically, the motion estimating unit 24 detects a motion of the person detected by the human detecting unit 22 reaching out toward the shelf, looking at the shelf, or facing the shelf (facing the shelf). Depending on whether or not any of the conditions are met, the motion performed by the person on the object is estimated, and the estimation result is output to the range estimation unit 25 and the hand-stretching action recognition unit 26 .

FIG. 6 is a diagram for explaining motion estimation of a person. In FIG. 6, a person is depicted by connecting each joint included in the skeleton information with a line. As shown in FIG. 6, the motion estimating unit 24 uses the shelf position information stored in the target location information DB 15 and the skeleton information detected by the skeleton detecting unit 23 to previously learn the motion of the detected person. It is determined whether or not it matches the action that was performed.

For example, if the motion estimating unit 24 is located within a predetermined position from the shelf and the detected skeleton information is similar to the skeleton information indicating the motion A prepared in advance, the motion estimation unit 24 determines that motion A as the estimation result. . In addition, the motion estimating unit 24 determines that the transition of each skeleton information detected so far is similar to the transition of the skeleton information of the motion B that has been prepared in advance while the motion estimating unit 24 is positioned within a predetermined position from the shelf. , and its motion B is assumed to be the estimation result. It should be noted that the degree of similarity can employ a known method, such as whether or not the total value of the differences of each coordinate is less than a threshold.

Here, the motion estimating unit 24 determines whether the user is looking at the shelf or is facing the shelf. use to do

For example, the motion estimating unit 24 may use, as a determination condition, a motion in which the hand is stretched out, and furthermore, the angle of the arm being the largest in a series of motions. Specifically, the motion estimating unit 24 can also estimate that the hand is extended when the angle of the arm using skeletal information Nos. 5 and 8 is equal to or greater than a threshold value. In addition, the motion estimating unit 24 can also estimate that the hand is extended when the hand moves along a trajectory in one direction, in addition to the motion of extending the hand in the horizontal and vertical directions after the hand is stretched out.

In addition, the motion estimating unit 24 may determine that the action of looking at the place is the motion of looking at the place by looking at the detected hand in addition to the above-mentioned predetermined area or the object area such as the shelf. No. For example, the motion estimating unit 24 monitors the transition of the 3rd (HEAD) of the skeleton information using several frames. Presumed to be the action of looking at (shelf). Further, the motion estimating unit 24 estimates a motion of looking at a place (shelf) when detecting a motion of extending a hand after detecting a motion of moving the face a plurality of times.

Further, the motion estimating unit 24 may use, as a determination condition, the motion of standing, squatting, and sitting as the motion of facing the shelf, and that the body is facing the front or diagonally with respect to the area or object. No. For example, when the motion estimating unit 24 detects any motion of standing, crouching, or sitting while the body is positioned within a predetermined position from the shelf and the body faces the object (shelf), Presumed to be facing motion. Each motion described above can be specified by monitoring the skeleton information and the transition of the skeleton information. Also, the motion estimation unit 24 may estimate a plurality of motions.

Returning to FIG. 2, the range estimating unit 25 is a processing unit that estimates, as the distance between the person and the object (shelf), the reachable range of the person based on the skeleton information of the person and the area information of the shelf. For example, the range estimating unit 25 uses the result of estimation by the motion estimating unit 24 and position information (object region) of a shelf set in advance. , a certain distance (e.g., shoulder width) from the perpendicular line of the shortest shelf is estimated as the reachable range. The range estimating unit 25 also determines that the length of the hand estimated from the skeletal information detected by the skeletal detection unit 23, such as the length of the legs, the height of the back, and the width of the shoulders, and the angle of the arm, for example, is 180 degrees or more. The reachable range of the hand may be estimated using the position of the hand estimated from the biological characteristics such as the absence of the hand.

FIG. 7 is a diagram for explaining estimation of a hand stretching range. As shown in FIG. 7, the range estimating unit 25 first detects the center point of both feet specified by the skeleton information detected by the skeleton detecting unit 23 as the standing position (S1). Subsequently, the range estimating unit 25 calculates, for example, a perpendicular line to the underline of the shelf ROI 30 as the shortest distance from the center point of both feet to the shelf (position information) stored in the target location information DB 15 (S2). . Then, the range estimating unit 25 calculates a vertical line in the height direction of the shelf (positional information) with respect to the shortest distance (S3). After that, the range estimating unit 25 uses the height (S4) from the feet to the shoulders and the shoulder width information (S5) in the skeleton information detected by the skeleton detecting unit 23 to obtain, for example, 90% to 110% of the height and the width of the shoulders. ±100% is estimated as the reachable range (reachable range) (S6). Then, the range estimating section 25 outputs the estimation result to the hand reaching action recognizing section 26 . Note that the numerical values shown here are only examples, and can be changed arbitrarily.

It should be noted that the range estimating unit 25 is not limited to the standing state, and even if it is determined that the action is sitting, the range estimating unit 25 performs the above-described determinations from S1 to S6 from the sitting state. It is also possible to estimate the reach of In addition, the range estimating unit 25 can also estimate the reachable range from the shoulder skeleton information and the perpendicular to the shelf using the length of the arm specified by the shoulder, elbow and hand skeleton information.

Returning to FIG. 2, the hand-stretching action recognition unit 26 determines whether the person reached out to the target object based on the motion estimated by the motion estimation unit 24 and the reachable range of the person estimated by the range estimation unit 25. It is a processing unit that recognizes actions. Then, the hand-stretching behavior recognition unit 26 outputs the recognition result to the hand-stretching position calculation unit 27 .

Specifically, the hand-stretching behavior recognition unit 26 uses the motion estimated by the motion estimation unit 24 and the reachable range of the person estimated by the range estimation unit 25, and determines with a preset threshold value. By doing so, it is recognized as an action of reaching out. Alternatively, these determination items may be digitized as feature amounts, Mahalabino's distances may be calculated, and the results of machine learning may be used for determination.

The stretched hand position calculation unit 27 calculates the position information of the extended hand using the skeleton information, the shelf ROI 30, etc. in the image (video frame) recognized as the stretched hand behavior by the stretched hand behavior recognition unit 26. It is a processing unit that

In addition, the hand-stretching position calculation unit 27 can also calculate hand position information at the time when the hand-stretching action is recognized, and use it as access to the corresponding product. For example, the stretched-hand position calculator 27 identifies the product that the person picked up or was about to pick up by comparing the position information of the product in the shelf ROI 30 and the position information of the outstretched hand. In this way, when a person is specified from each frame of the video data, the stretched-hand position calculation unit 27 specifies and tallies the products stretched out by the person.

Action recognition and position calculation will now be described in detail. FIG. 8 is a diagram for explaining recognition of a person's hand stretching action and position calculation. As shown in FIG. 8, the reaching action recognizing unit 26 determines whether or not the reaching action is a reaching action based on the result of estimating the range of reach of the hand and the result of estimating the action of the person, and the score (for example, matching rate) of the predetermined condition. recognize what

For example, the hand-stretching action recognition unit 26 recognizes the hand-stretching action when a predetermined number or more of conditions are matched, or when a predetermined percentage of all conditions are matched. It should be noted that the conditions can also be generated and set in advance using experimental data, etc., and can be defined separately for the results of estimating the reach of the hand and the results of estimating the motion of a person. can also be determined.

After that, the stretching position calculation unit 27 identifies the image data recognized as the stretching behavior, and uses the skeleton information of the person identified using the image data, the shelf ROI 30, the product position information, etc., to calculate the hand position. Calculate the position information of For example, the hand-stretching position calculation unit 27 calculates the coordinates of the shelves and products overlapping the hand in the image data as the position information of the hand, using the coordinates of the shelves, the coordinates of the products, and the like. In addition, the stretched hand position calculation unit 27 inputs the coordinates of the shelf, the coordinates of the product, the recognition result of the stretched hand action, and the skeleton information of the person to the trained machine learning model, and acquires the position information of the hand of the person. can also

Then, the stretched-hand position calculation unit 27 identifies the product accessed by the person using the position information of the product and the position information of the person's hand. For example, the stretched-hand position calculation unit 27 considers a product that matches the coordinates of the hand, a product that is within a predetermined range from the coordinates of the hand, or a product that is on an extension line from the coordinates of the hand to the shelf direction as the accessed product. Identify.

[Process flow]
Next, the flow of the action recognition processing described above will be described. Here, the flow of the overall processing and the flow of processing by each processing unit will be described.

(Overall processing flow)
FIG. 9 is a flowchart illustrating an overall flow of action recognition processing according to the first embodiment; As shown in FIG. 9, when the information processing apparatus 10 starts processing (S101: Yes), it acquires video data from the imaging device (S102).

Subsequently, the information processing apparatus 10 detects a person from each frame (image data) in the video data (S103), and if the person cannot be detected (S104: No), acquires the next video data. On the other hand, when a person can be detected (S104: Yes), the information processing apparatus 10 detects the skeleton of the detected person (S105).

Then, the information processing apparatus 10 estimates the motion of the detected person (S106), and estimates the stretched-out range of the detected person (S107). Here, the information processing apparatus 10 acquires the next image when the product or the shelf is out of reach (S108: No) according to the estimated reachable range. On the other hand, if the product or the shelf is within reach (S108: Yes), the information processing apparatus 10 recognizes the action of the person (S109) and calculates the position of the person's hand (S110).

(Flow of person detection processing)
FIG. 10 is a flowchart illustrating the flow of human detection processing according to the first embodiment. As shown in FIG. 10, the human detection unit 22 acquires video data (S201), applies the first machine learning model to each frame (image data) in the video data, and performs human detection. (S202).

Then, when the score (probability) of the output value of the first machine learning model is less than the threshold (S203: No), the human detection unit 22 determines that the prediction accuracy of the first machine learning model is low, and Get data. On the other hand, when the score (probability) of the output value of the first machine learning model is equal to or greater than the threshold (S203: Yes), the person detection unit 22 determines that the prediction accuracy of the first machine learning model is high, and The position information of the person is output to the storage unit 13 or the like (S204).

(Flow of skeleton detection processing)
FIG. 11 is a flowchart illustrating the flow of skeleton detection processing according to the first embodiment. As shown in FIG. 11, the skeleton detection unit 23 acquires video data (S301), and acquires position information of a person detected by the person detection unit 22 (S302).

Then, the skeleton detection unit 23 inputs the image data in which the person is detected in the video data and the position information of the person to the second machine learning model, acquires the skeleton information of the detected person (S303), and stores the information. 13, etc. (S304). Also here, as in FIG. 10, the determination based on the score may be executed.

(Flow of motion estimation processing)
FIG. 12 is a flowchart illustrating the flow of motion estimation processing according to the first embodiment. As shown in FIG. 12, the motion estimating unit 24 acquires target place information (area information) including information such as shelves in the imaging area from the target place information DB 15 (S401). Also, the motion estimation unit 24 acquires the skeleton information of the person from the skeleton detection unit 23 (S402).

Subsequently, the motion estimating unit 24 estimates a motion performed by the person on the target based on the information on the skeleton of the person and the information on the target location. Specifically, the motion estimating unit 24 estimates a hand stretching motion (S403), estimates a motion to look at a place (S404), and estimates a motion to face a place (S405). Note that the motion estimation unit 24 outputs information about the estimated motion to the storage unit 13 or the like.

(Flow of range estimation processing)
13 is a flowchart illustrating the flow of range estimation processing according to the first embodiment; FIG. As shown in FIG. 13, the range estimating unit 25 acquires target place information (area information) including information such as shelves in the imaging area from the target place information DB 15 (S501). Also, the motion estimation unit 24 acquires the skeleton information of the person from the skeleton detection unit 23 (S502).

Subsequently, the range estimation unit 25 estimates the shortest distance from the person to the target location based on the skeleton information of the person and the information on the target location (shelf) (S503). Then, the range estimation unit 25 estimates the reachable range based on the estimated shortest distance and the skeleton information of the person (S504). Note that the range estimation unit 25 outputs information about the estimated reachable range to the storage unit 13 or the like.

(Flow of Recognition Processing of Hand Reaching Action)
FIG. 14 is a flowchart illustrating a flow of recognition processing for a hand-stretching action according to the first embodiment. As shown in FIG. 14, the hand stretching action recognition unit 26 acquires the information of the motion estimation estimated by the motion estimation unit 24 (S601), and the estimation result of the reachable range estimated by the range estimation unit 25 Acquire (S602).

Subsequently, the hand-stretching action recognition unit 26 uses the acquired information to calculate the score of the action of the person reaching out to the object (shelf) (S603). Here, if the score is less than the threshold (S603: No), the hand-stretching action recognition unit 26 returns to S601 and repeats the subsequent processes. On the other hand, if the score is equal to or greater than the threshold (S603: Yes), the hand-stretching action recognition unit 26 recognizes the hand-stretching action (S604). Note that the reaching action recognition unit 26 outputs the recognition result of the recognized reaching action to the storage unit 13 or the like.

(Flow of calculation processing of hand stretching position)
FIG. 15 is a flowchart of a process for calculating a stretched-hand position according to the first embodiment. As shown in FIG. 15, the stretched-hand position calculation unit 27 acquires the recognition result of the stretched-hand action by the stretched-hand action recognition unit 26 (S701), acquires the skeleton information of the person (S702), Information and the like are used to calculate the position information of the outstretched hand (S703). Note that the stretched-hand position calculation unit 27 outputs the calculated hand position information to the storage unit 13 or the like.

[effect]
As described above, the information processing apparatus 10 can highly accurately calculate the stretched-out position based on the human body characteristics from the skeleton information acquired from the video data and the predefined shelf position information. As a result, the information processing apparatus 10 can generate and output useful information used for work state analysis, purchase analysis, and the like. In addition, by further analyzing the time required from reaching the shelf to reaching the hand, the speed of reaching, the trajectory and posture, and the number of times, it is also possible to understand the psychological state of the worker and the purchaser. Become.

In addition, since the information processing apparatus 10 performs person detection and skeleton detection using a machine learning model, detection accuracy can be improved by periodically retraining the machine learning model. Further, the information processing apparatus 10 prepares skeleton information of an assumed motion in advance, and can estimate the motion by comparing the detected skeleton information with the information. Moreover, high-level analysis of image data is not required, and detection processing can be speeded up.

In addition, since the recognition technology using only sensors uses only the positional relationship obtained by the sensors, it is determined that the hand is stretched when the product overlaps with the position of the hand or the orientation of the body. For this reason, when a large shelf such as the product shelf described in the first embodiment is used as a target object, erroneous detection is often made. On the other hand, the information processing apparatus 10 dynamically changes the reachable range according to the position of the person (skeletal information) and the position of the shelf. It is possible to improve the detection accuracy by adopting a mechanism that does not detect when it is out of range.

Incidentally, by estimating the attribute of a person, the information processing apparatus 10 can generate and output more useful information that can be used for work state analysis, purchase analysis, and the like.

FIG. 16 is a flowchart illustrating an overall flow of action recognition processing according to the second embodiment; As shown in FIG. 16, when the information processing apparatus 10 starts processing (S801: Yes), it acquires video data from the imaging device (S802).

Subsequently, the information processing apparatus 10 detects a person from each frame (image data) in the video data (S803), and if the person cannot be detected (S804: No), acquires the next video data. On the other hand, when a person can be detected (S804: Yes), the information processing apparatus 10 detects the skeleton of the detected person (S805).

Then, the information processing apparatus 10 estimates the motion of the detected person (S806), and estimates the stretched-out range of the detected person (S807). Here, the information processing apparatus 10 acquires the next image when the product or the shelf is out of the reach of the product or the shelf according to the estimated reachable range (S808: No). On the other hand, if the product or the shelf is within reach (S808: Yes), the information processing apparatus 10 recognizes the action of the person (S809).

In parallel with these, the information processing apparatus 10 estimates the attributes of the person (S810). For example, an attribute estimating unit (not shown) of the information processing apparatus 10 inputs skeletal information of the detected person to a machine learning model generated by deep learning or the like to perform attribute estimation. As attributes of a person, in addition to gender and age estimation, clerk detection, suspicious person detection, etc. may be estimated. It is also possible to use a parameter such as a threshold value to determine the likelihood of an attribute, and then proceed to the next frame video acquisition process.

After that, the information processing apparatus 10 calculates hand position information at the time when the hand stretching action was recognized, adds the estimated attribute information, and outputs which attribute person accessed the corresponding product (S811). .

In this way, the information processing apparatus 10 further estimates the attributes of a person using video data for a predetermined period of time, such as one day or one week. It is possible to aggregate whether or not to pick up the product and output it to the user. FIG. 17 is a diagram for explaining an example of the output result screen. As shown in FIG. 17, the information processing apparatus 10 collects information on the number of people who passed in front of the shelf and the number of people who stayed in front of the shelf during each time period of the store, and various information generated using attribute information. Conversion analysis results 50 may be generated and output that include: This conversion analysis result 50 includes an image 51 to be acquired, the ratio of genders who picked up the product with their left hand and the total result 52 for each time period, the ratio of age groups who picked up the product with their left hand and the number of hours for each time period. The tally result 53 and the like are included.

Although the embodiments of the present invention have been described so far, the present invention may be implemented in various different forms other than the embodiments described above.

[Numbers, etc.]
Numerical examples, operation examples, screen examples, attribute examples, and the like used in the above embodiments are only examples, and can be arbitrarily changed. Also, a cloud system can be adopted. For example, it is possible to upload the results processed by an edge terminal and display the results in a browser via the cloud. You can also upload camera footage, process it in the cloud, and display the results in a browser.

Data to be detected is not limited to video data, and may be image data or moving image data. Also, an example of recognizing an action of extending a hand and calculating the position of the extended hand has been described, but the present invention is not limited to this, and various actions of a person can be detected. For example, the information processing apparatus 10 can also detect tampering with a shelf by detecting a motion of stretching a leg. In addition, the information processing device 10 can also recognize an action of activating a sensor in the trunk of a car with a foot in a parking lot, and collect and output information useful for vehicle development.

[Action recognition example]
For example, when a continuous motion is estimated from a plurality of frames (image data) in a video frame, the information processing device 10 estimates that it is a stretching motion, thereby suppressing erroneous detection and improving estimation accuracy. be able to. FIG. 18 is a diagram for explaining motion estimation processing using motion transitions. Specifically, as shown in FIG. 18, the information processing apparatus 10 performs (1) an action of coming in front of the shelf, (2) an action of looking at the front of the shelf, standing or crouching in front of the shelf, or putting one's hand forward. , (3) When a motion of extending a hand is continuously detected within a predetermined frame, it can be estimated as a specific motion that shifts to recognition of the motion of extending a hand.

For example, in the first frame, the information processing apparatus 10 detects an action of "come in front of the shelf" in the passage ROI 40 in front of the shelf ROI 30 although it is at least a predetermined distance away from the shelf ROI 30 . Next, the information processing apparatus 10 detects an action of “looking at the front of the shelf” at a position less than a predetermined distance from the shelf ROI 30 in the second frame acquired within the predetermined number of frames from the first frame. After that, the information processing apparatus 10 detects the action of "stretching the hand" in the third frame acquired within the predetermined number of frames from the second frame. In this way, the information processing apparatus 10 can also recognize a hand-stretching action when an assumed continuous action is detected in advance.

Although an example in which a continuous motion is estimated from a plurality of frames (image data) in a video frame has been described here, the present invention is not limited to this. For example, the information processing apparatus 10 can also recognize a hand-stretching action when a plurality of pre-specified actions are estimated in a pre-specified order.

[system]
Information including processing procedures, control procedures, specific names, and various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified.

Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. That is, the specific forms of distribution and integration of each device are not limited to those shown in the drawings. That is, all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions.

Further, each processing function performed by each device may be implemented in whole or in part by a CPU and a program analyzed and executed by the CPU, or implemented as hardware based on wired logic.

[hardware]
FIG. 19 is a diagram illustrating a hardware configuration example. As shown in FIG. 19, the information processing device 19 has a communication device 10a, a HDD (Hard Disk Drive) 10b, a memory 10c, and a processor 10d. 2 are interconnected by a bus or the like.

The communication device 10a is a network interface card or the like, and communicates with other devices. The HDD 10b stores programs and DBs for operating the functions shown in FIG.

The processor 10d reads from the HDD 10b or the like a program that executes the same processing as each processing unit shown in FIG. 2 and develops it in the memory 10c, thereby operating the process of executing each function described with reference to FIG. 2 and the like. For example, this process executes the same function as each processing unit of the information processing apparatus 10 . Specifically, the processor 10d is similar to the video acquisition unit 21, the person detection unit 22, the skeleton detection unit 23, the motion estimation unit 24, the range estimation unit 25, the hand stretching action recognition unit 26, the hand stretching position calculation unit 27, and the like. from the HDD 10b or the like. Then, the processor 10d performs the same processing as the video acquisition unit 21, the person detection unit 22, the skeleton detection unit 23, the motion estimation unit 24, the range estimation unit 25, the hand stretching action recognition unit 26, the hand stretching position calculation unit 27, and the like. Run the process you want to run.

Thus, the information processing apparatus 10 operates as an information processing apparatus that executes the action recognition method by reading and executing the program. Further, the information processing apparatus 10 can read the program from the recording medium by the medium reading device and execute the read program, thereby realizing the same function as the embodiment described above. Note that the programs referred to in other embodiments are not limited to being executed by the information processing apparatus 10 . For example, the present invention can be applied in the same way when another computer or server executes the program, or when they cooperate to execute the program.

This program can be distributed via a network such as the Internet. Also, this program is recorded on a computer-readable recording medium such as a hard disk, flexible disk (FD), CD-ROM, MO (Magneto-Optical disk), DVD (Digital Versatile Disc), etc., and is read from the recording medium by a computer. It can be executed by being read.

REFERENCE SIGNS LIST 10 information processing device 11 communication unit 12 output unit 13 storage unit 14 machine learning model DB
15 Target location information DB
20 control unit 21 image acquisition unit 22 person detection unit 23 skeleton detection unit 24 motion estimation unit 25 range estimation unit 26 hand stretching action recognition unit 27 hand stretching position calculation unit

Claims (10)

to the computer,
Detecting skeletal information of a person from an image acquired by an imaging device,
Acquiring area information about the position where the person is imaged, the target area, and the position of the target object;
estimating an action performed by the person on the object based on the skeletal information of the person and the area information;
Recognizing the action of the person based on the distance between the person and the object and the estimated action;
An action recognition program characterized by causing a process to be executed. The estimating process includes:
Depending on whether or not any one of the motion of reaching out to the target, the motion of looking at the target, or the motion of facing the target is met, the person may move toward the target. estimating the action to be taken,
The recognizing process includes:
When the distance between the person and the object is less than a threshold and an action that satisfies any of the conditions is estimated, the person recognizes the action of reaching out to the object.
2. The action recognition program according to claim 1, characterized by: The recognizing process includes:
When the distance between the person and the object is less than a threshold and a plurality of pre-specified actions are estimated in a pre-specified order within a predetermined time, the person reaches out to the object. act and recognize
3. The action recognition program according to claim 2, characterized by: causing the computer to perform processing for estimating, as the distance between the person and the object, a reachable range of the person from the skeleton information of the person and the region information;
The recognizing process includes:
Using the estimated motion and the reachable range of the person, recognizing the action of the person reaching out to the object;
4. The action recognition program according to any one of claims 1 to 3, characterized by: The process of estimating the reachable range of the person includes:
detecting the center point of both legs specified by the skeleton information as the position where the person stands;
calculating the shortest distance from the center point of both feet to the object;
calculating a perpendicular line in the height direction of the object with respect to the shortest distance;
estimating the position of the perpendicular line corresponding to the height from the foot to the shoulder specified by the skeleton information as the reachable range of the person;
5. The action recognition program according to claim 4, characterized by: The process of estimating the reachable range of the person includes:
calculating the shoulder width of the person based on the skeleton information;
estimating the area corresponding to the shoulder width at the position of the perpendicular line corresponding to the height from the foot to the shoulder specified by the skeleton information as the reachable range of the person;
6. The action recognition program according to claim 5, characterized by: calculating the position of the hand based on the skeleton information and the area information when the person is recognized as reaching out to the object;
7. The method according to any one of claims 1 to 6, wherein the computer is caused to execute a process of specifying the object to be accessed by the hand, based on the position of the hand and the positional relationship of the object. Action recognition program as described. causing the computer to execute a process of estimating attributes of the person based on the skeleton information;
The recognizing process includes:
outputting a result of recognizing that the person reached out his or her hand toward the object in association with attributes of the person;
8. The action recognition program according to any one of claims 1 to 7, characterized by: the computer
Detecting skeletal information of a person from an image acquired by an imaging device,
Acquiring area information about the position where the person is imaged, the target area, and the position of the target object;
estimating an action performed by the person on the object based on the skeletal information of the person and the area information;
Recognizing the action of the person based on the distance between the person and the object and the estimated action;
An action recognition method characterized by executing processing. a detection unit that detects human skeleton information from an image acquired by an imaging device;
an acquisition unit that acquires area information about the position where the person is imaged, the target area, and the position of the target object;
an estimation unit for estimating an action performed by the person on the object based on the skeletal information of the person and the area information;
a recognition unit that recognizes the action of the person based on the distance between the person and the object and the estimated action;
An information processing device comprising:

Images