JP2023080835A - Setting program, setting method, and information processing apparatus - Google Patents

Abstract

To provide a setting program, a setting method, and an information processing apparatus capable of suppressing deterioration in the accuracy of behavior analysis.SOLUTION: An information processing apparatus identifies a first area corresponding to a passage from image data obtained by capturing interior, and identifies a body direction and face direction of a person moving in the identified first area. When the identified body direction of the person is different from the face direction of the person, the information processing apparatus identifies a second area adjacent to the first area among a plurality of areas constituting the interior on the basis of the face direction of the person or the body direction of the person, and sets the identified second area as an area where an object related to the person is stored.SELECTED DRAWING: Figure 5

Description

The present invention relates to a setting program, a setting method, and an information processing apparatus.

Techniques for analyzing human behavior from video data captured by cameras are being developed. For example, from each image data included in the video data, we extract the attention area, which is the area where purchasing behavior is likely to occur, and detect the movement of raising the arm to a certain position in the attention area as a picking motion, thereby analyzing the purchase behavior. do. In recent years, as an attention area detection method, manual attention area setting and attention area setting using semantic segmentation are used for each image data.

JP 2012-173903 A JP 2013-50945 A

However, with the above technique, it is difficult to accurately set the attention area, and as a result, the accuracy of behavior analysis deteriorates. For example, in a manual method, an attention area is set for a huge amount of image data, which not only takes time but also makes it difficult to prevent human error. In addition, in the method using semantic segmentation, the entire aisle that the consumer walks in the store is set as the region of interest, so unnecessary picking actions are detected, and the accuracy of behavior analysis deteriorates.

An object of one aspect of the present invention is to provide a setting program, a setting method, and an information processing apparatus capable of suppressing deterioration in accuracy of behavior analysis.

In the first scheme, the setting program causes the computer to specify a first area corresponding to the passage from the image data of the interior, and the direction of the body of the person moving in the specified first area. a face orientation of a person is identified, and when the identified body orientation of the person is different from the person's face orientation, the room is navigated based on the person's face orientation or the person's body orientation; A process of identifying a second area adjacent to a first area among a plurality of areas, and setting the identified second area as an area containing an object related to the person. It is characterized by executing

According to one embodiment, deterioration in accuracy of behavior analysis can be suppressed.

FIG. 1 is a diagram illustrating the overall configuration of a system including an information processing apparatus according to a first embodiment; FIG. 2 is a diagram for explaining behavior of a recognition target according to the first embodiment; FIG. 3 is a diagram explaining detection of a region of interest by semantic segmentation. FIG. 4 is a diagram illustrating detection of a region of interest in the reference technique. FIG. 5 is a functional block diagram of the functional configuration of the information processing apparatus according to the first embodiment; FIG. 6 is a diagram illustrating generation of the first machine learning model. FIG. 7 is a diagram for explaining motion analysis using the second machine learning model. FIG. 8 is a diagram illustrating generation of a user's movement trajectory. FIG. 9 is a diagram for explaining plots of face orientation and body orientation. FIG. 10 is a diagram for explaining extraction of a region of interest. FIG. 11 is a diagram for explaining the setting of the product shelf area. FIG. 12 is a flowchart showing the flow of processing. FIG. 13 is a diagram illustrating a hardware configuration example.

Hereinafter, embodiments of a setting program, a setting method, and an information processing apparatus disclosed in the present application will be described in detail based on the drawings. In addition, this invention is not limited by this Example. Moreover, each embodiment can be appropriately combined within a range without contradiction.

[overall structure]
FIG. 1 is a diagram illustrating the overall configuration of a system including an information processing apparatus 10 according to the first embodiment. As shown in FIG. 1, this system includes a store 1, which is an example of a space, a plurality of cameras 2 installed at different locations within the store 1, and an information processing device 10. FIG.

Each of the plurality of cameras 2 is an example of a monitoring camera that captures an image of a predetermined area within the store 1 and transmits captured image data to the information processing apparatus 100 . In the following description, video data may be referred to as "video data". Video data also includes a plurality of time-series image frames. A frame number is assigned to each image frame in ascending chronological order. One image frame is image data of a still image captured by the camera 2 at a certain timing.

The information processing device 10 is an example of a computer that analyzes each image data captured by each of the plurality of cameras 2 . Note that each of the plurality of cameras 2 and the information processing apparatus 10 are connected using various networks such as the Internet and dedicated lines, regardless of whether they are wired or wireless.

In recent years, technology development for analyzing human behavior from video data captured by the camera 2 has been advanced. For example, from each image data included in the video data, we extract the attention area, which is the area where purchasing behavior is likely to occur, and detect the movement of raising the arm to a certain position in the attention area as a picking motion, thereby analyzing the purchase behavior. is being done.

FIG. 2 is a diagram for explaining behavior of a recognition target according to the first embodiment; Assume that the region A shown in FIG. 2 is the region of interest. In this case, the picking motion of the person (user) positioned in front of the product shelf shown in (1) of FIG. 2 is the recognition target. However, as shown in (2) and (3) in Fig. 2, a person who does not reach out to pick a product in a place where there is no product shelf but performs a similar picking action is also recognized, resulting in erroneous detection. occurs.

In order to reduce erroneous detection, it is conceivable to set the area of the product shelf where the hand reaches as the attention area. In that case, the person shown in FIG. Therefore, it is erroneously detected. Another method is to set the passage at your feet as the attention area. For example, when the area A shown in FIG. 1 is set as the attention area, the persons shown in (2) and (3) in FIG. 2 are not detected, and erroneous detection is suppressed.

Manual setting is often used to set the foot area as the attention area. However, with manual setting, the attention area is set for a huge amount of image data, which not only takes time but also makes it difficult to prevent human error.

Another technique uses automatic configuration through semantic segmentation, a technique that categorizes what is in each pixel of image data. FIG. 3 is a diagram explaining detection of a region of interest by semantic segmentation. As shown in FIG. 3, semantic segmentation inputs image data into a machine learning model (convolutional encoder-decoder) and obtains an output result in which each region of the image data is labeled. However, the label "passage" is set not only for the attention area but also for all passages including areas other than the attention area, and the persons (2) and (3) in FIG. An erroneous detection occurs when a person's picking action is detected.

In addition, it is possible to use a reference technology that automatically provides ROI (Region Of Interest) by extracting the working positions of people from the image data of the camera and clustering the working positions. FIG. 4 is a diagram illustrating detection of a region of interest in the reference technique. As shown in FIG. 4, in the reference technique, since the area where the purchase behavior occurs while still is extracted, only the position where the person is stationary shown in FIG. difficult to adequately cover. In other words, with the reference technology, it is difficult to detect an action of moving slowly to pick up an item (picking action).

In general, in image data, areas set as attention areas and areas where picking motions are detected are set as product shelf areas where products picked up by people are displayed, and then human behavior analysis, etc. is performed. is done. However, as described above, it is difficult to accurately extract the attention area by manual setting, semantic segmentation, or setting the attention area by the reference technique. For this reason, a product shelf area setting error occurs, and the accuracy of the final behavior analysis also deteriorates.

Therefore, the information processing apparatus 10 according to the first embodiment identifies the first area corresponding to the passage from the image data of the interior of the room. The information processing apparatus 10 measures the orientation of the body and face of the person moving in the specified first area. When the measured orientation of the person's body is different from the orientation of the person's face, the information processing device 10 selects one of the plurality of areas within the store 1 based on the orientation of the person's face or body. A second area adjacent to the first area is identified. The information processing device 10 sets the specified second area as the product shelf area.

That is, in the information processing apparatus 10, since movement and selection of products are mainly generated in the purchase behavior at the retail store, the face and the body face the same direction when moving, and when the selection is performed, the face and the body face the same direction. A region of interest is extracted using variations in the direction of the body. As a result, the information processing apparatus 10 can accurately extract the region of interest without error, thereby suppressing deterioration in accuracy of behavior analysis.

[Function configuration]
FIG. 5 is a functional block diagram of the functional configuration of the information processing apparatus 10 according to the first embodiment. As shown in FIG. 5 , the information processing device 10 has a communication section 11 , a storage section 12 and a control section 20 .

The communication unit 11 is a processing unit that controls communication with other devices, and is realized by, for example, a communication interface. For example, the communication unit 11 receives video data from the camera 2 and transmits the result of processing by the control unit 20 to a management terminal or the like.

The storage unit 12 is a processing unit that stores various data, programs executed by the control unit 20, and the like, and is implemented by a memory, a hard disk, or the like. The storage unit 12 stores a training data DB 13, a first machine learning model 14, a second machine learning model 15, a video data DB 16, a segment result DB 17, an ROI information DB 18, and a setting result DB 19.

The training data DB 13 is a database that stores each training data used for training the first machine learning model 14 . Specifically, each training data consists of RGB image data as an explanatory variable and semantic segmentation execution results for the image data as an objective variable (correct information) (hereinafter referred to as a segment result or a segmentation result). ) are associated with each other.

The first machine learning model 14 is a model that performs semantic segmentation. Specifically, the first machine learning model 14 outputs a segmentation result according to the input of RGB image data. The segmentation result sets an identified label for each region in the image data. For example, the first machine learning model 14 can employ a convolutional encoder-decoder or the like.

The second machine learning model 15 is a model for executing motion analysis. Specifically, the second machine learning model 15 is a machine-learned model, and two-dimensional joint positions (skeletal coordinates) such as the head, wrists, waist, and ankles are calculated for two-dimensional image data of a person. It is an example of a deep learner that infers, recognizes basic actions, and recognizes user-defined rules. By using this second machine learning model 15, it is possible to recognize the basic motion of a person, and to acquire the ankle position, face orientation, and body orientation. Note that basic motions are, for example, walking, running, and stopping. The rules defined by the user are, for example, the transition of skeleton information corresponding to each action until picking up the product.

The image data DB 16 is a database that stores image data captured by each of the plurality of cameras 2 installed in the store 1 . For example, the video data DB 16 stores video data for each camera 2 or for each time period during which the image was captured.

The segment result DB 17 is a database that stores execution results of semantic segmentation. Specifically, the segment result DB 17 stores the output result of the first machine learning model 14 . For example, the segment result DB 17 associates and stores RGB image data and execution results of semantic segmentation.

The ROI information DB 18 is a database that stores ROIs of attention areas, ROIs of product shelves, and the like obtained by the control unit 20, which will be described later. For example, the ROI information DB 18 associates and stores the ROI of the attention area, the ROI of the product shelf, and the like for each RGB image data.

The setting result DB 19 is a database that stores the result of setting the product shelf area for the segment result by the control unit 20, which will be described later. For example, the setting result DB 19 associates and stores RGB image data and setting information for each label set for the image data.

The control unit 20 is a processing unit that controls the entire information processing apparatus 10, and is implemented by, for example, a processor. The control unit 20 has a pre-learning unit 21 , an acquisition unit 22 , a motion analysis unit 23 , a variation extraction unit 24 , a trajectory generation unit 25 , an attention area extraction unit 26 and an area setting unit 27 . The pre-learning unit 21, the acquisition unit 22, the motion analysis unit 23, the variation extraction unit 24, the trajectory generation unit 25, the attention area extraction unit 26, and the area setting unit 27 are implemented by electronic circuits possessed by the processor, processes executed by the processor, and the like. It is realized by

The pre-learning unit 21 is a processing unit that generates the first machine learning model 14 . Specifically, the pre-learning unit 21 executes training of the first machine learning model by machine learning using each training data stored in the training data DB 13 .

FIG. 6 is a diagram illustrating generation of the first machine learning model 14. As shown in FIG. As shown in FIG. 6, the pre-learning unit 21 inputs training data including RGB image data and correct information (segmentation result) to the first machine learning model 14 and acquires an output result (segmentation result). Then, the pre-learning unit 21 optimizes the parameters and the like of the first machine learning model 14 so as to minimize the error between the correct information of the training data and the output result.

The acquisition unit 22 is a processing unit that acquires video data from each camera 2 and stores it in the video data DB 16 . For example, the acquisition unit 22 may acquire from each camera 2 at any time or periodically.

The motion analysis unit 23 is a processing unit that analyzes the motion of a person appearing in the video data captured by the camera 2 . Specifically, the motion analysis unit 23 inputs each image data (frame) included in the video data to the second machine learning model 15, and recognizes the motion of the person appearing in each image data.

FIG. 7 is a diagram for explaining motion analysis using the second machine learning model 15. As shown in FIG. As shown in FIG. 7, the motion analysis unit 23 inputs the RGB image data to the second machine learning model 15 and acquires the two-dimensional skeleton coordinates of the person appearing in the image data. Then, the motion analysis unit 23 identifies the ankle position, face orientation, and body orientation of the person according to the two-dimensional skeletal coordinates, and outputs the identified results to the variation extraction unit 24, the trajectory generation unit 25, and the like. .

In this way, the motion analysis unit 23 inputs each image data (for example, 100 frames) included in each video data acquired at predetermined time intervals to the second machine learning model 15, and By measuring the ankle position, face orientation, and body orientation of the person, it is possible to identify the transition of the ankle position, the face orientation, and the body orientation transition in the video data.

The variation extraction unit 24 is a processing unit that uses the two-dimensional skeletal coordinates of the person specified by the motion analysis unit 23 to extract variations in the orientation of the body and the orientation of the face of the person. In the above example, the variation extractor 24 acquires from the motion analyzer 23 the orientation of the face and the orientation of the body for each piece of image data (for example, 100 frames) included in the video data. Subsequently, the variation extracting unit 24 calculates the angle formed by the face directions of the persons in each image data as the variation.

The trajectory generation unit 25 is a processing unit that generates a movement trajectory of each person appearing in the video data. Specifically, the trajectory generation unit 25 plots the positions of the person's ankles obtained by the motion analysis unit 23 on the execution result of semantic segmentation for the image data in the video data, thereby generating the movement trajectory of the person. do.

FIG. 8 is a diagram illustrating generation of a user's movement trajectory. As shown in FIG. 8 , the trajectory generator 25 inputs image data (for example, the last image data) in video data to the first machine learning model 14 . Then, the motion analysis unit 23 acquires a segmentation result in which regions (areas) are identified by the first machine learning model 14 and labels are set for each area.

After that, the trajectory generation unit 25 identifies the region of the passage to which the label “passage” is set from each label included in the segmentation result. Subsequently, the trajectory generation unit 25 plots the positions of the ankles of each person identified from each image data in the video data as a trajectory with respect to the area of the passage. The trajectory generation unit 25 then outputs the plot result to the attention area extraction unit 26 .

In this way, the trajectory generation unit 25 can generate a movement trajectory along which a person appearing in the video data moves through the area of the passage.

The attention area extraction unit 26 is a processing unit that extracts an attention area based on the variation between the orientation of the body and the orientation of the face of the person. Specifically, the region-of-interest extraction unit 26 selects, from among the movement trajectories generated by the trajectory generation unit 25, a region that includes a movement trajectory in which the angle between the face orientations of the person is equal to or greater than a threshold value. Extract as a region of interest.

FIG. 9 is a diagram for explaining plotting of the orientation of the face and the orientation of the body, and FIG. 10 is a diagram for explaining the extraction of the region of interest. As shown in FIG. 9 , the attention area extraction unit 26 plots the face direction and body direction of the person specified by the motion analysis unit 23 on the movement trajectory generated by the trajectory generation unit 25 . Subsequently, the attention area extracting unit 26 identifies the angle of the person's face orientation and body orientation for each trajectory based on the angles (variation) calculated by the variation extracting unit 24 .

After that, as shown in FIG. 10, the region-of-interest extraction unit 26 clusters the point group of the movement trajectory based on variations in the orientation of the face and the orientation of the body. Then, the attention area extracting unit 26 extracts the areas A1 and A2 clustered as having an angle equal to or larger than the threshold and having large variations as attention areas, and extracts the area A3 clustered as having an angle less than the threshold and having small variations. is extracted as the passage region.

In this manner, the attention area extracting unit 26 can narrow down the attention area, which is the target area for the behavior analysis of the person in the video data and the target area for detecting the picking motion for the product. Note that the attention area extracting unit 26 is not limited to clustering, and can also use a method such as extracting each area that includes the maximum number of trajectories whose angles are equal to or greater than a threshold as attention areas.

The area setting unit 27 identifies a second area adjacent to the first area corresponding to the attention area among the plurality of areas constituting the room, and designates the identified second area as an object related to the person. is a processing unit that sets an area in which is stored. Specifically, the area setting unit 27 assigns an area, which is in contact with the attention area extracted by the attention area extraction unit 26, among the areas to which each label obtained from the execution result of the semantic segmentation is attached, to the product shelf area. set.

FIG. 11 is a diagram for explaining the setting of the product shelf area. As shown in FIG. 11 , the area setting unit 27 acquires the coordinates of a polygon surrounding the trajectory (point group) included in each cluster extracted as the attention area by the attention area extraction unit 26 . Then, the area setting unit 27 specifies the area C and the area D surrounded by polygons. After that, the area setting unit 27 stores the coordinates of the polygon for generating the area C and the coordinates of the polygon for generating the area D in the ROI information DB 18 as the attention area.

In addition, the area setting unit 27 associates each point group (trajectory) of the polygon that generates the area C with the orientation of the face specified by the variation extraction unit 24 . Then, the area setting unit 27 sets area C1 and area C2 adjacent to the aisle area (label: aisle) including the area C among the areas in the face direction as the product shelf area. That is, the area setting unit 27 resets the labels set for each of the areas C1 and C2 in the segmentation result to labels indicating product shelves.

Similarly, the area setting unit 27 associates each point group (trajectory) of the polygon that generates the area D with the orientation of the face identified by the variation extraction unit 24 . Then, the area setting unit 27 sets the area C3 and the area C4 adjacent to the aisle area including the area D among the areas in the direction of the face direction as the product shelf area. That is, the area setting unit 27 resets the labels set for each of the areas C3 and C4 in the segmentation result to labels indicating product shelves.

After that, the area setting unit 27 stores the segmentation result with each label set in the setting result DB 19 . In this manner, the area setting unit 27 can set the attention area for which the picking operation is specified and the product shelf area for which the picking operation is to be performed for the segmentation result generated from the video data. can.

Note that the area setting unit 27 can perform the above settings not only for the representative image data in the video data, but also for each image data. The setting of the area and the setting of the product shelf area can also be applied to each image data. That is, the area setting unit 27 uses the setting result of the product shelf area using the segmentation result to set the regions corresponding to the areas C1, C2, C3, and C4 for each image data of the video data to the product shelf area. Area can also be set.

[Process flow]
FIG. 12 is a flowchart showing the flow of processing. Here, an example in which the first machine learning model 14 has already been generated by pre-learning will be described.

As shown in FIG. 12, when the start of processing is instructed (S101: Yes), the motion analysis unit 23 executes motion analysis based on the video data acquired by the acquisition unit 22 (S102). Then, the motion analysis unit 23 detects the orientation of the person's face and the like based on the motion analysis (S103). For example, the motion analysis unit 23 inputs each image data included in the video data to the second machine learning model, identifies the two-dimensional skeleton information of the person included in each image data and the transition of the two-dimensional skeleton information, The ankle position, face direction, and body direction of each person are detected.

Subsequently, the trajectory generation unit 25 inputs the image data included in the video data to the first machine learning model 14, and acquires the segmentation result, which is the execution result of semantic segmentation (S104).

Then, the trajectory generation unit 25 generates the movement trajectory of the person from each image data included in the video data (S105). For example, the trajectory generation unit 25 generates the movement trajectory of each person by plotting the positions of the ankles specified for the person in each image data on the segmentation result.

After that, the region-of-interest extraction unit 26 plots the orientation of the face and the orientation of the body on each trajectory in the segmentation results in which the trajectories are plotted (S106). Then, the region-of-interest extraction unit 26 detects variations between the orientation of the face and the orientation of the body (S107). For example, the region-of-interest extracting unit 26 acquires the angle formed by each vector of the orientation of the face and the orientation of the body for each movement trajectory as a variation.

Subsequently, the attention area extracting unit 26 performs clustering based on variations in the orientation of the face and the orientation of the body (S108), and extracts the attention area based on the clustering result (S109). For example, the attention area extracting unit 26 extracts a cluster of trajectories whose angles are equal to or greater than a threshold as attention areas.

After that, the area setting unit 27 generates the ROI coordinates of the attention area (S110), and sets the product shelf area (S111). For example, the area setting unit 27 generates, as ROI coordinates, the coordinates of a polygon surrounding the trajectory (point group) included in the cluster. In addition, the area setting unit 27 sets, as the product shelf area, an area that is adjacent to the attention area and that is in the vector direction of the face orientation associated with the trajectory.

[effect]
As described above, the information processing apparatus 10 generates a segmentation result for segmenting the image data into regions, re-extracts the passage region from the segmentation result and the motion analysis result, and detects variations in face orientation and body orientation from the motion analysis result. to extract Then, the information processing apparatus 10 extracts an attention area by clustering from the passage area and the variation information. As a result, the information processing apparatus 10 can automatically provide an appropriate region of interest.

For example, by using the information processing apparatus 10, there is no need to manually set the attention area, so human error can be reduced, and compared to manual setting, accurate and high-speed attention can be paid to a huge amount of image data. Region setting can be implemented. In addition, the information processing apparatus 10 can extract, as an attention area, an area in which an action of moving the face that a person is interested in is performed. Therefore, unlike the reference technique in FIG. .

In addition, the information processing apparatus 10 can identify a product shelf as an area adjacent to the correct area of interest. A picking action to pick up a product can be detected. As a result, the information processing apparatus 10 can improve the detection accuracy of the picking motion, and can improve the accuracy of action analysis and the like.

Although the embodiments of the present invention have been described so far, the present invention may be implemented in various different forms other than the embodiments described above.

[Numbers, etc.]
Numerical examples, the number of cameras, label names, the number of trajectories, etc. used in the above embodiment are only examples, and can be changed arbitrarily. Also, the flow of processing described in each flowchart can be changed as appropriate within a consistent range. In addition, in the above embodiment, the store was explained as an example, but the present invention is not limited to this, and can be applied to, for example, a warehouse, a factory, a classroom, a train car, an airplane cabin, and the like. In these cases, instead of the product shelf area described as an example of the area in which objects related to people are stored, an area where objects are placed and an area where luggage is stored are detected and set.

Also, in the above embodiment, an example using the position of the person's ankle has been described, but the present invention is not limited to this, and the position of the foot, the position of the shoe, or the like can also be used. In the above embodiment, an example was described in which the area in the direction of the face is specified as the product shelf area, but the area in the direction of the body can also be specified as the product shelf area. Also, each machine learning model can use a neural network or the like.

[system]
Information including processing procedures, control procedures, specific names, and various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified.

Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. That is, the specific forms of distribution and integration of each device are not limited to those shown in the drawings. That is, all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions.

Further, each processing function performed by each device may be implemented in whole or in part by a CPU and a program analyzed and executed by the CPU, or implemented as hardware based on wired logic.

[hardware]
FIG. 13 is a diagram illustrating a hardware configuration example. As shown in FIG. 13, the information processing device 10 has a communication device 10a, a HDD (Hard Disk Drive) 10b, a memory 10c, and a processor 10d. 13 are interconnected by a bus or the like.

The communication device 10a is a network interface card or the like, and communicates with other devices. The HDD 10b stores programs and DBs for operating the functions shown in FIG.

The processor 10d reads from the HDD 10b or the like a program that executes the same processing as each processing unit shown in FIG. 5 and develops it in the memory 10c, thereby operating the process of executing each function described with reference to FIG. 5 and the like. For example, this process executes the same function as each processing unit of the information processing apparatus 10 . Specifically, the processor 10d has functions similar to those of the pre-learning unit 21, the acquisition unit 22, the motion analysis unit 23, the variation extraction unit 24, the trajectory generation unit 25, the attention area extraction unit 26, the area setting unit 27, and the like. A program is read from the HDD 10b or the like. Then, the processor 10d executes processes similar to those of the pre-learning unit 21, the acquisition unit 22, the motion analysis unit 23, the variation extraction unit 24, the trajectory generation unit 25, the attention area extraction unit 26, the area setting unit 27, and the like. Execute.

Thus, the information processing apparatus 10 operates as an information processing apparatus that executes the setting method by reading and executing the program. Further, the information processing apparatus 10 can read the program from the recording medium by the medium reading device and execute the read program, thereby realizing the same function as the embodiment described above. Note that the programs referred to in other embodiments are not limited to being executed by the information processing apparatus 10 . For example, the above embodiments may be similarly applied when another computer or server executes the program, or when they cooperate to execute the program.

This program may be distributed via a network such as the Internet. In addition, this program is recorded on a computer-readable recording medium such as a hard disk, flexible disk (FD), CD-ROM, MO (Magneto-Optical disk), DVD (Digital Versatile Disc), etc., and is read from the recording medium by a computer. It may be executed by being read.

10 Information Processing Device 11 Communication Unit 12 Storage Unit 13 Training Data DB
14 first machine learning model 15 second machine learning model 16 video data DB
17 Segment result DB
18 ROI information DB
19 Setting result DB
20 control unit 21 pre-learning unit 22 acquisition unit 23 motion analysis unit 24 variation extraction unit 25 trajectory generation unit 26 attention area extraction unit 27 area setting unit

Claims (8)

to the computer,
Identify the first area corresponding to the passage from the image data taken in the room,
identifying the orientation of the body and the orientation of the face of a person moving in the identified first area;
When the identified orientation of the person's body is different from the orientation of the person's face, the first area among the plurality of areas forming the room is based on the orientation of the person's face or the orientation of the person's body. identifying one area and an adjacent second area;
setting the identified second area as an area containing an object related to the person;
A setting program characterized by executing processing. identifying the position of each person appearing in the video data from each image data in the video data including the image data;
The computer performs a process of identifying a region of interest to be analyzed for behavior of the person in the first area based on the orientation of the person's face and the orientation of the body of the person at the position of each person. and run
The process of specifying
Identifying the second area adjacent to the attention area in the first area;
The setting process is
2. The setting program according to claim 1, wherein said second area is set as an area in which an object related to said person is stored. The process of specifying
performing clustering based on the angle formed by the orientation of the face of the person and the orientation of the body of the person;
3. The setting program according to claim 2, wherein, as a result of said clustering, when said angle is equal to or greater than a threshold, an area including said person's position belonging to a cluster is specified as said attention area. The process of specifying
Among a plurality of areas adjacent to the first area in the image data, an area located in a direction of a vector indicating the orientation of the face of the person or the orientation of the body of the person included in the attention area is defined as the first area. 4. The setting program according to claim 3, wherein the second area is specified. The process of specifying
Identifying the first area based on segmentation results of identifying each area of the image data by semantic segmentation;
The process of specifying
5. Inputting the image data to a machine learning model that has undergone machine learning, and identifying the orientation of the person's body and the person's face based on the output result of the machine learning model. A configuration program according to any one of . The setting process is
6. The setting program according to claim 5, wherein the second area is set for the segment result or the image data on which the segment result is based. the computer
Identify the first area corresponding to the passage from the image data taken in the room,
identifying the orientation of the body and the orientation of the face of a person moving in the identified first area;
When the identified orientation of the person's body is different from the orientation of the person's face, the first area among the plurality of areas forming the room is based on the orientation of the person's face or the orientation of the person's body. identifying one area and an adjacent second area;
setting the identified second area as an area containing an object related to the person;
A setting method characterized by executing processing. Identify the first area corresponding to the passage from the image data taken in the room,
identifying the orientation of the body and the orientation of the face of a person moving in the identified first area;
When the identified orientation of the person's body is different from the orientation of the person's face, the first area among the plurality of areas forming the room is based on the orientation of the person's face or the orientation of the person's body. identifying one area and an adjacent second area;
setting the identified second area as an area containing an object related to the person;
An information processing apparatus comprising a control unit.

Images