JP7337354B2 - Information processing device and information processing program - Google Patents

Description

The present invention relates to an information processing device and an information processing program.

Non-Patent Document 1 describes a store where payment is automatically completed without waiting in line at the register.
Non-Patent Document 2 describes a product shelf image recognition engine that can analyze product display status in real time by photographing product shelves in retail stores such as supermarkets and convenience stores with a smartphone or the like.
Non-Patent Document 3 describes a technology for recognizing face images and the like.
Non-Patent Document 4 describes items that businesses need to consider from the viewpoint of privacy protection for consumers when utilizing camera images.
Non-Patent Document 5 describes that anonymization of personal data is important.

What is Amazon Go, where is it, and how does it work?, https://www.pocket-lint.com/phones/news/amazon/139650-what-is-amazon-go-where-is-it- and-how-does-it-work NTT DoCoMo Develops Product Shelf Image Recognition Engine - DoCoMo's AI Technology Shows Product Shelf Layout at a Glance -, https://www.nttdocomo.co.jp/info/news_release/2018/03/01_01. html MathWorks, Image Recognition, https://jp.mathworks.com/discovery/image-recognition.html Ministry of Economy, Trade and Industry, Camera Image Utilization Guidebook, http://www.meti.go.jp/press/2016/01/20170131002/20170131002-1.pdf National Institute of Informatics, Personal Data Utilization (NII Today), https://www.nii.ac.jp/userdata/results/pr_data/NII_Today/63/p8-9.pdf

2. Description of the Related Art An image of a person, such as a customer in a store, is captured to obtain marketing data.
However, images taken of people are personal data, and in order to use such images, it is necessary to comply with the Personal Information Protection Law and consider the privacy of consumers, so it is necessary to perform anonymization processing. . For example, it is necessary to subject camera images to mosaic processing or the like to process them so that a specific individual cannot be identified.
SUMMARY OF THE INVENTION It is an object of the present invention to provide an information processing apparatus and an information processing program for photographing a person and using wireframe data in which a specific individual cannot be identified when analyzing the behavior of the person. there is

The gist of the present invention for achieving this object lies in the following inventions. The "claims" below are the claims as originally filed.
The invention of claim 1 converts an image of a person photographed into wireframe data in which the parts of the person are expressed in vectors, and uses a wireframe conversion means for anonymizing the person, and the wireframe data, generation means for generating human movement trajectory data representing movements of body parts of the person; analysis means for analyzing the behavior of the person using the human movement trajectory data and predetermined behavior pattern data; The information processing apparatus is characterized by having notification means for notifying analysis results by the analysis means.

According to a second aspect of the present invention, the image is a photographed image of a person in the store, and a second generation method for generating commodity movement trajectory data representing a movement trajectory of the commodity extracted from the photographed image of the commodity. means, wherein the behavior pattern data is pattern data indicating behavior of a person with respect to a product, and the analysis means analyzes the pattern data from the human movement trajectory data and the product movement trajectory data 2. The information processing apparatus according to claim 1, which determines whether or not

In the invention according to claim 3, the behavior pattern data is pattern data indicating behaviors of a plurality of persons belonging to the group with respect to the product, and the analysis means estimates the persons belonging to the group, 3. The information processing apparatus according to claim 2, which determines whether or not the pattern data corresponds to the pattern data from the person movement trajectory data and the product movement trajectory data.

The invention according to claim 4 further comprises recognition means for recognizing a person's face from a photographed image of the person, and the generation means uses the recognition result of the recognition means to generate human movement trajectory data for the same person. 2. The information processing apparatus according to claim 1, which generates a .

In the invention according to claim 5, the recognition result by the recognition means is used only for the generation of the human movement trajectory data by the generation means, and the recognition result by the recognition means is deleted when the person leaves the store. 5. The information processing apparatus according to item 4.

According to a sixth aspect of the present invention, there is provided a photographing means for photographing an image, a conversion means for converting the image photographed by the photographing means into wire frame data expressing a human part as a vector, and an image photographed by the photographing means. and output means for outputting the wire frame data converted by the conversion means.

The invention of claim 7 comprises a computer that converts a photographed image of a person into wireframe data that expresses parts of the person in vectors, and anonymizes the person. generating means for generating human movement trajectory data expressing movement of parts of said person using said human movement trajectory data; and analysis means for analyzing said person's behavior using said human movement trajectory data and predetermined behavior pattern data. and an information processing program characterized by functioning as notification means for notifying analysis results by the analysis means.

According to an eighth aspect of the invention, a computer having a photographing means for photographing an image is provided with conversion means for converting an image photographed by the photographing means into wire frame data in which human parts are vector-expressed, and photographing by the photographing means. The image capturing program functions as an output means for outputting the wire frame data converted by the conversion means instead of the converted image.

According to the information processing apparatus of claim 1, in photographing a person and analyzing the behavior of the person, it is possible to use wire frame data in which a specific individual cannot be identified.

According to the information processing apparatus of claim 2, it is possible to photograph a person in the store and analyze the behavior of the person with respect to the product.

According to the information processing apparatus of claim 3, it is possible to analyze actions taken by a plurality of people belonging to a group with respect to a product.

According to the information processing apparatus of claim 4, more accurate human movement trajectory data can be generated than when face recognition is not used.

According to the information processing apparatus of claim 5, the face recognition result can be used only for the generation of human movement trajectory data.

According to the image capturing apparatus of claim 6, it is possible to output wire-frame data in which a specific individual cannot be identified, instead of an image.

According to the information processing program of claim 7, in photographing a person and analyzing the behavior of the person, wire frame data in which a specific individual cannot be identified can be used.

According to the image capturing program of claim 8, it is possible to output wire frame data in which a specific individual cannot be identified, instead of an image.

2 is a conceptual module configuration diagram of configuration example (1) of the first embodiment; FIG. FIG. 4 is a conceptual module configuration diagram of configuration example (2) of the first embodiment; FIG. 11 is a conceptual module configuration diagram of configuration example (3) of the first embodiment; It is explanatory drawing which shows the example at the time of installing the structure of this Embodiment in a store. FIG. 3 is an explanatory diagram showing a configuration example of a model according to the embodiment; 4 is a flow chart showing an example of processing according to the first embodiment; FIG. 4 is an explanatory diagram showing an example data structure of a human part movement trajectory table; 4 is a flow chart showing an example of processing according to the first embodiment; FIG. 9 is an explanatory diagram showing an example data structure of a product movement trajectory table; 4 is a flow chart showing an example of processing according to the first embodiment; FIG. 10 is an explanatory diagram showing an example of processing for estimating a group; FIG. 10 is an explanatory diagram showing an example of a notification user interface; FIG. 7 is a conceptual module configuration diagram of a configuration example of a second embodiment; 9 is a flow chart showing an example of processing according to the second embodiment; FIG. 9 is an explanatory diagram showing an example data structure of a face movement trajectory table; 9 is a flow chart showing an example of processing according to the second embodiment; 1 is a block diagram showing a hardware configuration example of a computer that implements the embodiment; FIG.

Various preferred embodiments for realizing the present invention will be described below with reference to the drawings.
<First Embodiment>
FIG. 1 shows a conceptual module configuration diagram for configuration example (1) of the first embodiment.
A module generally refers to a component such as software (computer program) or hardware that can be logically separated. Therefore, modules in this embodiment refer not only to modules in a computer program, but also to modules in a hardware configuration. Therefore, the present embodiment provides a computer program for functioning as those modules (a program for causing the computer to execute each procedure, a program for causing the computer to function as each means, a program for causing the computer to function as each function). It also serves as an explanation of the program, system and method for realizing However, for the convenience of explanation, the terms "stored", "stored", and equivalent terms are used, but when the embodiment is a computer program, these terms are stored in a storage device or stored in a storage device. It means to control to store in the device. Also, modules may correspond to functions one-to-one. may consist of multiple programs. Also, multiple modules may be executed by one computer, and one module may be executed by multiple computers by computers in a distributed or parallel environment. Note that one module may include other modules. Further, hereinafter, "connection" is used not only for physical connection but also for logical connection (data transfer, instruction, reference relationship between data, login, etc.). The term "predetermined" means that it is determined before the target process, and not only before the process according to the present embodiment starts, but also after the process according to the present embodiment starts. Also, if it is before the target process, it is used including the meaning that it is determined according to the situation/state at that time or according to the situation/state up to that time. When there are a plurality of "predetermined values", they may be different values, or two or more values (including, of course, all values) may be the same. Also, the description "if A, do B" is used in the sense of "judge whether it is A, and if it is judged to be A, do B". However, this excludes the case where it is not necessary to judge whether or not it is A. In addition, when things are listed like "A, B, C", etc., unless otherwise specified, it is an exemplary list, and includes cases where only one of them is selected (for example, only A).
In addition, a system or device is configured by connecting a plurality of computers, hardware, devices, etc. by communication means such as a network (including one-to-one correspondence communication connection), or a single computer, hardware, device The case where it is realized by etc. is also included. The terms "apparatus" and "system" are used interchangeably. Of course, "system" does not include what is merely a social "mechanism" (social system) that is an artificial arrangement.
In addition, for each process performed by each module or for each process when multiple processes are performed within a module, the target information is read from the storage device, and after the processing is performed, the processing result is written to the storage device. be. Therefore, the description of reading from the storage device before processing and writing to the storage device after processing may be omitted. The storage device here includes a hard disk drive, a RAM (abbreviation for Random Access Memory), an external storage medium, a storage device via a communication line, a register in a CPU (abbreviation for Central Processing Unit), and the like. good too.

The information processing apparatus 100 and the photographing apparatus 150 according to the first embodiment have a function of photographing a person and analyzing the behavior of the person.
In the present embodiment, for example, anonymized marketing behavior detection is performed using a plurality of cameras targeting a wireframe model of human behavior. As a specific example, a plurality of cameras are installed in retail stores (supermarkets, convenience stores, etc.), and anonymized behavior analysis is performed by acquiring 3D wire frames of the human body from the cameras. Furthermore, by combining with the movement trajectory of the product, the trajectory of the action of picking up the product from the shelf and immediately returning it to the shelf, and a few people who are physically close to a large number of candidates are grouped as the same group, The trajectory of the group up to the checkout at the register is provided as marketing data. Here, for example, family members, friends, lovers, etc. correspond to groups.
It should be noted that this method can be applied not only to retail stores, but also to the case of analyzing behavior after anonymizing the person photographed by the camera. For example, sports facilities (eg, gymnasiums, soccer fields, baseball fields, etc.) may be equipped with multiple cameras in a similar manner so that spectator behavior can be analyzed. In the following, as a main example, the case of application to a retail store will be described.

The information processing device 100 and the imaging device 150 are connected via a communication line. The communication line may be wireless, wired, or a combination thereof, and may be, for example, the Internet, an intranet, etc. as a communication infrastructure. Also, the functions of the information processing apparatus 100 may be implemented as a cloud service. 2, a communication line between the camera group 355 and the information processing apparatus 100 described in FIG. 3, and a photographing apparatus 1350 and the information processing apparatus 1300 described in FIG. The communication line, the information processing device 200, and the information processing device 1300 are the same.
The photographing device 150 sends wireframe data or human body part movement trajectory data (hereinafter also referred to as human part movement trajectory data), which is the result of processing by the wireframe module 165 , to the information processing device 100 . The product movement trajectory data (hereinafter also referred to as product movement trajectory data), which is the processing result, is transmitted.

The photographing device 150 has a camera group 155, a wire frame module 165, and a product identification module 170, and is connected to the information processing device 100, as shown in the example of FIG.
The camera group 155 has a camera 160A, a camera 160B, and the like. Camera group 155 has a plurality of cameras 160 .
Camera 160 (camera 160A, camera 160B, etc.) takes a picture of a person. Generally, it is a moving image, but it may be a still image (a plurality of still images) captured at predetermined intervals. A camera 160 is installed in the facility. For example, if it is installed in a retail store, it will photograph people in the store. Specific examples of the camera 160 include commercially available monocular cameras, omnidirectional cameras (also called 360° cameras, VR cameras, and omnidirectional cameras). etc. may be used as they are.

The wireframe module 165 converts an image of a person (an image taken by the camera group 155) into wireframe data representing the parts of the person in vector form, and anonymizes the person. Human parts include parts between joints, such as the head, neck, shoulders, torso, arms (the part from the shoulder to the elbow, the part from the elbow to the hand), hands, and feet (the part from the torso to the knee). part, the part from the knee to the ankle, the part from the ankle to the tip), etc. As a method for generating wireframe data, an already known technique may be used. For example, the joints of the human body (legs, ankles, knees, hip joints, elbows, wrists, necks, etc.) may be detected, and lines connecting the joints may be extracted as wire frames. Therefore, when using wire frame data, it is not possible to identify a specific individual compared to when using the image itself. In other words, the degree of anonymization is enhanced by acquiring only the motion of the human body included in the image, not the image, without performing any conventional mask processing or the like.
Specifically, the wireframe module 165 uses an existing method (for example, the technology described in Non-Patent Document 2) to determine the three-dimensional position (coordinates) of each part of the person in the image within the store. to get

Furthermore, the wireframe module 165 does not convert the “face portion”, which is a human part, into wireframe data (there is no wireframe data of the face portion), or converts it into simplified wireframe data. can be This is because detailed wireframe data that can reproduce the contours of the eyes, nose, mouth, etc., may be able to identify a specific individual. It should be noted that "simplified wireframe data" may be to the extent that a specific individual cannot be identified. Alternatively, only a portion of the face may be wireframe data. Specifically, "only a part of the face" may be set so that only the mouth is converted into wire-frame data, and other parts such as the eyes are not converted into wire-frame data. In this case, it is possible to identify whether or not the person is having a conversation from the mouth wireframe data, but it is not possible to identify a specific individual.
Then, the wireframe module 165 uses the wireframe data to generate human movement trajectory data that expresses the movements of human parts. For example, a human part movement trajectory table 700, which will be described later, is generated.

The product identification module 170 generates product movement trajectory data expressing the movement trajectory of the product extracted from the photographed image of the product. For example, a commodity movement trajectory table 900, which will be described later, is generated.
The wireframe module 165 and product identification module 170 may be identified by an algorithm obtained by machine learning.
Specifically, the product identification module 170 uses an existing method (for example, the technology described in Non-Patent Document 3) to recognize and identify the label of the product in the image, and identify the store of the product. Get the 3D position (coordinates) in the .

As shown in the example of FIG. 1, the information processing apparatus 100 includes a reception module 105, a human body part movement trajectory DB (abbreviation of DataBase) 110, a product movement trajectory DB 115, a behavior analysis module 120, and a notification module 130. It is connected to the imaging device 150 .
The reception module 105 is connected to the human body part movement trajectory DB 110 and the product movement trajectory DB 115 . The receiving module 105 receives human movement trajectory data and product movement trajectory data from the photographing device 150 . Then, the human movement trajectory data is stored in the human body part movement trajectory DB 110 , and the product movement trajectory data is stored in the product movement trajectory DB 115 .

Human body part movement trajectory DB 110 is connected to reception module 105 and behavior analysis module 120 . The human body part movement trajectory DB 110 stores human movement trajectory data.
The product movement trajectory DB 115 is connected with the reception module 105 and the behavior analysis module 120 . The product movement trajectory DB 115 stores product movement trajectory data.

The behavior analysis module 120 includes a person location tracking module 122, a group estimation module 124, a single marketing behavior detection module 126, a group marketing behavior detection module 128, a human body part movement trajectory DB 110, a product movement trajectory DB 115, and a notification module. 130. The behavior analysis module 120 analyzes the behavior of the person(s) in the image. If it's an in-store image, you're analyzing the customer's marketing behavior.

The human position tracking module 122 integrates the human movement trajectory data in the human body part movement trajectory DB 110 and the product movement trajectory data in the product movement trajectory DB 115 to determine what kind of product the target person picks up. or conversely, what kind of merchandise was returned to the shelf. For example, if the shooting date and time (year, month, day, hour, minute, second, seconds or less, or a combination of these) overlap, the person's position and the product's position at that shooting date and time overlap, It can be grasped that the person picked up the product, and if the product returned to the position of the shelf after picking up the product, it should be grasped that the product was returned to the shelf.
Group inference module 124 infers who belongs to a group. For example, (1) the distance between people is calculated, and if the period of time within the predetermined distance is longer than or equal to the predetermined period, the people may be determined to belong to a group; (2) Calculate the distance between people, and if the mouths of people within a predetermined distance are alternately open (that is, if they are talking), determine that people as a group. and (3) when a product is handed over from one person to another person, those people may be judged to belong to the group.

The single marketing behavior detection module 126 uses the human trajectory data and predetermined behavior pattern data to analyze the behavior of the person. For example, pattern data indicating the behavior of a person (single person) with respect to a product may be used as the behavior pattern data. In this case, the single marketing action detection module 126 determines whether the pattern data corresponds to the pattern data from the human movement trajectory data and the product movement trajectory data. The pattern data may be obtained by performing a predetermined action by the tester and using the trajectory of the action as the pattern data. , the trajectory of the action may be created as pattern data. Predetermined actions include, for example, ``returning an item picked up once from the shelf to the shelf immediately'', ``repeating the action of picking up two items alternately from the shelf and putting one item in the basket'', and `` Compare the labels among the same products, decide on one product, and put it in the basket. Then, the number of actions that match these pattern data may be counted. In other words, as an analysis result, for example, how many times "a product once picked up from the shelf is immediately returned to the shelf" occurs. The product may be specified (extracted product name) and counted. For example, the number of times "the product A was once picked up from the shelf and immediately returned to the shelf" has occurred.

The group marketing behavior detection module 128 uses the processing result (people belonging to the group) of the group estimation module 124 to determine whether behavior pattern data corresponds to behavior pattern data from the person movement trajectory data and product movement trajectory data of a plurality of people in the group. determine whether or not The action pattern data here is pattern data indicating actions taken by a plurality of people belonging to a group with respect to a product. The pattern data may be obtained by a plurality of testers performing predetermined actions as a group, and the trajectory of the actions of the plurality of testers may be used as pattern data, or the actions taken by a plurality of customers in the group may be confirmed by the administrator. However, when a predetermined action is performed, the trajectory of the action may be created as pattern data. Predetermined actions include, for example, "three people pick up two items one after another and put one item in the basket", "one person picks up the item but the other person faces the shelf". return to”, etc. Then, the number of actions that match these pattern data may be counted. In other words, as an analysis result, for example, how many times "three people pick up two products one after another and put one product in the basket" occurs. The product may be specified (extracted product) and counted. For example, how many times have three people picked up two products A and B one after another and put one product B in the basket?

Notification module 130 is connected to behavior analysis module 120 . The notification module 130 notifies the user (for example, administrator, etc.) of the analysis result by the behavior analysis module 120 . For example, the processing result by the single marketing behavior detection module 126 and the processing result by the group marketing behavior detection module 128 are output to a terminal such as a manager.

FIG. 2 is a conceptual module configuration diagram of configuration example (2) of the first embodiment.
The parts of the same kind as those shown in the example of FIG. 1 are denoted by the same reference numerals, and redundant explanations are omitted (the same applies hereinafter).
The camera group 155 and the information processing device 200 are connected via a communication line. Note that the camera 160 and the information processing device 200 may be connected individually via a communication line.
The camera group 155 may have a plurality of conventional cameras (cameras 160).
The information processing device 200 has a reception module 105 , a wireframe module 265 , a product identification module 270 , a human body part movement trajectory DB 110 , a product movement trajectory DB 115 , a behavior analysis module 120 , and a notification module 130 , and is connected to the camera group 155 . It is
That is, the information processing apparatus 200 is obtained by adding a wire frame module 265 corresponding to the wire frame module 165 and a product identification module 270 corresponding to the product identification module 170 to the information processing apparatus 100 illustrated in FIG.

The reception module 105 is connected to the wireframe module 265 and the product identification module 270 . Receiving module 105 receives images captured by camera 160 from camera group 155 .
The wireframe module 265 is connected to the reception module 105 and the human body part movement trajectory DB 110 . The wireframe module 265 performs processing equivalent to that of the wireframe module 165 on the image received by the reception module 105 .
The product identification module 270 is connected to the reception module 105 and the product movement trajectory DB 115 . The product identification module 270 performs processing equivalent to that of the product identification module 170 on the image received by the reception module 105 .
The human body part movement trajectory DB 110 is connected to the wireframe module 265 and the behavior analysis module 120 .
The product movement trajectory DB 115 is connected with the product identification module 270 and the behavior analysis module 120 .

FIG. 3 is a conceptual module configuration diagram for configuration example (3) of the first embodiment.
The camera group 355 and the information processing apparatus 100 are connected via a communication line. Note that the camera 360 and the information processing apparatus 100 may be connected individually via a communication line.
The information processing apparatus 100 has a reception module 105 , a human body part movement trajectory DB 110 , a product movement trajectory DB 115 , a behavior analysis module 120 and a notification module 130 , and is connected to a camera group 355 .

Camera group 355 includes camera 360A, camera 360B, etc., and is connected to information processing apparatus 100 .
Each camera 360 has an imaging module 350 , a wireframe module 165 and an item identification module 170 .
The imaging module 350 is a camera that is an example of imaging means for capturing an image.
The wireframe module 165 converts the image captured by the imaging module 350 into wireframe data that represents a human body part as a vector. Then, instead of the image captured by the imaging module 350 , the wireframe data is transmitted to the reception module 105 .
The product identification module 170 extracts the product from the image captured by the imaging module 350 , generates product movement trajectory data expressing the movement trajectory of the product, and transmits the product movement trajectory data to the reception module 105 .

Note that the product identification module 170 may be a function within the information processing apparatus 100 instead of being built in the camera 360, but the wireframe module 165 is built in.
In other words, each camera 360 has a built-in wireframe module 165 that converts the image captured by the imaging module 350 into a wireframe, and outputs only wireframe data (further, product movement trajectory data may be included). However, it may be a camera that does not output images (for example, an image of a person being photographed). Using such a camera 360 eliminates the need for the information processing apparatus 100 to handle personal data that can identify a specific individual. Then, such leakage of personal data can be prevented.

FIG. 4 is an explanatory diagram showing an example when the configuration of this embodiment is installed in a store.
In the store 400, a product shelf 410 (product shelf 410-1, etc.), a register 420 (register 420-1, etc.), and an imaging device 450 (imaging device 450-1, etc.) are installed. For example, the imaging device 450 is installed on the ceiling, wall, or the like of the store 400 . The imaging device 450 corresponds to the imaging device 150, the camera group 155, and the camera group 355 described above.
The customer 430 (customer 430-1, etc.) takes out the product 440 (product 440-1, etc.) from the product shelf 410 (product shelf 410-1, etc.) and picks it up (puts it in a shopping cart, shopping basket, etc.). may be). These actions are photographed by the photographing device 450, and human movement trajectory data is transmitted to the information processing device 100 (or the information processing device 200). The information processing device 100 (or the information processing device 200) analyzes the received human movement trajectory data and outputs the result to the user terminal 490 used by the administrator or the like. Then, the manager or the like can make use of it in estimating the cause of the action. It also tracks the group's trajectory to checkout at the register 420, analyzes who has the power to make purchase decisions, and the percentage of items that are added to the basket immediately and items that are added to the basket after lengthy discussion. It is possible to analyze marketing behavior before purchase, such as analysis, and use it for product development.

FIG. 5 is an explanatory diagram showing a configuration example of a model according to this embodiment.
This embodiment will be described using a conceptual model. In this embodiment, images captured by a plurality of cameras are anonymized by a wireframe model 510 of human motion, and the motion is analyzed by a motion model 530 based on vector representation of each part of the body. , data of the product movement trajectory 520 is generated from the image. Next, a marketing behavior analysis 540 is performed using the product movement trajectory 520 and the results of processing by the behavior model 530 based on the vector representation of each part of the body.
Specifically, anonymized behavior analysis is performed by installing a plurality of cameras in a retail store and acquiring a three-dimensional wire frame of the human body from the cameras. Furthermore, by combining with the movement trajectory of the product, the trajectory of the action of picking up the product from the shelf and immediately returning it to the shelf. and provide the trajectory of the group up to the checkout at the register as marketing data.

FIG. 6 is a flow chart showing an example of processing by the first embodiment (image capturing device 150, camera 360).
In step S602, the inside of the store is photographed. A person (customer) is to be photographed.
In step S604, a wire frame of the human body is generated from the captured image.
In step S606, a movement trajectory for each part of the body is generated. For example, the human part movement trajectory table 700 is generated.
In step S<b>608 , data of the movement trajectory is transmitted to the information processing apparatus 100 .

FIG. 7 is an explanatory diagram showing an example data structure of the human part movement trajectory table 700. As shown in FIG. The human part movement trajectory table 700 has a person ID column 705 , a part ID column 710 , an in-store coordinate x column 715 , an in-store coordinate y column 720 , an in-store coordinate z column 725 , and a time column 730 . The person ID column 705 stores information for uniquely identifying a person (person ID: abbreviation for IDentification) in this embodiment. It should be noted that "uniquely identify" here only needs to indicate that it is the same person in the store, and it is not necessary to recognize the person's name (rather, it does not become personal data). . Whether or not they are the same person can be determined by determining that a person who is within a predetermined distance (for example, within 30 cm) at the next time (for example, 1 second later) is the same person. In the present embodiment, the part ID column 710 stores information (part ID) for uniquely identifying the part of the person. The in-store coordinate x column 715 stores the in-store coordinate x of the part. The in-store coordinate y column 720 stores the in-store coordinate y of the part. The in-store coordinate z column 725 stores the in-store coordinate z of the part. The time column 730 stores the time when the part was photographed. A movement trajectory (vector data) can be generated by tracing the coordinates of the same part of the same person in chronological order. Each row in the human part movement trajectory table 700 is the result of processing images captured at predetermined intervals. Note that the in-store coordinates are obtained by previously recording the photographing directions and positions of the plurality of cameras 160, converting them into the in-store coordinates from the plurality of cameras 160, and then recording them.

FIG. 8 is a flow chart showing an example of processing by the first embodiment (imaging device 150, camera 360).
In step S802, the inside of the store (particularly, product shelves) is photographed. The product will be photographed.
In step S804, the product is identified from the photographed image. For example, a bar code or QR code (registered trademark) printed on a product may be identified, or a product name or the like printed on the product may be recognized.
In step S806, a movement trajectory of the product is generated. For example, the commodity movement trajectory table 900 is generated.
In step S808, data of the movement locus of the product is transmitted.

FIG. 9 is an explanatory diagram showing an example data structure of the product movement trajectory table 900. As shown in FIG. The product movement trajectory table 900 has a product ID column 905 , an in-store coordinate x column 910 , an in-store coordinate y column 915 , an in-store coordinate z column 920 , and a time column 925 . The product ID column 905 stores information (product ID) for uniquely identifying a product in this embodiment. The in-store coordinate x column 910 stores the in-store coordinate x of the product. The in-store coordinate y column 915 stores the in-store coordinate y of the product. The in-store coordinate z column 920 stores the in-store coordinate z of the product. The time column 925 stores the time when the product was photographed. A movement trajectory (vector data) can be generated by tracing the coordinates of the same product in time series. Each row in the product movement trajectory table 900 is the result of processing images captured at predetermined intervals. Note that the in-store coordinates are obtained by previously recording the photographing directions and positions of the plurality of cameras 160, converting them into the in-store coordinates from the plurality of cameras 160, and then recording them.

FIG. 10 is a flow chart showing an example of processing according to the first embodiment (information processing device 100, information processing device 200).
In step S1002, the human part movement trajectory table 700 and the product movement trajectory table 900 are extracted.
In step S1004, the human part movement trajectory table 700 and the product movement trajectory table 900 are collated to generate combinations of people and products. Specifically, the position (in-store coordinates x, y, z) of the human part (especially the hand part) and the position of the product (in-store coordinates x, y, z) match (some deviation may be allowed), it is recognized that the person has picked up the product.

In step S1006, the movement trajectory of the person is compared with a predetermined pattern. As mentioned above, the patterns are ``return the product picked up once from the shelf to the shelf immediately'', ``repeatedly take two products from the shelf alternately and put one product in the basket'', and `` Compare the labels among the same products, decide on one product, and put it in the basket. In other words, behaviors to be analyzed are registered as patterns.
In step S1008, a group is estimated using the human movement trajectory. A specific example will be described later with reference to FIG.
In step S1010, the movement trajectories of multiple people in the group are compared with a predetermined pattern. As mentioned above, the patterns are as follows: ``Three people pick up two products one after another and put one product in the basket''; return to”, etc. In other words, the group behavior to be analyzed is registered as a pattern.
In step S1012, the person in charge is notified of the analysis result. A specific example will be described later with reference to FIG.

FIG. 11 is an explanatory diagram illustrating an example of processing for estimating groups.
Using results from the person location tracking module 122, a group estimation module 124 determines groups of people.
The group estimation module 124 determines that one person is within a predetermined distance (e.g., 3 meters, etc.) from another person for a predetermined time (e.g., 10 minutes or more, etc.). If so, they are assumed to be in the same group.
For example, as shown in FIG. 11, assume that there are three customers (person 1101, person 1102, and person 1103) in a store whose positions are indicated by an x-axis 1110 and a y-axis 1120. FIG.
These three customers are at the positions shown in FIG. 11(a) at a certain time. In other words, a person 1102 is present within a range 1130 (a range indicating a predetermined distance) centered on the person 1101 and a person 1103 is present outside the range 1130 . At the next time, the three customers are at the positions shown in FIG. 11(b). In other words, there is still a person 1102 within a range 1130 centering on the person 1101 and a person 1103 outside the range 1130 . Furthermore, at the next time, the three customers are at the positions shown in FIG. 11(c). In other words, there is no one within the range 1130 centered on the person 1101, and the people 1102 and 1103 are outside the range 1130. FIG. The person 1101 and the person 1102 are separated by a predetermined distance or more. In this case, if the time interval between FIGS. 11(a) and 11(b) is within a predetermined time, it is estimated that the persons 1101 and 1102 belong to the same group.

FIG. 12 is an explanatory diagram showing an example of a notification user interface.
The notification module 130 displays an in-store map display area 1210 and an analysis result display area 1220 on the screen 1200 .
The in-store map display area 1210 displays the inside of the store. It may be displayed in real time, or an action selected in the analysis result display area 1220 may be displayed. Customers are represented by points instead of images. That is, it is not personal data (because the information processing apparatus 100 does not acquire an image of the customer in the first place), and is displayed in such a way that the individual cannot be identified.
The analysis result display area 1220 displays the analysis result. For example, (1) the number of customers (customers) and the number of groups as the current situation, (2) the total number of customers as today's statistics (analysis results as an individual), the behavior "I picked up the product from the shelf once The number of actions "return to the shelf immediately", the number of actions "take two products alternately from the shelf and put one back in the basket", the number of actions "check the label and decide from among the same products", (3 ) Today's statistics (analysis results as a group) include the total number of groups, the number of behaviors "n people pick up m products and decide on one product", the behavior "1 person picked up the product "returns to another person", (4) stats for the week, (5) stats for the month, etc. Thus, real-time customer counts, group counts, defined marketing actions, etc. may be displayed with statistics on a daily, weekly, monthly, etc. basis.

<Second Embodiment>
FIG. 13 is a conceptual module configuration diagram of a configuration example of the second embodiment.
The second embodiment is obtained by adding a face recognition module 1380 and a face trajectory DB 1315 to the first embodiment.
The same parts as those in the first embodiment are denoted by the same reference numerals, and redundant explanations are omitted.
In the first embodiment, as described above, the same person is determined to be the same person if the person is within a predetermined distance at the next time. When people are nearby, it becomes difficult to determine that they are the same person. Therefore, face recognition is performed and used to determine that the person is the same person.

The photographing device 1350 has a camera group 155 , a wire frame module 165 , a product identification module 170 and a face recognition module 1380 and is connected to the information processing device 1300 . The photographing device 1350 performs processing equivalent to that of the photographing device 150 and then performs processing by the face recognition module 1380 .
The face recognition module 1380 recognizes a person's face from an image captured by the camera 160 (image of a person). However, the face recognition here is not a process for specifying an individual, and may be performed to the extent that it is possible to recognize that the person is the same person in the store. No further recognition (eg recognition of the individual's name, etc.) is performed. That is, it is sufficient to be able to determine whether or not the person is the same person by comparing with the face image of the person at another time. For example, feature extraction (hair color, length, relative positional relationship of eyes and nose, etc.) may be performed on the face image, and distance calculation (so-called similarity judgment) on the feature space may be performed, or machine learning may be performed. You may make it judge by the algorithm obtained by . The output data from the face recognition module 1380 only needs to guarantee that the person ID at a certain time is the same as the person ID at another time. For example, the face recognition module 1380 generates a face movement trajectory table 1500, which will be described later. Then, the recognition result by the face recognition module 1380 (which may include data such as a face image to be recognized and an intermediate result accompanying the recognition process) is sent when the recognition result is transmitted, or when the person leaves the store. Delete if you do.
Specifically, the face recognition module 1380 uses an existing method (for example, the technology described in Non-Patent Document 4) to identify the face in the image, and the three-dimensional position of the face in the store. Get (coordinates).
2, the wireframe module 165, product identification module 170, and face recognition module 1380 may be provided in the information processing device 1300 instead of the imaging device 1350. FIG. However, it is desirable to install only the face recognition module 1380 in the imaging device 1350 . This is to prevent personal data from being handled by the information processing apparatus 1300 . Also, as shown in the example of FIG. 3, the wireframe module 165, product identification module 170, and face recognition module 1380 may be built into each camera 160. FIG.

The information processing device 1300 has a reception module 105 , a human body part movement trajectory DB 110 , a product movement trajectory DB 115 , a face trajectory DB 1315 , a behavior analysis module 120 and a notification module 130 , and is connected to an imaging device 1350 .
The reception module 105 is connected to the human body part movement trajectory DB 110 , the product movement trajectory DB 115 , and the face trajectory DB 1315 . The reception module 105 stores the face movement trajectory data from the face recognition module 1380 in the face trajectory DB 1315 .
The facial trajectory DB 1315 is connected to the reception module 105 and the behavior analysis module 120 . The face trajectory DB 1315 stores face movement trajectory data.
The behavior analysis module 120 includes a person location tracking module 122, a group estimation module 124, a single marketing behavior detection module 126, a group marketing behavior detection module 128, a human body part movement trajectory DB 110, a product movement trajectory DB 115, and a face trajectory. It is connected with DB1315 and the action analysis module 120. FIG.
The human position tracking module 122 uses the human movement trajectory data and the face movement trajectory data (recognition result by the face recognition module 1380) to generate movement trajectory data for the same person. In other words, for example, if the shooting date and time, and the position of the person and the position of the face at the shooting date and time overlap, the person and the face correspond to each other, and whether or not the person at the other time is the same person. is determined using face movement trajectory data.
Note that the face movement trajectory data is used only for generating human movement trajectory data by the human position tracking module 122, and is not used for other processes. Then, the face movement trajectory data is deleted when the processing by the person position tracking module 122 is completed or when the person leaves the store.

FIG. 14 is a flow chart showing an example of processing by the second embodiment (imaging device 1350).
In step S1402, the inside of the store is photographed. A person (customer) is to be photographed.
In step S1404, the face is recognized from the captured image.
In step S1406, a movement trajectory of the face is generated. For example, a face movement trajectory table 1500 is generated.
In step S1408, data is transmitted.
In step S1410, the data is deleted.

FIG. 15 is an explanatory diagram showing an example data structure of the face movement trajectory table 1500. As shown in FIG. The face movement trajectory table 1500 has a person ID column 1505 , a store coordinate x column 1510 , a store coordinate y column 1515 , a store coordinate z column 1520 and a time column 1525 . The person ID column 1505 stores person IDs. The in-store coordinate x column 1510 stores the in-store coordinate x of the face. The in-store coordinate y column 1515 stores the in-store coordinate y of the face. The in-store coordinate z column 1520 stores the in-store coordinate z of the face. The time column 1525 stores the time when the face was photographed. A movement trajectory (vector data) can be generated by tracing the coordinates of the same face of the same person in chronological order. Each row in the face movement trajectory table 1500 is the result of processing images captured at predetermined intervals. Note that the in-store coordinates are obtained by previously recording the photographing directions and positions of the plurality of cameras 160, converting them into the in-store coordinates from the plurality of cameras 160, and then recording them. Note that the face is an example of a part of the human body.

FIG. 16 is a flow chart showing an example of processing by the second embodiment (information processing apparatus 1300).
In step S1602, the human part movement trajectory table 700, the face movement trajectory table 1500, and the product movement trajectory table 900 are extracted.
In step S1604, the human part movement trajectory table 700 and the face movement trajectory table 1500 are collated to generate the movement trajectory of the same person. Specifically, the position (in-store coordinates x, y, z) of the human part (particularly the head) and the position of the face (in-store coordinates x , y, z) match (some deviation may be allowed), the person is recognized as the same person, and the same person is specified using the person ID resulting from face recognition at other times as well.

In step S1606, the unnecessary face movement trajectory table 1500 is deleted.
In step S1608, the movement trajectory generated in step S1604 is collated with the product movement trajectory table 900 to generate combinations of people and products. Specifically, the position (in-store coordinates x, y, z) of the human part (especially the hand part) and the position of the product (in-store coordinates x, y, z) match (some deviation may be allowed), it is recognized that the person has picked up the product.
In step S1610, the movement trajectory of the person is compared with a predetermined pattern. This process is equivalent to step S1006 in the flowchart shown in the example of FIG.

In step S1612, the group is estimated using the movement trajectory of the person. This process is equivalent to step S1008 in the flowchart shown in the example of FIG.
In step S1614, the movement trajectories of multiple people in the group are compared with a predetermined pattern. This process is equivalent to step S1010 in the flowchart shown in the example of FIG.
In step S1616, the person in charge is notified of the analysis result. This process is equivalent to step S1012 in the flowchart shown in the example of FIG.

In the technology described in Non-Patent Document 1, many cameras are installed in retail stores to analyze consumer behavior, but in the second embodiment, anonymization is performed by wire frame analysis, In particular, by physically separating the facial trajectory DB 1315 (which may include the human position tracking module 122) from the information processing device 1300, marketing analysis with more complete anonymization becomes possible.

Non-Patent Literature 2 shows a standard for how much personal information is used when camera images are used. Then, a method of anonymizing the captured images/videos by performing conventional image processing such as mosaic processing is disclosed. Of course, Non-Patent Document 2 does not include a method of anonymizing only human body movements by wireframe analysis.

Non-Patent Literature 5 explains a social movement to achieve both privacy protection and usefulness of data. There is a debate about how to anonymize faces. Of course, Non-Patent Document 5 also does not include a method of anonymizing only human body movements by wireframe analysis.

FIG. 17 shows the hardware configuration of a computer that executes a program as the present embodiment (information processing device 100, imaging device 150, information processing device 200, camera 360, information processing device 1300, imaging device 1350). As exemplified, it is a general computer, specifically a personal computer, a built-in computer such as a camera, a computer that can serve as a server, and the like. That is, as a specific example, a CPU 1701 is used as a processing unit (calculating unit), and a RAM 1702, ROM 1703, and HDD 1704 are used as storage devices. As the HDD 1704, for example, an HDD (abbreviation for Hard Disk Drive), an SSD (abbreviation for Solid State Drive) which is a flash memory, or the like may be used. reception module 105, behavior analysis module 120, person location tracking module 122, group estimation module 124, single marketing behavior detection module 126, group marketing behavior detection module 128, notification module 130, wireframe module 165, product identification module 170, wire A CPU 1701 that executes programs such as the frame module 265, the product identification module 270, and the face recognition module 1380; a RAM 1702 that stores the programs and data; HDD 1704, which is an auxiliary storage device having functions as part movement trajectory DB 110, product movement trajectory DB 115, face trajectory DB 1315, etc., and user's access to keyboard, mouse, touch screen, microphone, camera (including line-of-sight detection camera, etc.), etc. A reception device 1706 that receives data based on operations (including motion, voice, line of sight, etc.), an output device 1705 such as a CRT, liquid crystal display, speaker, etc., and a communication line interface such as a network interface card for connecting to a communication network. 1707 and a bus 1708 for connecting them to exchange data. A plurality of these computers may be connected to each other via a network.

Of the above-described embodiments, with respect to the computer program, a computer program, which is software, is loaded into the system of this hardware configuration, and the software and hardware resources cooperate to perform the above-described embodiment. is realized.
Note that the hardware configuration shown in FIG. 17 shows one configuration example, and the present embodiment is not limited to the configuration shown in FIG. I wish I had. For example, some modules may be configured with dedicated hardware (for example, an application specific integrated circuit (ASIC), a reconfigurable integrated circuit (field-programmable gate array: FPGA), etc.). A part of the modules may be in an external system and connected by a communication line, or a plurality of systems shown in FIG. 17 may be connected by a communication line and cooperate with each other. Moreover, in addition to personal computers, it may be incorporated in portable information communication devices (including mobile phones, smartphones, mobile devices, wearable computers, etc.), home information appliances, robots, and the like.

In the above-described embodiment, the case of application to a retail store was exemplified, but the behavior of people in other facilities may be analyzed. As described above, for example, multiple cameras are installed in a sports facility in a similar manner, and specific patterns (dangerous behavior patterns, etc.) are defined in advance in the single marketing behavior detection module 126 and the group marketing behavior detection module 128. Prediction of safe spectators and dangerous spectators (for example, hooligans, etc.) and anomaly detection are possible by analyzing them.

The program described above may be stored in a recording medium and provided, or the program may be provided by communication means. In that case, for example, the program described above may be regarded as an invention of "a computer-readable recording medium on which the program is recorded."
"Program-recorded computer-readable recording medium" refers to a computer-readable recording medium recording a program, which is used for program installation, execution, program distribution, and the like.
As a recording medium, for example, a digital versatile disc (DVD), which is a standard established by the DVD Forum, "DVD-R, DVD-RW, DVD-RAM, etc." Standard "DVD + R, DVD + RW, etc.", compact disc (CD), read-only memory (CD-ROM), CD recordable (CD-R), CD rewritable (CD-RW), Blu-ray disc ( Blu-ray (registered trademark) Disc), magneto-optical disk (MO), flexible disk (FD), magnetic tape, hard disk, read-only memory (ROM), electrically erasable and rewritable read-only memory (EEPROM (registered trademark) )), flash memory, random access memory (RAM), SD (abbreviation for Secure Digital) memory card, and the like.
Then, the whole or part of the program may be recorded on the recording medium and stored or distributed. Also, by communication, for example, a wired network used for a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), the Internet, an intranet, an extranet, etc., or a wireless communication. It may be transmitted using a transmission medium such as a network or a combination thereof, or may be carried on a carrier wave.
Furthermore, the program may be part or all of another program, or may be recorded on a recording medium together with a separate program. Moreover, it may be divided and recorded on a plurality of recording media. Also, it may be recorded in any form, such as compression or encryption, as long as it can be restored.

DESCRIPTION OF SYMBOLS 100... Information processing apparatus 105... Reception module 110... Human body part movement locus DB
115 Product movement trajectory DB
120 Behavior analysis module 122 Person location tracking module 124 Group estimation module 126 Single marketing behavior detection module 128 Group marketing behavior detection module 130 Notification module 150 Imager 155 Camera group 160 Camera 165 Wireframe Module 170 Product identification module 200 Information processing device 265 Wire frame module 270 Product identification module 350 Imaging module 355 Camera group 360 Camera 350 Imaging module 1300 Information processing device 1315 Face trajectory DB
1350... Imaging device 1380... Face recognition module

Claims (4)

a wireframe conversion means for anonymizing a person by converting an image of the person in the store into wireframe data representing a vector representation of the part of the person;
a generation means for generating human movement trajectory data representing movements of parts of the person using the wire frame data;
a second generation means for generating product movement trajectory data expressing the movement trajectory of the product extracted from the photographed image of the product;
analysis means for determining whether or not the human movement trajectory data corresponds to predetermined behavior pattern data, and using the determination result as an analysis result of the behavior of the person;
a notification means for notifying the analysis result by the analysis means ;
the behavior pattern data is pattern data indicating behaviors of a plurality of people belonging to the group with respect to the product;
The analysis means estimates a person belonging to a group, determines whether the person movement trajectory data and the product movement trajectory data of a plurality of people in the group correspond to the pattern data,
When estimating a person belonging to a group, the analysis means (1) calculates the distance between people, and if the mouths of people within a predetermined distance are open alternately, (2) assume that people are a group if the goods are passed from one person to the other;
An information processing device characterized by: further comprising recognition means for recognizing a person's face from an image of the person,
The generating means generates human movement trajectory data for the same person using the recognition result by the recognizing means.
The information processing device according to claim 1 . the result of recognition by the recognition means is used only for the generation of human movement trajectory data by the generation means,
deleting the recognition result by the recognition means when the person leaves the store;
The information processing apparatus according to claim 2 . the computer,
a wireframe conversion means for anonymizing a person by converting an image of the person in the store into wireframe data representing a vector representation of the part of the person;
a generation means for generating human movement trajectory data representing movements of parts of the person using the wire frame data;
a second generation means for generating product movement trajectory data expressing the movement trajectory of the product extracted from the photographed image of the product;
analysis means for determining whether or not the human movement trajectory data corresponds to predetermined behavior pattern data, and using the determination result as an analysis result of the behavior of the person;
Functioning as notification means for notifying analysis results by the analysis means ,
the behavior pattern data is pattern data indicating behaviors of a plurality of people belonging to the group with respect to the product;
The analysis means estimates a person belonging to a group, determines whether the person movement trajectory data and the product movement trajectory data of a plurality of people in the group correspond to the pattern data,
When estimating a person belonging to a group, the analysis means (1) calculates the distance between people, and if the mouths of people within a predetermined distance are open alternately, (2) assume that people are a group if the goods are passed from one person to the other;
An information processing program characterized by:

Images