JP2019133530A - Information processing apparatus - Google Patents

Abstract

To determine whether a person moving in a space and other persons are related to each other or not.SOLUTION: A storage unit 12 stores feature quantities of trajectories of multiple persons when the multiple persons who are related to each other co-exist in a specific space. A detection unit 111 detects the multiple persons co-existing in the specific space repeatedly. A calculation unit 112 calculates feature quantities of trajectories of the persons detected by the detection unit 11. A determination unit 113 determines whether the persons detected by the detection unit 111 are related to each other or not by use of the feature quantities calculated by the calculation unit 112 and the contents stored in the storage unit 12.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an information processing apparatus.

  As a document disclosing a technique for associating information related to purchase and information related to a flow of a customer in a store, there is Patent Document 1. The information processing apparatus described in Patent Literature 1 includes, from an image taken in a store, a first association unit that associates information about a customer who enters a store at an entrance and information about a customer's flow line in the store; A second correlating means for associating information relating to a customer's flow line from the payment area in the store to the entrance / exit and information relating to purchase in the store; information relating to the customer who entered the store / exit at the entrance / exit of the store; Using the information associated by the third association means for associating information, the first association means, the second association means, and the third association means, the information relating to the purchase and the in-store A fourth correlating means for correlating information on the flow line of the customer at

Japanese Patent Laid-Open No. 2006-162256

  An object of the present invention is to provide a technique for determining whether or not a person moving in a space and other persons in the space have a mutual relationship.

  The information processing apparatus according to claim 1, wherein a plurality of mutually related persons coexist in a specific space, a storage unit that stores feature amounts of the trajectories of the plurality of persons, and a coexistence in the specific space A detection unit that repeatedly detects a plurality of people, a calculation unit that calculates feature amounts of the trajectories of the plurality of people detected by the detection unit, the feature amounts calculated by the calculation unit, and the storage unit A discriminating unit that discriminates whether or not a plurality of persons detected by the detecting unit have a relationship with each other using the stored content;

  The information processing device according to claim 2 is the information processing device according to claim 1, wherein the storage unit stores the feature amount for each type of the relationship, and the determination unit is calculated by the calculation unit. The type of the relationship corresponding to the feature amount is determined using the storage content of the storage unit.

  An information processing apparatus according to a third aspect is the information processing apparatus according to the first or second aspect, wherein the feature amount represents a change over time in the distances of the plurality of persons.

  An information processing apparatus according to a fourth aspect of the present invention is the information processing apparatus according to the first or second aspect, wherein the feature amount represents a change over time in a difference between velocity vectors of the plurality of persons.

  An information processing apparatus according to a fifth aspect is the information processing apparatus according to the first or second aspect, wherein the feature amount represents a residence time of the plurality of persons for each region into which the space is divided.

  The information processing device according to claim 6 is the information processing device according to any one of claims 3 to 5, wherein the determination unit is multiplied by a weight determined for each region into which the space is divided. Discrimination is performed using the feature amount.

  An information processing apparatus according to a seventh aspect is the information processing apparatus according to the first or second aspect, wherein the feature amount represents a change in position or speed of the center of gravity of the plurality of persons over time.

  An information processing apparatus according to an eighth aspect of the present invention is the information processing apparatus according to the second aspect, further comprising an output unit that outputs information corresponding to the type of the relationship determined by the determination unit.

  The information processing device according to claim 9 is the information processing device according to claim 8, wherein the output unit includes a plurality of persons detected by the detection unit among a plurality of display devices arranged in the space. The information is output to a display device whose distance matches a predetermined condition.

  The information processing device according to claim 10 is the information processing device according to claim 8 or 9, wherein the output unit is configured such that the movement speeds of the plurality of persons detected by the detection unit are equal to or higher than a threshold value. The information is not output.

  The information processing device according to claim 11 is the information processing device according to any one of claims 8 to 10, wherein the output unit corresponds to the moving speeds of the plurality of persons detected by the detection unit. To change the form of the information.

  The information processing device according to claim 12 is the information processing device according to claim 8, wherein the output unit sends the information to a mobile terminal possessed by any of the plurality of persons detected by the detection unit. Output.

  An information processing apparatus according to claim 13 is the information processing apparatus according to claim 8, further comprising a reception unit that receives an operation, wherein the determination unit causes the output unit to output the determined type. When an operation for correcting the type is received by the receiving unit, the corrected type and the feature amount calculated by the calculating unit are associated with each other and stored in the storage unit.

  The information processing device according to claim 14 is the information processing device according to claim 1, wherein the determination unit is configured such that distances at specific locations of a plurality of persons detected by the detection unit are predetermined. If they match, the determined type is corrected, and the corrected type and the feature amount calculated by the calculation unit are associated with each other and stored in the storage unit.

According to the first aspect of the present invention, it is determined whether or not the person moving in the space and the other person have a mutual relationship.
According to the second aspect of the present invention, the type of the relationship between the person moving in the space and the other person is determined.
According to the third aspect of the present invention, the type of relationship is determined from the change over time of the distances of a plurality of persons.
According to the fourth aspect of the present invention, the type of the relationship is determined from the change with time of the difference between the speed vectors of a plurality of persons.
According to the fifth aspect of the present invention, the type of relationship is determined from the residence time of a plurality of persons for each area into which the space is divided.
According to the sixth aspect of the present invention, the weight for each region is added to the determination of the relationship type.
According to the seventh aspect of the present invention, the type of relationship is determined from the temporal change in the position or speed of the center of gravity of a plurality of persons.
According to invention of Claim 8, the information according to the classification of relationship is shown.
According to the ninth aspect of the present invention, information is more easily transmitted to a plurality of people than when information is output regardless of the distance between the display device and the plurality of people.
According to the invention described in claim 10, the load on the information processing apparatus is reduced.
According to the eleventh aspect of the present invention, information is more easily transmitted to a plurality of people than when the information form is not changed according to the moving speed.
According to the twelfth aspect of the present invention, information is presented without arranging a device for presenting information in the space.
According to the invention described in claim 13, when an operation for correcting the determined type is performed, the correction content is reflected in the subsequent determination.
According to the fourteenth aspect of the present invention, the correction content of the type based on the distance at a specific location of a plurality of people is reflected in subsequent determination.

The top view of the shop A. 1 is a diagram illustrating a configuration of an information processing device 10. The figure which shows the feature-value database 121. FIG. The figure which shows the presentation information database. The figure which shows the display apparatus table. The flowchart which shows the whole process which the control part 11 performs. The flowchart of the detection process which the detection part 111 performs. The figure which shows the detection result of a person. The flowchart of a 1st calculation process. The figure which shows the movement log | history R of the person area | region H. FIG. The figure which shows the created locus | trajectory K. FIG. The flowchart of a 2nd calculation process. The figure which shows the example of the integrated locus | trajectory map. The figure which shows the 1st feature-value. The figure which shows the 2nd feature-value. The figure which shows a 3rd feature-value. The figure which shows the 4th feature-value. The figure which shows the 5th feature-value. A scatter diagram showing the classification of feature vectors.

  An example for carrying out the present invention will be described. In the present embodiment, the type of mutual relationship among a plurality of customers is determined using the trajectories in the store of a plurality of customers (an example of a plurality of people) who have visited the store (an example of a specific space). It has the composition to do.

  FIG. 1 is a plan view of the store A. FIG. The store A includes a checkout table B, display shelves C (1) to C (3) (hereinafter collectively referred to as display shelves C), and an entrance D. On the checkout table B, a cash register is arranged. In the store A, the imaging devices 20 (1) to 20 (4) (hereinafter collectively referred to as the imaging device 20), the display devices 30 (1) to 30 (6) (hereinafter referred to as the display device 30). Are collectively called). In addition, the number, arrangement | positioning, etc. of the display shelf C shown in FIG. 1, the imaging | photography device 20, and the display apparatus 30 are illustrations, Comprising: They may differ from FIG.

  FIG. 2 is a diagram illustrating a configuration of the information processing apparatus 10. The information processing apparatus 10 may be disposed in the store A or may be disposed outside the store A. The information processing apparatus 10 includes a control unit 11, a storage unit 12, a communication unit 13, a display unit 14, and a reception unit 15. The control unit 11 includes an arithmetic device such as a CPU (Central Processing Unit) and a storage device such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The ROM stores firmware that describes the procedure for starting up hardware and an OS (Operating System). The RAM is used for storing data when the CPU executes an operation.

  The storage unit 12 includes, for example, a memory and a hard disk storage device, and stores software such as an OS and an application, a feature amount database 121, a presentation information database 122, and a display device table 123. The communication unit 13 includes an I / F (Interface) for performing communication with other electronic devices. The imaging device 20 and the display device 30 are connected to the communication unit 13.

  The display unit 14 includes a display device such as a liquid crystal display panel. The control unit 11 causes the display unit 14 to display a GUI (graphical user interface) screen on which widgets for operating the information processing apparatus 10 are arranged. The accepting unit 15 includes, for example, a keyboard, a pointing device, and the like, and accepts an operation on the GUI. The pointing device is, for example, a touch panel, a mouse, a touch pad, or the like that covers the display surface of the display unit 14. The display unit 14 and the reception unit 15 may be configured as devices separated from the information processing apparatus 10, and the display unit 14 and the reception unit 15 may be connected to the communication unit 13.

  The photographing device 20 is, for example, a video camera, photographs a subject, and outputs image data representing an image of the subject to the information processing device 10. At least one photographing device 20 is arranged on the ceiling, wall, etc. of the store A. The imaging device 20 is preferably an omnidirectional camera. It is desirable that a plurality of photographing devices 20 be distributed and arranged on the ceiling or wall in the store A so that no blind spots are generated in the store A. Further, in order to increase the accuracy of the processing described later, it is desirable that each visitor is photographed by a plurality of photographing devices 20. For this purpose, it is desirable that the plurality of photographing devices 20 be arranged so that the photographing ranges of the plurality of photographing devices 20 overlap. Depending on the size of the store A, the shape of the space in the store A, and the height of the ceiling, there may be a case where one image capturing device 20 can capture images without causing blind spots, so only one image capturing device 20 is provided in the store A. There may be cases.

  The display device 30 displays an image representing information output from the control unit 11. At least one display device 30 is arranged in the store A. It is desirable that the plurality of display devices 30 be arranged so that at least one display device 30 always enters the field of view of a visitor in the store A. The display device 30 is disposed on each of the surfaces indicated by the long sides of the rectangle indicating the display shelf C, for example. The display device 30 may be disposed on a wall in the store, the checkout table B, or the like. The display device 30 may be arranged on the back side of the cash register (the side opposite to the operation unit operated by the employee). It is desirable that the display device 30 be arranged at a height that can be accommodated by the child's field of view. The display device 30 may be arranged at a height for adults and a height for children. The display device 30 is desirably a thin type including an organic EL (electroluminescence) panel, a liquid crystal panel, and the like.

  FIG. 3 is a diagram showing the feature amount database 121. The feature amount database 121 stores, for each relationship type, feature amounts of trajectories of a plurality of people when a plurality of people having a relationship with each other coexist in a specific space. Each record of the feature quantity database 121 has a field of type ID, type name, and feature quantity. The type ID is an identifier for identifying the type of mutual relationship among a plurality of people. The type name is the name of the type of relationship. In this example, “parent and child (infant)”, “friend”, and “lover” are set as the relationship type. The feature amount will be described later.

  FIG. 4 is a diagram showing the presentation information database 122. The presentation information database 122 stores information to be presented to the plurality of people for each type of mutual relationship among the plurality of people. Each record of the presentation information database 122 has fields of type ID, product field ID, and presentation information. The type ID is the same as the type ID in the feature amount database 121. The product field ID is an identifier for identifying the product field. In this embodiment, in order to facilitate understanding, for example, for each display shelf, the display shelf C (1) is fresh food, the display shelf C (2) is confectionery, and the display shelf C (3) is daily necessities. Assume that the fields of products displayed are different. In this example, the product field ID = 1 indicates fresh food, the product field ID = 2 indicates confectionery, and the product field ID = 2 indicates daily necessities.

  The presentation information is information presented to the store visitor. In the field of presentation information, binary data of a file indicating the presentation information may be stored, and a path of a file storage location may be stored. The content of the presentation information includes, for example, information indicating a popular product extracted from sales results, information indicating a new product, information indicating a product that the seller or manufacturer intends to increase the sales amount, information indicating a recipe for cooking, etc. It is.

  FIG. 5 is a diagram showing the display device table 123. Each record in the display device table 123 has fields for a display device ID, a product field ID, and position information. The display device ID is an identifier for identifying the display device 30. The product field ID is the same as the product field ID in the presentation information database 122. The position information is the coordinates of the position where the display device 30 is arranged.

  FIG. 6 is a flowchart showing the entire processing executed by the control unit 11. A program describing the procedure of these processes is stored in the storage unit 12, and the control unit 11 executes the process according to this program. The control unit 11 functions as a detection unit 111, a calculation unit 112, a determination unit 113, and an output unit 114 (see FIG. 2). In step S01, the detection unit 111 repeatedly detects a plurality of people coexisting in a specific space (detection process). In step S02, the calculation unit 112 calculates feature amounts of a plurality of human tracks detected by the detection unit 111 (calculation processing). In step S <b> 03, the determination unit 113 determines the type of relationship corresponding to the feature amount calculated by the calculation unit 112 using the stored content of the storage unit 12 (determination process). In step S04, the output unit 114 outputs information according to the type of relationship determined by the determination unit 113 (output processing). The detailed contents of the above steps are as follows.

  FIG. 7 is a flowchart of the detection process (step S01 in FIG. 6) executed by the detection unit 111. The detection unit 111 repeatedly executes detection processing at predetermined time intervals. In order to detect a person who moves on foot, the time interval is set to 1 second, for example, but may be set to an interval shorter than 1 second or an interval longer than 1 second. In step S <b> 101, the detection unit 111 detects an object from a frame included in the image data output from the imaging device 20. In step S102, the detection unit 111 detects a region (human region H) indicating the shape of a person from the detected object. In step S <b> 103, the detection unit 111 obtains the coordinates of the detected human region H, and stores the information indicating the shooting time of the frame or the order of the shooting times and the coordinates in the storage unit 12 in association with each other. The coordinates of the human area H are expressed using a common coordinate system with the position information of the display device 30 (see FIG. 5).

  FIG. 8 is a diagram illustrating a human detection result. The illustrated example is an example in which an omnidirectional camera is used as the photographing apparatus 20 and a circular photographing range E is photographed. A human region H is represented by a rectangle surrounding the person. In this example, three human regions H (1), H (2), and H (3) are detected.

  Human detection is performed by, for example, deep learning object extraction technology (for example, S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. ”, NIPS 2015. For example, W. Liu, D. Anhuelov, D. Erhan, C. Szegedy, S. Reed, and C. Fu, and A. Berg,“ SSD: Single Shot MultiBox Detector. ”, ECCV 2016 .). According to the deep learning object extraction technique, a person is detected even if the face of the person is not recognized.

Next, the calculation process (step S02 in FIG. 6) executed by the calculation unit 112 will be described. The calculation unit 112 calculates feature amounts of the trajectories of a plurality of people detected by the detection unit 111.
The calculation process includes a first calculation process and a second calculation process. The first calculation process is a process of calculating the trajectories of a plurality of people detected by the detection unit 111 for each imaging range E of the plurality of imaging devices 20. The second calculation process is a process of integrating the trajectory for each shooting range E calculated by the first calculation process and calculating the feature amount of the integrated trajectory.

  FIG. 9 is a flowchart of the first calculation process. In the following description, the subscript i represents the order of the photographing times of the frames output from the photographing device 20. The calculation unit 112 repeatedly executes the first calculation process in order of shooting time.

In step S201, the calculation unit 112 reads out the coordinates of all human regions H from the storage unit 12 for three consecutive frames (I _i−1 , I _i , I _{i + 1} ). Hereinafter, the frames I _i−1 , I _i , and I _{i + 1} are referred to as a first frame, a second frame, and a third frame, respectively.

  In step S202, the calculation unit 112 calculates the velocity vector of the central coordinates of the human area H for two sets of two consecutive frames (a set of the first frame and the second frame, and a set of the second frame and the third frame). Create Hereinafter, a set of the first frame and the second frame is referred to as a first set, and a set of the second frame and the third frame is referred to as a second set. For example, in the case of the first set, a set of velocity vectors starting from the centers of the plurality of human regions H included in the first frame and ending as the centers of the plurality of human regions H included in the second frame is created. In this embodiment, since no means for identifying a plurality of human areas H is provided, all combinations of a plurality of human areas H included in the first frame and a plurality of human areas H included in the second frame are created. . In this example, six sets of three velocity vectors are created. Similarly, six sets of three velocity vectors are created for the sets of the second frame and the third frame.

  In step S <b> 203, the calculation unit 112 selects a likely combination of the human region H using the created velocity vector. Specifically, the calculation unit 112 calculates the first set to the second set from all combinations of the six sets of velocity vectors created in the first set and the six sets of velocity vectors created in the second set. A combination is selected in which the amount of change in the velocity vector matches a predetermined condition. The predetermined condition is, for example, that the change amount of the velocity vector from the first group to the second group is the minimum. Note that a combination of the human regions H may be selected using the characteristics of the image in the human region H, for example, a color histogram.

In step S204, the calculation unit 112 corrects the combination of the human regions H so as to match the result of the first calculation process for the immediately preceding three frames. Specifically, the calculation unit 112 selects the human region H selected in the set of the frame I _i-1 and the frame I _i included in the immediately preceding three frames (I _i-2 , I _i-1 , I _i ). Human region H selected in the combination of frame I _i-1 and frame I _i included in three frames (hereinafter referred to as the first combination) and three frames (I _i-1 , I _i , I _{i + 1} ) Are compared with each other (hereinafter referred to as the second combination), and if the two are different, either the first combination or the second combination is corrected so that the two match. In this way, movement histories R of a plurality of human areas H over four frames (I _i-2 , I _i-1 , I _i , I _{i + 1} ) are created. The above process is executed for all three consecutive frames.

  FIG. 10 is a diagram showing the movement history R of the human area H created by the calculation unit 112. This figure is an example of the movement history R (3) of the human area H (3) shown in FIG. The movement histories R (1) and R (2) are created in the same procedure for the human areas H (1) and H (2). The calculation unit 112 creates an approximate curve that connects the center coordinates of the human region H in the movement history R created in this way in the order of the photographing times of the frames. A human trajectory K is represented by the created approximate curve.

  FIG. 11 is a diagram showing the created trajectory K. As shown in FIG. This figure is an example of the locus K (3) of the human region H (3). Trajectories K (1) and K (2) are created for human areas H (1) and H (2) in the same procedure. A locus map M is created by projecting the locus K onto a plan view of the store A. The creation of the movement history R and the trajectory map M is executed for each imaging range E of the plurality of imaging devices 20, and the created movement history R and the trajectory map M are associated with the identifier of the imaging device 20 in the storage unit 12. Remembered.

  As a method for creating the locus map M, the following two methods are conceivable. The first method is a method in which a mapping function F to a trajectory map M is learned in advance from the location of the human region H with an omnidirectional camera. For example, a technique such as multilayer perceptron is used. A configuration in which the input is two-dimensional (the upper left coordinates and the lower right coordinates of the human area H) +1 dimension (the camera position) and the output is two dimensions (coordinates of the trajectory map M) can be considered. The second method is a method of calculating the position in the human trajectory map M from the height h from the floor of the center of gravity of the human region H by projective transformation. For the value of h, a predetermined numerical value (for example, half the average height of a human being) is used.

  FIG. 12 is a flowchart of the second calculation process. The second calculation process is a process of integrating the trajectory K for each shooting range E calculated by the first calculation process and calculating the feature amount of the integrated trajectory. Here, an example in which the imaging ranges E of the plurality of imaging devices 20 do not overlap is shown. The following processing is executed by selecting two shooting ranges E adjacent to each other. Of the two shooting ranges E adjacent to each other, the shooting range E that is relatively close to the entrance D is referred to as a first shooting range, and the shooting range E that is relatively far from the entrance D is referred to as a second shooting range. First, the shooting range E (the shooting range E of the shooting device 20 (1)) closest to the entrance D is selected as the first shooting range, and the shooting range E of the shooting device 20 (2) is selected as the second shooting range. The

  In step S301, the calculation unit 112 reads a plurality of movement histories R and a plurality of trajectory maps M in the first imaging range from the storage unit 12, and selects a person from the read movement history R and the trajectory map M. A person ID for identification is given. For example, a person ID = 1 is assigned to the movement history R and the trajectory map M of the person area H (1), and a person ID = 2 is assigned to the movement history R and the trajectory map M of the person area H (2). Then, the person ID = 3 is given to the movement history R and the trajectory map M of the person area H (3). Let X be the set of all human regions H included in the plurality of movement histories R.

  In step S <b> 302, the calculation unit 112 reads a plurality of movement histories R and a plurality of trajectory maps M in the second imaging range from the storage unit 12. A set of all human regions H included in the plurality of movement histories R is assumed to be Y.

  In step S303, the calculation unit 112 integrates a plurality of trajectory maps M in the first imaging range and a plurality of trajectory maps M in the second imaging range. Specifically, the distance between the central coordinates of each of the plurality of human areas H (1) included in the set X and the central coordinates of all the human areas H included in the set Y is calculated, and the person with the shortest distance is calculated. The same person ID as the movement history R of the human area H (1) included in the set X is assigned to the movement history R and the trajectory map M including the area H. By assigning the person ID, the trajectory map M (1) in the first shooting range and the trajectory map M (1) in the second shooting range are associated one-to-one. By this association, the trajectory map M (1) in the first shooting range and the trajectory map M (1) in the second shooting range are integrated. Similar processing is executed for the human areas H (2) and H (3) included in the set X. By executing the same process as described above for all combinations of two shooting ranges E adjacent to each other, the trajectory maps M in all the shooting ranges E are integrated.

  FIG. 13 is a diagram illustrating an example of the integrated trajectory map M. This example shows a trajectory map M (1) of the human area H (1) and a trajectory map M (2) of the human area H (2). For example, the person area H (1) is a parent and the person area H (2) is a child (infant). Fresh food is displayed on the display shelf C (1), and confectionery is displayed on the display shelf C (2). Parents are trying to buy sweets for their children. Since the parent tries to reach the display shelf C (2) through the shortest possible route while showing interest in fresh food, the parent's trajectory map M (1) is recorded as shown. In the areas a0 and a1, the child has no interest in fresh food, so he walks away from his / her parents. When the child reaches the area a2, he approaches the parent to buy the confectionery, and is accompanied by the parent to the display shelf C (2). Choose a confectionery. As a result, a child trajectory map M (2) is recorded as shown.

  As described above, this trajectory represents a tendency when the mutual relationship between a plurality of people coexisting in the store A is a parent and child. On the other hand, when the mutual relationship of a plurality of people coexisting in the store A is a friend or a lover, it is considered that the time to walk with the store A tends to be longer than in the case of the parent and child. In addition, when comparing lovers with friends, lovers tend to walk at shorter distances.

  In step S304, the calculation unit 112 calculates the feature amount of the trajectory. Hereinafter, five examples (first to fifth feature amounts) of feature amounts will be described. Here, description will be made using an example of the trajectory shown by the trajectory map M (1) and the trajectory map M (2) shown in FIG.

(1) First Feature Amount FIG. 14 is a diagram showing a first feature amount. The first feature amount represents a change over time in the distance of a plurality of people. The distance between a plurality of people is represented by the length of a line segment L with the center coordinates P of the plurality of human regions H at each photographing time as endpoints. The first feature amount is represented by, for example, a feature vector whose components are distances arranged in order of shooting time. Alternatively, the first feature amount may be represented by a feature vector having a histogram frequency component with the distance range as the horizontal axis and the appearance frequency of the values included in each range as the vertical axis.

(2) Second Feature Amount FIG. 15 is a diagram showing a second feature amount. The second feature amount represents a change over time in the difference between the speed vectors of a plurality of persons. The speed vector of the person is represented by a vector V having the center coordinate P of the person area H at one shooting time as a start point and the end point of the center coordinate P of the person area H at the next shooting time (FIG. 15). (See (a)). The difference between the velocity vectors of a plurality of people is, for example, the difference V (1) −V (2) between the velocity vector V (1) of the human region H (1) and the velocity vector V (2) of the human region H (2). ) (See FIG. 15B). The second feature amount is represented by a feature vector having, for example, a difference component or magnitude of speed vectors arranged in order of shooting time. Alternatively, the second feature amount is represented by a feature vector having a frequency vector difference component or magnitude range as a horizontal axis and the frequency of appearance of a value included in each range as a vertical axis as a component. May be.

(3) Third Feature Amount FIG. 16 is a diagram showing a third feature amount. The third feature amount represents the residence time of a plurality of people for each region obtained by dividing the space into a plurality of regions. For example, an area where the space of the store A is divided into a plurality of areas is defined (see FIG. 16A), and for each of a plurality of persons, the residence time in each of the areas is obtained from the frame shooting time (FIG. 16B). (See (c)). The third feature amount is represented, for example, by a feature vector having each residence time of a histogram with the region as the horizontal axis and the residence time in each region as the vertical axis.

(4) Fourth Feature Amount FIG. 17 is a diagram illustrating a fourth feature amount. The fourth feature value is obtained by weighting the first feature value to the third feature value for each region. The illustrated example is an example in which weighting is performed on the first feature amount (change in the distance of a plurality of people over time). The first feature amount is represented by a line segment L having P (2) and P (1) as end points, whereas the fourth feature amount has P (2) and Q (1) as end points. It is represented by a line segment W. In this example, a weight of 1.5 is set in the areas a0 and a2, and a weight of 0.5 is set in the areas a1 and a3.

  The setting of this weight is based on the following assumptions. The illustrated trajectory map M represents a tendency when a plurality of people coexisting in the store A are parents and children as described above, but the distance between the parent and the child increases in the region a0, and the parent in the region a2 There is a feature that the distance between the child and the child decreases. It is conceivable that the above-described weighting emphasizes the change over time in the distance between the parent and child and improves the efficiency of the determination process described later.

(5) Fifth Feature Amount FIG. 18 is a diagram illustrating a fifth feature amount. The fifth feature amount represents a change in the position or speed of the center of gravity G of a plurality of people over time. The center of gravity G of a plurality of people is represented by the midpoint of a line segment L whose end point is the center coordinate P of the plurality of person regions H at each photographing time. For example, the fifth feature amount is represented by a feature vector whose component is the position or speed of the center of gravity G arranged in order of shooting time.

  In the feature amount database 121, any one of the first feature amount to the fifth feature amount is stored for each type of relationship. The stored feature quantity may be a feature quantity calculated from a past visitor's trajectory, or may be a feature quantity calculated from a trajectory assumed for each type of relationship. Note that two or more of the first to fifth feature values may be stored in the feature value database 121.

  The calculation process may be executed every time the detection process is performed once. However, since a significant feature amount cannot be obtained unless human detection results are accumulated to some extent, The calculation process may be executed after the detection process is executed a number of times (a plurality of times). Thereafter, the trajectory map M and the feature amount may be periodically updated by executing a calculation process every time a predetermined number (one or more) of detection processes are executed.

  Next, the determination process (step S03 in FIG. 6) executed by the determination unit 113 will be described. The determination unit 113 determines the type of relationship corresponding to the feature amount calculated by the calculation unit 112 using the storage content of the storage unit 12. Specifically, the determination unit 113 extracts a feature amount corresponding to the calculated feature amount from the feature amount database 121 using a pattern recognition model such as SVM (support vector machine), and converts the feature amount into the extracted feature amount. The associated type ID is specified. However, since the length of time from entering the store to leaving the store varies, the feature values (first feature value, second feature value, and fifth feature value) indicating changes over time are normalized in the time axis direction. The discrimination process is performed after unifying the dimensionality of the feature vector by the conversion.

  FIG. 19 is a scatter diagram showing the classification of feature vectors calculated from the trajectory map M. In this scatter diagram, components of multidimensional feature vectors are displayed in a two-dimensional manner for convenience. For example, by specifying the principal component axis of a feature vector using a principal component analysis method, the feature vector can be represented as a plurality of types of mutual relationships among multiple people such as others, friends, parents and children, and lovers as shown in the figure. Classified into groups. By creating the boundary surface of the area occupied by each group by learning and specifying the area corresponding to the feature amount of the detected trajectories of the plurality of persons, the type of mutual relationship between the detected persons can be determined. Determined.

  First, it is determined whether or not a plurality of detected persons are related to each other depending on whether or not the feature amount corresponds to another person's area (see FIG. 19). As long as the type of relationship is determined, the determination unit 113 may be configured.

  Next, output processing (step S04 in FIG. 6) executed by the output unit 114 will be described. The output unit 114 is discriminated by the discriminating unit 113 so that the distance between the plurality of display devices 30 arranged in the space and the plurality of people detected by the detecting unit 111 matches a predetermined condition. Outputs information according to the type of relationship. As described above, the trajectory map M is periodically updated. The positions of the tips of a plurality of trajectories indicated by the latest trajectory map M are the latest positions of a plurality of people. The output unit 114 specifies the display device ID of the display device 30 arranged at a position closest to a plurality of people using the position information (see FIG. 5) of the display device table 123. The output unit 114 displays the presentation information corresponding to the product field ID (see FIG. 5) corresponding to the specified display device ID and the type ID of the type determined by the determination unit 113, as the presentation information database 122 (FIG. 4). Extract from). The output unit 114 outputs the extracted presentation information to the display device 30 corresponding to the specified display device ID. The display device 30 displays the output presentation information.

  The content of the presentation information includes, for example, information indicating a popular product extracted from sales results, information indicating a new product, information indicating a product that the seller or manufacturer intends to increase the sales amount, information indicating a recipe for cooking, etc. It is. Any one of these pieces of information may be output, or a plurality of pieces of information may be output. When a plurality of pieces of information are output, a screen showing any one of the plurality of pieces of information may be displayed every time a predetermined time elapses, or a screen on which a plurality of pieces of information are arranged may be displayed Good.

  As an example of the presentation information, when type ID = 1 (parent and child (infant)) and product field ID = 1 (fresh food), the recipe of the dish preferred by the infant, the ingredients included in the recipe, and the like are shown. Information is output. In the case of type ID = 1 (parent and child (infant)) and product field ID = 2 (confectionery), information indicating a confection for infants is output. In this case, hiragana information may be output so that even an infant can easily understand, and a photograph or animation of a product, a video of a commercial message (TV CM), or the like may be output. When type ID = 1 (parent and child (infant)) and product field ID = 3 (daily necessities), information indicating daily necessities for infants is output.

  The same presentation information may be stored for different combinations of types and product fields. For example, in the case of type ID = 2 (friend), product field ID = 3 (fresh food), type ID = 3 (lover), and product field ID = 3 (fresh food), the presentation information is for home parties. Information indicating the recipe and ingredients contained in the recipe are output. Depending on the combination of the type and the product field, the presentation information may not be stored in the presentation information database 122. In that case, the presentation information is not output to the display device 30.

  According to the present embodiment, it is determined whether or not a person moving in the space and another person have a mutual relationship. Further, according to the present embodiment, the type of relationship between the person moving in the space and the other person is determined. Further, according to the present embodiment, the type of relationship is determined from the change over time of the distances of a plurality of people. Further, according to the present embodiment, the type of relationship is determined from the change over time in the difference between the velocity vectors of a plurality of people. In addition, according to the present embodiment, the type of relationship is determined from the residence time of a plurality of people for each area into which the space is divided. Further, according to the present embodiment, the weight for each region is added to the determination of the relationship type. In addition, according to the present embodiment, the type of relationship is determined from the temporal change in the position or speed of the center of gravity of a plurality of people. Further, according to the present embodiment, information according to the type of relationship is presented. In addition, according to the present embodiment, information is more easily transmitted to a plurality of people than when information is output regardless of the distance between the display device 30 and the plurality of people.

  The above embodiment may be modified as follows.

(1) The output unit 114 may be configured not to output information when the movement speeds of a plurality of persons detected by the detection unit 111 are equal to or higher than a threshold value. The movement speeds of a plurality of people are obtained by the method described in the second feature amount (see FIG. 15), for example. The case where the moving speeds of a plurality of people are equal to or higher than a threshold value is assumed, for example, when a visitor passes in front of a display shelf that is not of interest. In such a case, it is considered that even if the presentation information is output, the store visitor cannot visually recognize the presentation information, and the presentation information is not transmitted to the store visitor. According to this modification, output of information that is not transmitted to a person is avoided, so that the load on the information processing apparatus 10 is reduced.

(2) The output unit 114 may be configured to change the form of information according to the moving speeds of a plurality of people detected by the detection unit 111. When information is displayed on the display device 30 whose position is fixed, it is generally considered that the faster the movement speed of the person, the more difficult it is for the person to visually recognize the information. Therefore, if the moving speed is equal to or higher than the threshold, intuitively understood information such as a photograph or a picture of the product is displayed, and if the moving speed is lower than the threshold, a sentence explaining the product is displayed. Specifically, information in a form corresponding to the moving speed may be output. Alternatively, when the moving speed is equal to or higher than the threshold, the sentence is displayed with a relatively large character, and when the moving speed is less than the threshold, the sentence is displayed with a relatively small character. May be. According to this modification, information is more easily transmitted to a plurality of people than when the form of information is not changed according to the moving speed.

(3) When three or more types of relations are set in the feature quantity database 121 as in the example of FIG. 3, the classifiers configured by positive examples and negative examples are used for each type. A certain probability may be calculated and information corresponding to the type having the maximum probability may be output. Further, information corresponding to a type whose probability is equal to or greater than a threshold value may be output. When the probability of a plurality of types is equal to or greater than a threshold value, information corresponding to each of the plurality of types may be output, or information common to the plurality of types may be output.

(4) In the above embodiment, an example in which the presentation information is output to the display device 30 arranged in the store A is shown. However, the information is transmitted to the mobile terminal possessed by any of a plurality of people detected by the detection unit 111. May be output. For example, a wireless LAN base station is placed in the store A, and a visitor is encouraged to enable the wireless LAN function of the mobile terminal using a poster or in-store broadcast, and the mobile terminal is connected to the mobile terminal via the wireless LAN base station. The presentation information may be output. According to this modification, information is presented without arranging a device for presenting information in the space.

(5) A configuration for enhancing the feature amount database 121 may be provided. For example, when the determination unit 113 outputs the determined type to the output unit 114 and an operation for correcting the output type is received by the reception unit 15, the correction type and the calculation unit 112 calculate the corrected type. The feature amount may be associated with and stored in the storage unit 12. Specifically, when the determined type is parent and child, a screen showing presentation information corresponding to the parent and child is displayed. On this screen, the message "Information for parent and child is displayed. If you are not a parent and child, please select information with a button.", "Information for friends" button, "Information for lovers" A button is displayed. When the visitor operates the button, the presentation information corresponding to the operated button is displayed. The determination unit 113 stores the type corresponding to the operated button and the feature amount calculated by the calculation unit 112 in the feature amount database 121 in association with each other.

(6) Another example of a configuration for enhancing the feature quantity database 121 is shown below. The determination unit 113 corrects the determined type when the distances of the plurality of persons detected by the detection unit 111 at a specific location meet a predetermined condition, and the corrected type and the calculation unit 112 The calculated feature value may be associated with and stored in the storage unit 12. For example, if it is determined from the feature amount of the trajectory that two people are lovers, and it is determined from the updated trajectory that payment has been made separately at the checkout table B, the discriminating unit 113 determines the two people. The type of the relationship is corrected to a friend, and the corrected type and the feature amount calculated by the calculation unit 112 are stored in the feature amount database 121.

(7) A configuration may be adopted in which targets for determining the type of relationship are limited in advance. Specifically, a photographing device for photographing the outside of the store A is disposed near the entrance D of the store A, and a person entering the store A is photographed. Only when two or more people are detected from the captured image at the same time or with a time difference equal to or less than a predetermined threshold, the control unit 11 executes the processing shown in FIG. For detection of the person at the entrance D, the detection process (see FIG. 7) exemplified in the above embodiment may be used, or the face recognition process of the front face may be used. The ages of a plurality of persons estimated by the face recognition process may be used supplementarily for determining the relationship type. The heights of a plurality of persons estimated from the image may be used as auxiliary information for determining the type of relationship. The human height may be estimated using a stereo camera.

(8) The above embodiment is an example in which the imaging ranges E of the plurality of imaging devices 20 do not overlap, but even if the imaging ranges E of the plurality of imaging devices 20 overlap, only the image in one imaging range E Similar processing is executed even when not used. In addition, when the imaging ranges E of the plurality of imaging devices 20 overlap, the movement history R and the trajectory map M in the plurality of imaging ranges E are obtained by using the images of the plurality of imaging ranges E that overlap the imaging ranges E. You can simply connect them.

(9) In step S303 of the second calculation process of the above embodiment, the example in which the distance between the central coordinates of the human area H is determined only by the distance between the two human areas H is shown. A plurality of upper person areas H are selected in the order of short distance from the set X with respect to the area H, and the person ID with the highest frequency is selected among the person IDs assigned to the selected plurality of person areas H. May be. Feature extractors for calculating distances include simple color histograms and human identification models learned by deep learning (eg, E. Ahmed, M. Jones, and T. Marks, “An Improved Deep Learning Architecture for Person Re -Identification. ”, CVPR, 2015.) etc. may be used. Further, after assigning a person ID to each image of the set Y, the accuracy of the trajectory may be improved by correcting the trajectory.

(10) In the above embodiment, an example is shown in which a person is detected from a captured image, but a person may be detected by other methods. For example, beacon receivers are arranged at a plurality of locations in the store A, beacon oscillators are distributed to the store visitors, and the location of the store visitor is specified by the strength of the beacon signal received by the receiver. May be configured.

(11) In the above-described embodiment, the example in which the presentation information is output to the display device 30 arranged at a position closest to a plurality of people has been described. However, the presentation information may be output by a human operation. For example, the presentation information is configured to be output to the display device 30 when a person touches a touch panel provided in the display device 30 or when a person operates a switch provided in the display device 30. Also good. Alternatively, a printing device may be arranged in the store, and the presentation information may be printed by operating a touch panel of the display device 30. Alternatively, the presentation information may include image data of a coupon and the coupon may be printed. Moreover, it replaces with the display apparatus 30 and an audio | voice apparatus may be arrange | positioned and it may be comprised so that the audio | voice which shows presentation information may be output. Moreover, both the display device 30 and the acoustic device may be disposed.

(12) The presentation information may be output to the display device 30 in which the distance from all of a plurality of persons who are related to each other is equal to or less than a threshold. Alternatively, the presentation information may be output to the display device 30 whose distance to at least one of a plurality of persons having a relationship with each other is equal to or less than a threshold value. Even in these two cases, the presentation information may not be output when the distance between the display device 30 and a person other than a plurality of persons who are related to each other is equal to or less than a threshold value.

(13) The above embodiment is an example in which the present invention is applied to a system that presents information related to products to a plurality of people who coexist in the store A. However, the example of the store A is merely an example. The present invention may be applied to a system that presents information related to exhibits to a plurality of people coexisting in a space such as an exhibition hall or an exhibition hall.

(14) A program for causing a computer to execute the above processing may be provided in a state of being continuously stored in a computer-readable recording medium such as an optical recording medium or a semiconductor memory, or the Internet Etc. may be provided via a communication network. When the program according to the present invention is provided in a state of being continuously stored in a recording medium, the computer reads the program from the recording medium and uses it. Further, when the program according to the present invention is provided via a communication network, the computer receives the program from a distribution source device and uses it.

DESCRIPTION OF SYMBOLS 10 ... Information processing apparatus, 11 ... Control part, 12 ... Memory | storage part, 13 ... Communication part, 14 ... Display part, 15 ... Reception part, 111 ... Detection part, 112 ... Calculation part, 113 ... Discrimination part, 114 ... Output part 121 ... feature quantity database, 122 ... presentation information database, 123 ... display device table, 20 ... photographing device, 30 ... display device

Claims (14)

A storage unit that stores feature amounts of trajectories of the plurality of people when a plurality of people having mutual relationships coexist in a specific space;
A detection unit that repeatedly detects a plurality of people coexisting in the specific space;
A calculating unit that calculates feature amounts of the trajectories of the plurality of people detected by the detecting unit;
Information comprising: a determination unit that determines whether or not a plurality of people detected by the detection unit are related to each other using the feature amount calculated by the calculation unit and the storage content of the storage unit Processing equipment. The storage unit stores the feature amount for each type of the relationship,
The information processing apparatus according to claim 1, wherein the determination unit determines a type of the relationship corresponding to the feature amount calculated by the calculation unit using a storage content of the storage unit. The information processing apparatus according to claim 1, wherein the feature amount represents a change over time in the distances of the plurality of persons. The information processing apparatus according to claim 1, wherein the feature amount represents a change over time in a difference between speed vectors of the plurality of persons. The information processing apparatus according to claim 1, wherein the feature amount represents a residence time of the plurality of persons for each region obtained by dividing the space. The information processing apparatus according to any one of claims 3 to 5, wherein the determination unit performs determination using the feature amount multiplied by a weight determined for each area obtained by dividing the space. The information processing apparatus according to claim 1, wherein the feature amount represents a temporal change in a position or speed of a center of gravity of the plurality of persons. The information processing apparatus according to claim 2, further comprising: an output unit that outputs information according to the type of the relationship determined by the determination unit. The output unit outputs the information to a display device in which distances from a plurality of persons detected by the detection unit among a plurality of display devices arranged in the space meet a predetermined condition. The information processing apparatus described in 1. The information processing apparatus according to claim 8, wherein the output unit does not output the information when the moving speeds of the plurality of persons detected by the detection unit are equal to or higher than a threshold value. The information processing apparatus according to claim 8, wherein the output unit changes a form of the information according to a moving speed of the plurality of persons detected by the detection unit. The information processing apparatus according to claim 8, wherein the output unit outputs the information to a mobile terminal possessed by any of the plurality of persons detected by the detection unit. It has a reception unit that accepts operations,
The discriminating unit outputs the discriminated type to the output unit, and when an operation for correcting the output type is received by the receiving unit, the corrected type and the feature amount calculated by the calculating unit The information processing apparatus according to claim 8, wherein the information is stored in the storage unit. The discriminating unit corrects the discriminated type when the distances at specific locations of the plurality of people detected by the detecting unit meet a predetermined condition, and the corrected type and the calculating unit The information processing apparatus according to claim 1, wherein the calculated feature amount is associated and stored in the storage unit.

Images