JP2019101664A - Estimating program, estimating system, and estimating method - Google Patents

Abstract

To estimate at low cost or highly precisely a crowded situation of a staying area of objects in a space.SOLUTION: For an object detected from each of multiple first images that are images of a space, the objects correlated with each other between the multiple first images are grouped on the basis of detected positional information on each object and on a detection situation of each object in each of the multiple first images, and a staying area where each object stays in the space is estimated on the basis of the grouping.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an estimation program, an estimation system, and an estimation method.

  As a method of grasping the crowded situation of the store, for example, there is known a method of grasping the waiting condition of the service provision of the user by analyzing the usage situation of the ticketing machine by the user in the restaurant such as a meal ticket system.

  With the above-mentioned method, it is possible to grasp the crowded situation of the store, but it is difficult to grasp the crowded situation of the seat, for example, the seating or vacant seat situation of the seat.

  As a method of grasping the crowded condition of the seat, there is known a method of installing a pressure sensor in each seat or installing an infrared sensor in a store. In the method using the pressure sensor, for example, by installing one pressure sensor for each seat, it is possible to detect the seating situation in each seat and to grasp the crowded situation of the store. Moreover, in the method using an infrared sensor, for example, the position and the number of people can be grasped by detecting infrared rays from a human body by an infrared sensor installed in a store.

  However, in these methods, for example, one pressure sensor is attached to each seat, or a seat area where detection of infrared rays from the human body is difficult to be set as analysis software, or for analysis Operation costs such as operation and maintenance of software of

  Therefore, for example, in the case of a store having 100 or more seats such as a restaurant, the cost described above may occur for each store, so it is difficult to easily introduce it.

  By the way, the method of grasping | ascertaining the congestion condition in a shop is also known by analyzing the imaging | video image | photographed with the surveillance camera installed in the shop.

JP, 2010-86300, A JP, 2017-156956, A

  However, in the method of analyzing the video of the surveillance camera, the angle of the surveillance camera (for example, the angle and angle of view in the shooting direction in the store), the resolution of the video, the shooting conditions, etc. greatly affect the recognition accuracy. Further, for example, it is possible to designate and monitor an area to be monitored of the crowded situation, or to monitor by comparing a photographed image with a vacant seat image. However, in order to perform such monitoring, various conditions such as information on the seat position of the store and setting / installation conditions of the monitoring camera are set in the software.

  As the information on the seat position, there is a seat layout diagram in the above example. The seat position (area) is an example of a "retention area" in which an object such as a person is stagnating in space.

  In the case of a company having many stores, setting up such software on a store-by-store basis can result in a large cost.

  In one aspect, the present invention is directed to low-cost or high-accuracy estimation of congestion in a stagnant area of an object in space.

  In one aspect, the estimation program may cause a computer to execute the following processing. The process is performed based on position information of each detected object and a detection state of each object in each of the plurality of first images, for an object detected from each of the plurality of first images captured in space. And interrelating objects may be grouped among the plurality of first images. Further, the processing may estimate a staying area in which each object stays in the space based on the grouping result.

  In one aspect, the congestion status of the stagnant area of an object in space can be estimated at low cost or with high accuracy.

BRIEF DESCRIPTION OF THE DRAWINGS It is a block diagram which shows the structural example of the congestion degree estimation system which concerns on one Embodiment. It is a figure which shows an example of the operation | movement phase which concerns on one Embodiment. It is a figure which shows the example of a detection of the object by the server which concerns on one Embodiment. It is a figure which shows an example of detection information. It is a figure which shows an example of a static box and a non-static box. It is a figure which shows an example of the similarity determination processing of the coordinate of a box. It is a figure which shows an example of the similarity determination process of box itself. It is a figure which shows an example of the estimation process of a seat position. It is a figure which shows an example of the distance parameter | index of hierarchical clustering. It is a figure which shows an example of the presumed procedure of a seat position. (A)-(c) is a figure which shows an example of the estimated degree of congestion. It is a figure which shows an example of the presentation aspect of congestion degree. It is a flowchart which shows the operation example of the seat position estimation phase which concerns on one Embodiment. It is a flowchart which shows the operation example of the seat position estimation phase which concerns on one Embodiment. It is a flowchart which shows the operation example of the congestion condition estimation phase which concerns on one Embodiment. It is a figure showing the example of hardware constitutions of the computer concerning one embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the embodiments described below are merely examples, and there is no intention to exclude the application of various modifications and techniques not explicitly stated below. For example, the present embodiment can be variously modified and implemented without departing from the scope of the present invention. In the drawings used in the following embodiments, portions given the same reference numerals indicate the same or similar portions unless otherwise specified.

[1] One Embodiment As described above, in order to analyze the video taken by the surveillance camera and grasp the congestion status of the seat, the information of the seat layout, the setting / installation conditions of the surveillance camera, etc. are set in the software As the number of stores increases, the cost may increase.

  Therefore, in one embodiment, an estimation system capable of estimating a seat position based on an image captured by a surveillance camera will be described.

  For example, the estimation system according to an embodiment may perform the following processing.

  -For an object detected from each of the plurality of first images captured in space, the plurality of first based on position information of each detected object and a detection state of each object in each of the plurality of first images Group objects related to each other in one image.

  Based on the result of grouping, estimate the staying area in which each object stays in space.

  According to the above processing, the estimation system according to the embodiment can estimate the staying area where each object stays, for example, the seat position where a person is present, based on the plurality of first images. In addition, a retention area means a specific area where a person stays, and may include, for example, a standing area (space) as well as a seating area such as a chair or a cushion. As an example of a standing area, a standing food (or called “stand-by”) type restaurant or the like, a facility that receives a service while the customer stands, etc. There is a seating area.

  For example, by estimating the staying area in the space based on the first image obtained by imaging the space, it is possible to easily estimate the congestion state of the staying area based on the image obtained by imaging the space and the estimated staying area. it can.

  For example, when estimating the crowded situation of the seat based on the video of the surveillance camera, the process of setting (or updating) the conditions such as the information of the seat position in the software can be unnecessary, and the cost can be reduced. it can.

  In addition, when the crowded situation of the seat is grasped using the information of the seat layout prepared in advance, the photographed image is corrected based on the setting / installation conditions of the monitoring camera, etc. In some cases, it may not be possible to accurately estimate the congestion situation in order to compare with. On the other hand, in the estimation system according to one embodiment, in the estimation of the staying area and the estimation of the congestion state, a process using an image captured is performed. Therefore, the influence of the setting and installation conditions of the surveillance camera can be reduced, and the congestion situation can be estimated with high accuracy.

  From the above, the estimation system according to an embodiment can, for example, estimate the utilization state of the staying area in space at low cost or with high accuracy, based on the information of the estimated staying area.

  In addition, the surveillance camera which image | photographs space is highly likely to be introduce | transduced and used in the store etc. by the objective, such as crime prevention. Therefore, existing equipment can be used by using the monitoring camera to estimate the congestion situation of the seat, so it is better to introduce a pressure sensor, an infrared sensor, a ticket issue machine, etc. to estimate the congestion situation of the seat. Even cost can be reduced.

[1-1] Configuration Example of One Embodiment Hereinafter, a configuration example of one embodiment will be described. FIG. 1 is a block diagram showing a configuration example of a congestion degree estimation system 1 according to an embodiment. The congestion degree estimation system 1 is a system for estimating the congestion degree of the staying area in the facility to be estimated, and is an example of the estimation system described above. The congestion degree estimation system 1 may include, as shown in FIG. 1, one or more (a plurality of in the example of FIG. 1) surveillance cameras 2 and a server 4 as an example.

  The monitoring camera 2 and the server 4 may be communicably connected to each other by, for example, the network 5.

  The network 5 may be, for example, at least one of the Internet and an intranet, including a Local Area Network (LAN) or a Wide Area Network (WAN), or a combination thereof. The network 5 may also include a virtual network such as a VPN (Virtual Private Network). The network 5 may be formed by either or both of a wired network and a wireless network.

  The surveillance camera 2 may capture a space in the shooting direction and acquire an image sequence, for example, a plurality of images (which may be referred to as “frames”) arranged in time sequence such as a moving image.

  The surveillance camera 2 may be installed, for example, in the store 3 and, as an example, is arranged so as to include in the imaging range seats for customers (preferably all seats for customers) existing inside and outside the store 3 You may

  In order to include all the seats inside and outside of the store 3 in the shooting range, for example, a plurality of monitoring cameras 2 are installed at positions that complement each other's blind spots (or some of the shooting ranges overlap) Alternatively, one or more movable surveillance cameras 2 may be installed. Alternatively, a combination of these may be employed.

  The monitoring camera 2 may transmit the acquired video to the server 4 via the network 5. For example, the monitoring camera 2 may accumulate the acquired video in a recorder or the like (not shown) and transmit the data in the recorder to the server 4 at a predetermined timing. As the predetermined timing, various conditions such as the arrival of a predetermined time, the elapse of a predetermined time, the storage capacity to the recorder, the number of storage frames, and the like may be used. Alternatively, the monitoring camera 2 may transmit the captured video to the server 4 without passing through the recorder. When not via the recorder, for example, the monitoring camera 2 may transmit an image every one to several frames (in substantially real time).

  As surveillance camera 2, a box type camera, a dome camera, a network camera etc. are mentioned, for example. The network camera may, for example, be an IP (Internet Protocol) camera. In addition, in the shop 3 in which the illumination inside or outside the shop is dark, a night vision camera such as an infrared camera may be used as the monitoring camera 2. For example, cameras for various uses such as a security camera, a surveillance camera, a street camera, and a fixed point camera may be used as the surveillance camera 2.

  The server 4 estimates the seat position in the store 3 based on the video captured by the monitoring camera 2 and estimates the degree of congestion of the seat in the store 3 based on the estimated seat position and the video captured by the monitoring camera 2 You may Details of the server 4 will be described later.

  In addition, the information of the congestion degree of the seat which the server 4 estimated may be provided to the terminal device 6, for example, as shown in FIG. As an example, the server 4 may have a function of a Web server (for example, an information presentation unit 14 described later), and the Web server functions to terminal a Web page representing the degree of congestion of the store 3 via the network 5 It may be displayed on the device 6.

  The terminal device 6 is a computer that receives information on the degree of congestion of the store 3 estimated by the server 4. For example, the terminal device 6 may be a computer possessed by a candidate who uses the store 3 who is interested in the crowded situation of the seat of the store 3. Examples of the terminal device 6 include various information processing devices such as a PC (Personal Computer) such as a desktop, a laptop, or a mobile, a tablet, a smartphone, and a mobile phone.

  For example, the terminal device 6 transmits various requests such as transmission of a request for acquiring the congestion status of the seat of the store 3 and reception of a response regarding the estimation result of the congestion status with the server 4 via the network 5 You may do it.

  In addition, as illustrated in FIG. 1, when the terminal device 6 is a PC, a tablet, a smart phone, a mobile phone or the like that performs wireless communication, connection with the network 5 is performed via the mobile network via the base station 7. It is good.

  The terminal device 6 may include, for example, an input unit of information (operation request) from the user, an output unit of information to the user, a communication unit with the server 4, and the like.

[1-2] Operation Phase of Server 4 Next, the operation phase of the server 4 will be described.

  As described above, the server 4 may estimate the seat position based on the image captured by the monitoring camera 2. Then, the server 4 may estimate the (for example, the latest) congestion status of the seat based on the estimated seat position and the (for example, the latest) video captured by the monitoring camera 2.

  Although various image recognition techniques may be used for the estimation of the seat position based on the image, in one embodiment, it is exemplarily shown by Deep Learning using a neural network (NN; Neural Network). It is assumed that a detection model is used.

  Although it depends on the service content and the time zone provided by the store 3, the period in which the customer uses the seat in the store 3 is considered to be ten minutes to several tens of minutes (or more than one hour). The surveillance camera 2 can capture image data of one frame (1 FPS; Frame Per Second) or more at least per second, but all frames to detect changes in seating, such as seating, leaving and leaving of the customer Analyzing the data of is inefficient.

  For this reason, in machine learning, image data of a part of frames of the captured image data may be used. For example, in one embodiment, frames acquired at a predetermined sampling interval, such as five minutes apart from an image, may be input to the neural network.

  Also, in one embodiment, an estimation period of, for example, several hours to several days may be provided to estimate the seat position. In the following description, it is assumed that an estimation period of 3 days (predetermined period) is provided as an example.

  From the above points, the method according to one embodiment may be divided into the phases illustrated in FIG. 2 and implemented. For example, as shown in FIG. 2, the setting for transmitting the video of the monitoring camera 2 to the server 4 is performed for the store 3 in which the monitoring camera 2 is operated, and the seat position for three days as a predetermined period An estimation phase may be provided.

  When the seat position is estimated by the seat position estimation phase, then the congestion state estimation phase by the server 4 may be started. In the congestion state estimation phase, the server 4 may estimate the congestion state at a predetermined timing based on the estimated seat position and the image data sent from the monitoring camera 2.

  As such, the seat position estimation phase may be positioned as the initialization phase of the method according to an embodiment, and the congestion situation estimation phase may be positioned as a normal operation phase of the method according to one embodiment.

  In the store 3, for example, layout change in the store or movement of a seat by a customer or an employee may occur. Therefore, after the end of the seat position estimation phase, the seat position estimation (updating) phase for updating the estimated seat position may be performed also in the congestion situation estimation phase. Since the seat position estimation (updating) phase can be performed in the same process as the seat position estimation phase, in the following description, these are simply referred to as the seat position estimation phase without distinction.

[1-3] Configuration Example of Server 4 Next, a configuration example of the server 4 will be described. The server 4 is an example of one or more computers installed in the store 3, a company having the store 3 or the like, or a data center or the like. Examples of the server 4 include various physical server devices and / or virtual server devices. At least a part of the functions of the server 4 may be realized using, for example, resources provided by a cloud service, a framework, an application, and the like. Also, at least a part of the functions of the server 4 may be distributed or redundantly arranged on a plurality of computers.

  As illustrated in FIG. 1, the server 4 may include, for example, a memory unit 11, a control unit 12, an NN 13, and an information presentation unit 14.

  The memory unit 11 stores various information used for processing of the server 4. Information stored in the memory unit 11 will be described later in the description of the function of the server 4. The memory unit 11 may be one of a memory, a volatile memory such as a random access memory (RAM), and a storage unit such as a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). Both are mentioned.

  The control unit 12 performs control regarding estimation of the seat position and estimation of the congestion state. As illustrated in FIG. 1, the control unit 12 may include, for example, an information acquisition unit 121, a static box determination unit 122, a seat estimation unit 123, and a congestion degree calculation unit 124.

  The information acquisition unit 121 may receive the video data transmitted from the monitoring camera 2 (store 3) and store the video data in the memory unit 11 as the video data 111. In one embodiment, the control unit 12 and the NN 13 may use time-series image data (frames) captured at five-minute intervals. The image data arranged in time series is an example of a plurality of first images.

  For example, the information acquisition unit 121 may store information obtained by extracting frames at intervals of 5 minutes from the received video data as the video data 111 in the memory unit 11. Alternatively, in the store 3 or the monitoring camera 2, a frame group extracted at intervals of 5 minutes from the captured video data may be transmitted to the server 4 as video data.

  Note that the video data 111 may include image data of at least N (N is an integer; for example, “5”) frames. The information acquisition unit 121 sets an upper limit on the data size or the number of frames of the video data 111 from the viewpoint of saving the storage capacity of the memory unit 11, and for frames exceeding the upper limit, the image date and time in the video data 111 is in the order of the past frames. You may delete the frame.

  The static box determination unit 122 and the seat estimation unit 123 estimate the seat position of the store 3 from the video data 111 in cooperation with the NN 13 in the seat position estimation phase. In other words, the static box determination unit 122 and the seat estimation unit 123 are an example of the following grouping unit and estimation unit.

  The grouping unit is configured based on position information of each detected object and a detection state of each object in each of the plurality of first images with respect to the objects detected from each of the plurality of first images captured in the space. The mutually related objects may be grouped among the plurality of first images. The estimation unit may estimate a staying area in which each object stays in space based on the grouping result.

  The congestion degree calculation unit 124 calculates the congestion degree of the store 3 based on the estimated seat position. In other words, the congestion degree calculation unit 124 estimates the congestion degree of the staying area based on the position information of the object detected from the second image obtained by imaging the space and the information on the staying area in the estimated space. It is an example of a congestion degree estimation part.

  The information presentation unit 14 may have, for example, a function of a Web server or a DB (Database) server, and the congestion calculated by the congestion degree calculation unit 124 with respect to the terminal device 6 in response to a request from the terminal device 6 The degree information may be presented.

  Hereinafter, an example of the function and operation of the server 4 will be described.

[1-3-1] Description of NN First, the NN 13 will be described. The NN 13 detects an object from each of a plurality of first images of which the surveillance camera 2 has captured a space. For example, the NN 13 may detect an object based on video data 111 stored in the memory unit 11 and including a plurality of image data at 5-minute intervals.

  The NN 13 may be a system in which machine learning is performed in advance to detect an object to be detected from image data. For example, a detection model that detects the head of a person from image data may be applied to the NN 13. The head of a person is an example of an object to be detected.

  An example of detection of an object by the NN 13 is shown in FIG. As illustrated in FIG. 3, the NN 13 may calculate the position and score of the head of a person from the input image data. In addition, FIG. 3 shows a mode that the position and score of the head which NN13 detected based on the said image with respect to the image of the imaging | photography space 30 image | photographed with the monitoring camera 2 were fitted. The image of the imaging space 30 may have a coordinate system in which the upper left of the image is (0, 0) and the lower right of the image is (x, y). x may be the size of the width of the image (y), and y may be the size of the height of the image (number of pixels).

  The position of the head may be identified, for example, as a rectangular area surrounding the head. Hereinafter, the rectangular shape surrounding the head may be referred to as a "box". The shape of the box is not limited to the rectangular shape, and may be a circular shape, an elliptical shape, a polygonal shape, or the like.

  The score is an example of information indicating the certainty (likelihood) that the detected area is the head of a human body, the maximum value of which is “1.000”. The example of FIG. 3 is a frame photographed at 12 o'clock ("12:00").

  By setting the object to be detected as the head of a person (human body), it is possible to capture a part of the human body that is likely to move to the monitoring camera 2 and improve detection accuracy. Further, by detecting an object by deep learning, it is possible to detect the head of a person in various postures or states other than the front face.

  For example, the NN 13 may have various postures such as the back (the back of the head; see A in FIG. 3), the upper surface (the top of the head (rolled posture); B), and the sides (cross-face; see C). Even the head can be detected accurately.

  In addition, the NN 13 is in a state in which the head is hidden by another object, for example, a state in which it is blocked by a chair (refer to code D), a cap state (refer to code E), or others or obstacles in crowded places Even in the state of being blocked by an object, etc., the head can be detected accurately.

  The NN 13 may store the detected position and score of the head in the memory unit 11 as the detection information 112. The detection information 112 may be generated for each frame, or in the detection information 112, identification information of a frame may be associated with an object. The detection information 112 may be a file of various formats. For example, the detection information 112 may be a file of a format such as CSV (Comma-Separated Values).

  FIG. 4 is an example of the detection information 112. As shown in FIG. 4, the detection information 112 may illustratively include information of “No.”, “x1”, “y1”, “x2”, “y2”, and “score”. The example of FIG. 4 shows information of an object detected in one frame (image).

  “No.” is information for identifying a detected object. "No." may further include information identifying a frame. “X1” and “y1” indicate the coordinates (x1, y1) of the top left corner of the detected head box, and “x2” and “y2” indicate the bottom right corner of the detected head box Coordinates (x2, y2) may be indicated. The score may indicate the likelihood of the box represented by (x1, y1), (x2, y2). Note that the box set in the detection information 112 by the NN 13 may be limited to a box whose score is “0.500” or more.

[1-3-2] Description of Static Box Determination Unit The static box determination unit 122 sits on the seat based on the detection information 112 which is the detection result of the head of a person as pre-processing for estimation of the seat position. Position of the person who is

  Note that at least part of the following processing by the static box determination unit 122 may be performed by deep learning using an NN. For example, the static box determination unit 122 may include an NN, or the NN 13 may be configured to further execute the following processing.

  For example, the static box determination unit 122 compares the detection result of the head related to a certain frame with the detection result of the head related to the past few frames, and extracts a box remaining at the same position. The box staying at the same position may include, for example, (the head of) a person seated in a seat at the store 3. In other words, the static box determination unit 122 detects, from the captured image of the monitoring camera 2, the staying object including the object remaining in the staying area for each captured image.

  Similarity determination of box coordinates (e.g., size and position) and similarity determination of the box itself may be used to determine the identity of the box between the frames.

  Hereinafter, an example of a method of acquiring the data of the head position of “the person sitting” by the static box determination unit 122 will be described. In the following description, the box at the head position of the sitting person is referred to as a "static box".

  The static box determination unit 122 may divide and manage data of the detection result of the head related to the past frame into the following two types of data.

  (A) static box data.

  (B) Non-static box data other than static box in the past N frames. Note that N = 5.

  The static box determination unit 122 may store, for example, data of static box and non-static box in the memory unit 11 as box information 113. The box information 113 may include, for example, as static box data, identification information of a box determined to correspond to the static box in a plurality of frames for each static box. Also, the box information 113 may include, as non-static box data, identification information of a box that is not a static box in a plurality of frames.

  As described above, the static box determination unit 122 is an example of a management unit that manages a predetermined number or more of objects determined to be present at the same position among a predetermined number or more of first images in association with one static object. is there.

  The static box determination unit 122 performs, for example, the following processes (i) to (iii) on the image data of each time based on the detected position of the object and the detection condition of each object in each frame. May run.

  In the following description, the “frame of the current time” may be read as the frame to be determined. That is, the static box determination unit 122 may compare each object detected in the frame to be determined with N frames past the frame. In addition, when the static box determination unit 122 makes a determination, the next (five minutes later) frame may be set as a determination target frame and compared with the N frames past the frame.

  (I) The static box determination unit 122 compares the box detected in the frame of the current time with the static box of the frame of the past time, and adds the box determined to be identical to the data of the static box.

  The static box determination unit 122 may acquire, from the detection information 112, the information of the box detected in the frame at each time. A method of determining whether or not the static box is the same between the frames will be described later.

  (Ii) The static box determination unit 122 sets a box not determined to be a static box among the boxes detected in the current time frame to data of non-static boxes other than the static box of the past N frames. Compare. Then, when detecting a combination of boxes satisfying the condition of the static box, the static box determination unit 122 adds the box to data of a new static box.

  For example, if M frames in N (N = 5) frames include boxes that can be determined to be identical to each other, static box determination unit 122 newly adds a group of these boxes as a static box. You may judge. M is an integer less than N, and may be, for example, “3” or the like.

  (Iii) For a static box not observed (detected) continuously in M frames, the static box determination unit 122 writes the data of the box to a file of the storage unit, for example, and from the data of the static box on the memory, You may delete it. As such a situation, for example, an event such as a person who has been seated may have left the seat may be mentioned.

  As described above, the memory unit 11 may be one or both of the memory and the storage unit. Therefore, the process of (iii) above stores the data of the active static box detected in the M frame in the past N frames in the memory, and the data of the inactive static box not detected in the past M frame Means to evacuate the storage unit.

  FIG. 5 is a diagram showing an example of static and non-static boxes in three frames at times "10:00", "10:05" and "10:10". As shown in FIG. 5, for example, a box indicated by hatching in the upper right and a box indicated by hatching in the lower right, which are determined to be identical in three frames, are respectively static boxes (“static box” It is determined to be 1 "and" static box 2 "). On the other hand, it is determined that a white background box in which frames determined to be identical in the past 5 frames are less than 3 frames is a non-static box ("non-static box").

  Next, a method of determining the identity of boxes between frames will be described.

(Similarity determination of box coordinates)
First, a method of determining similarity of box coordinates will be described. The static box determination unit 122 may compare the coordinates of the boxes between the frames, and determine that both are “different boxes” if the comparison result exceeds the threshold. Coordinates to be compared may include at least one of size and position.

  The static box determination unit 122 determines a pair of boxes whose sizes are largely different and / or positions (in other words, distances) are largely separated by the similarity determination of the coordinate of the box as the determination target of the static box. May be excluded from This makes it possible to reduce the amount of calculation in similarity determination of the subsequent box itself.

  FIG. 6 is a diagram showing an example of the similarity determination process of the coordinates of the box. As illustrated in FIG. 6, the static box determination unit 122 may perform at least one of comparison of box sizes between frames and comparison of box positions between frames. In the example of FIG. 6, the current frame (f) and the one past frame (f-1) are used as two frames to be compared.

  For example, as shown in the upper left of FIG. 6, the static box determination unit 122 determines whether the size ratio of the two boxes to be compared is equal to or less than a threshold, for example, “1.3”. The box size ratio may be determined, for example, by (larger box size) / (smaller box size). The box size may be, for example, the average of the width and height of the box, and may be determined by (width + height) / 2.

  Also, for example, as shown in the upper right of FIG. 6, the static box determination unit 122 determines whether the distance between two boxes to be compared is a threshold, for example, “average box size × 1.5” or less. judge. Note that the average box size may be the average box size of two boxes to be compared or a static box. Alternatively, the average box size may be the average box size of the detected box in the current frame, past N frames, or all frames analyzed so far.

  If at least one of the size ratio of the boxes and the distance between the boxes is larger than the corresponding threshold value, the static box determination unit 122 determines that the two boxes are static boxes. You may exclude from judgment object.

(Similarity determination of the box itself)
Next, the method of judging the similarity of the box itself will be described. For example, the static box determination unit 122 may score the similarity of crop images of two boxes to be compared and the overlap between the boxes, respectively, and calculate the total score of these scores. Then, the static box determination unit 122 may determine that the two boxes are the same if the total score is equal to or greater than a threshold (for example, “1”).

  The cropped image is an image obtained by cutting out an object (in one embodiment, a head) from image data. Determining the similarity score of the cropped image may include, for example, determining the similarity score of the color histogram, and determining the similarity score of the pixel difference.

  FIG. 7 is a diagram showing an example of similarity determination processing of the box itself. As illustrated in FIG. 7, the static box determination unit 122 may calculate the following score.

  (I) Score of similarity of color histogram: "sim_color"

  An example of the color histogram is a histogram of RGB (Red Green Blue) values.

  (II) Similarity score by pixel difference: "sim_pixel"

  The similarity based on pixel differences may be determined, for example, by a method such as NRMSE (Normalized Root Mean-Squared Error). For the calculation method of pixel difference by NRMSE, for example, [http: // scikit. image. org / docs / dev / api / skimage. measure. html # skimage. measure. It is described in [compare_nrmse].

  (III) Box overlap degree: IoU (Intersection over Union)

  IoU is information indicating the degree of overlap of two boxes (in other words, areas). For example, as shown in FIG. 7, IoU indicates the ratio of overlapping overlapping area (AND area) to a combined area (OR area) in which two areas are connected when two areas are overlapped in the coordinate system. .

  The scores (I) to (III) may be calculated by a neural network.

  The static box determination unit 122 calculates “sim_color * sim_pixel + IoU” as a total score based on the calculated scores (I) to (III). The overall score is an example of an index for determining whether the image components of the object match.

  Then, the static box determination unit 122 determines whether or not the total score is equal to or higher than a threshold (for example, “1”). If the total score is equal to or higher than the threshold value, the static box determination unit 122 may determine that two boxes to be compared are the same box between frames.

  The static box determination unit 122 may determine the similarity in the following manner when comparing the box of the current frame with the static box of the past frame in the process of (i). For example, the static box determination unit 122 calculates an overall score of similarity between the comparison target box in the current frame and the comparison target static box in each of a plurality of past frames. Good. Then, the static box determination unit 122 may calculate the average value of the calculated overall score of the similarity, and determine whether the average value is larger than a threshold (for example, “1”). If the mean value is greater than the threshold, then the box to be compared may be added to the static box.

  In addition, the static box determination unit 122 may perform similarity determination as follows in comparison between the box of the current frame and the non-static box of the past frame in the process (ii). . For example, the static box determination unit 122 may calculate the similarity between the comparison target box in the current frame and the non-static box in each of a plurality of past frames. Then, the static box determination unit 122 determines whether the calculated overall score of similarity is greater than a threshold (for example, “1”) in M (for example, “3”) frames in the past N (for example, “5”) frames. It may be determined whether or not. If the total score is greater than the threshold value in M frames or more, the compared box and the compared non-static box may be newly determined as a static box.

  As described above, the static box determination unit 122 can reduce the probability of false detection of the seat position by determining the similarity of the box itself.

  For example, positions such as the front of the cash register and the front of the toilet are places where the customer may stay other than the seat. Such locations may be detected consecutively in the 5-minute interval between frames.

  On the other hand, the static box determination unit 122 determines the similarity of the boxes themselves, and for example, when the boxes to be compared are the heads of persons different from each other, the overall score becomes equal to or less than the threshold and static It is not determined to be a box. This reduces the possibility that the location will be detected as a seat position, even if the heads of different persons are detected across multiple frames at locations other than the seat, such as before the cash register or in the bathroom. It can be eliminated and the seat position can be accurately estimated.

[1-3-3] Description of Seat Estimation Unit The seat estimation unit 123 estimates the seat position based on the information on “the person sitting” accumulated as the box information 113, and determines the estimated seat position information It is stored in the memory unit 11 as the information 114.

  Note that at least part of the following processing by the seat estimation unit 123 may be performed by deep learning using an NN. For example, the seat estimation unit 123 may include the NN, or the NN 13 may be configured to further perform the following processing.

  For example, the seat estimation unit 123 uses the box information 113 in which the position data (static box) of the "person sitting" determined by the static box determination unit 122 is stored for a predetermined period (about three days in one embodiment) Seat position and number of seats may be estimated.

  As an example, the seat estimation unit 123 may perform clustering based on static box information, treat the estimated number of clusters as the number of seats, and handle the position of each cluster as the position information of each seat. In addition, a position, a size, and observed time (frame) are mentioned as information of a static box. Also, the position of each cluster may be calculated based on the average value of the positions of static boxes included in each cluster.

  In one embodiment, it is assumed that the number of seats of the store 3 is not acquired / set in order to simplify the initial setting. In this case, the clustering algorithm may be based on hierarchical clustering.

  For example, as shown in FIG. 8, the seat estimation unit 123 sets a state in which all the observed static boxes are separate clusters as an initial state (see “static boxes”), and sets close clusters as the same cluster. May merge (see “clustering result”). By providing a threshold for the distance to merge, the clusters separated by a distance remain unmerged, so the seat estimation unit 123 may estimate each remaining cluster as final seat data ("estimated seat positions "reference).

  Here, a cluster is a group of observed static boxes, and is an example of an object mutually related among a plurality of first images. Static boxes in the same cluster are considered as observation results of another person sitting in the same seat. That is, in one embodiment, the position of the finally remaining cluster is regarded as the position where the object stays, and the position of the object is estimated as the seat position to estimate the seat position.

  Hereinafter, with reference to FIG. 9, a distance index for performing hierarchical clustering will be described. FIG. 9 is a diagram showing an example of a distance index of hierarchical clustering.

The seat estimation unit 123 may set a representative value of a cluster for distance calculation. In the example of FIG. 9, the seat estimation unit 123 sets, for each static box, the average value of the central coordinates of the static box in the cluster to (c _x , c _y ), the average value of the box size to size, and The number of static boxes included may be n. In addition, the seat estimation unit 123 sets a binary array in which the observation time (frame) is “1” and the non-observation time (frame) is “0” for static boxes included in the cluster. It may be generated and held in the memory unit 11, for example. The array is an example of information indicating the detection status of each object in each of the plurality of first images.

  The seat estimation unit 123 may obtain the distance index between clusters by calculating the following equations (1) to (3) as illustrated in FIG. 9 using the above-described representative values.

_{Here, (c xA, c yA)} , size A, n A , respectively, show the coordinates of the cluster A, the size, the static box _{number, (c xB, c yB)} , size B, n B , respectively , Coordinates of cluster B, size, and number of static boxes.

  In addition, in said (3) Formula, parts other than "penalty" are the same as the distance index of hierarchical clustering which used the ward method. On the other hand, in one embodiment, the estimation accuracy of the seat position and the number of seats is improved by adding "penalty" to the ward method distance index.

  For example, when static boxes at the same time (the same frame) are observed between cluster A and cluster B, in other words, "person sitting", these clusters have different seats. Is expected.

  Therefore, in the above equation (3), when static boxes are observed at the same time (the same frame) between cluster A and cluster B, the distance between clusters is increased according to the number of times of observation. A penalty term has been added. For example, the seat estimation unit 123 may take AND of arrays of observation times, and apply SUM (total value) of the array of AND results as a penalty to the equation (3).

  As described above, the seat estimation unit 123 may generate, for each static object, information indicating whether or not there is an object associated with the static object in each of the plurality of first images. Then, based on the generated information, the seat estimation unit 123 may exclude static objects whose corresponding objects are present in one first image from grouping targets.

The seat estimation unit 123 uses d _ward obtained by the equation (3) as a distance index, and from the initial state in which the data of all static boxes are separate clusters (see “initial state” in FIG. 10), Hierarchical clustering may be performed according to the following procedure.

For example, the seat estimation unit 123 determines a cluster pair with the smallest distance index (d _ward ) between clusters, and determines whether the stop condition (stop condition) in the following equation (4) is satisfied.

  If the stop condition of the equation (4) is satisfied, the seat estimation unit 123 ends hierarchical clustering and stores information on remaining clusters as the seat information 114 in the memory unit 11.

  On the other hand, when the stop condition of the equation (4) is not satisfied, the seat estimation unit 123 merges the cluster pair and sets it as a new cluster.

  The seat estimation unit 123 repeatedly executes the above process until the stop condition is satisfied or the number of clusters becomes one.

Thus, clusters remaining without being merged by hierarchical clustering are estimated as seat positions (see “estimated result of seat position” in FIG. 10). The seat estimation unit 123 uses the average value (c _x , c _y ) of the center coordinates of the static box included in the cluster and the average value (size) of the box size as data of the seat position and the reference size. Good.

  The seat estimation unit 123 may determine that a cluster having an extremely small number of samples (value of n) included in a cluster among the clusters left unmerged may be determined as noise and excluded from the seat information. . A cluster with an extremely small number of samples may be, for example, a cluster having a few percent (exemplarily 2%) or less of the average value of the number of samples contained in other clusters.

[1-3-4] Description of Congestion Degree Calculation Unit The congestion degree calculation unit 124 calculates the congestion degree using the head detection result and the seat information 114 in the congestion state estimation phase.

  For example, the congestion degree calculation unit 124 calculates the distance between the detection result of the human head and each seat position of the seat information 114, and sequentially detects the pair of the detection result and the seat position closer to the seat position. The detection result of the head of a person present at a distance equal to or less than the threshold from the seat position is assigned.

  As a result, information of a seat (seat at which a person is present) to which the detection result of the head is assigned and information of a seat (vacant seat) to which the detection result is not assigned can be obtained. When the detected position of the head and the seat position are separated by a predetermined distance or more, the head may not be assigned to the seat position.

  As a detection result of a human head, for example, a detection result of an object in a second image acquired in the congestion state estimation phase may be used. As an example, the detection result of the human head is the object detected in the latest detection target frame (or at the time requested by the terminal device 6) included in the detection information 112 stored in the memory unit 11. Information may be used.

  The congestion degree calculation unit 124 may calculate the congestion rate based on the detection target frame, for example, by calculating “the number of seats with people / the total number of seats”. In addition, the congestion degree calculation unit 124 sets “Threshold” / “Normal” / “Heavy” by setting a threshold such as “0.3” or “0.6” to the calculated congestion rate. The degree of congestion (discrete value) representing the degree of congestion status such as In the above example, although the congestion level is estimated to be three levels, the number of levels of the congestion level may be increased or decreased by increasing or decreasing the number of threshold values.

  As described above, when the seat position estimation (update) phase is executed during the crowded situation estimation phase, the static box determination unit 122 and the seat estimation unit 123 select the second image from the second (first) image. The seat information 114 may be estimated (updated) using the detection result of the object.

  An example of the estimation result of the congestion degree by the congestion degree calculation part 124 is shown in FIG. Note that FIG. 11 shows a state in which a seat where a person is present and an empty seat detected by the congestion degree calculation unit 124 based on the image with respect to the image of the imaging space 30 captured by the monitoring camera 2. Show. In FIG. 11, the seat position is indicated by a circle and the seat position at which a person is seated is indicated by a square.

  FIG. 11A shows the case where the congestion degree is "vacant", for example, the congestion rate is less than "0.3". FIG. 11B shows the case where the congestion degree is “normal”, for example, the congestion rate is “0.3” or more and less than “0.6”. FIG. 11C shows the case where the congestion degree is "heavy", for example, the congestion rate is "0.6" or more.

  The congestion degree calculation unit 124 may store information on the estimated congestion degree as the congestion degree information 115 in the memory unit 11.

  Note that, as indicated by a symbol A in FIG. 11A, no seat may be allocated to the object detected in the area where the information on the seat position does not exist, and in this case, the object is used to calculate the congestion rate. You do not have to consider it. As a result, it is possible to prevent a store clerk, a moving customer, etc. from affecting the calculation of the congestion rate. In FIG. 11A, an object to which no seat is assigned is indicated by a double-lined square symbol.

[1-3-5] Description of Information Presentation Unit The information presentation unit 14 may present information on the degree of congestion estimated by the control unit 12 to the terminal device 6, for example.

  FIG. 12 is a diagram showing an example of presentation of information to the terminal device 6 by the information presentation unit 14. For example, as illustrated in FIG. 12, the information presentation unit 14 causes congestion degree information 115 which is an estimation result by the congestion degree calculation unit 124 to the display device 60 of the terminal device 6 in response to a request from the terminal device 6. The congestion degree of the store 3 based on may be presented.

  The information to be presented may be information on one store 3 or information on each of a plurality of stores 3 as shown in FIG. 12. Further, as shown in FIG. 12, the information presentation unit 14 may be able to browse the vacant seat status of the store 3. The vacant seat information may be, for example, information that generates a map based on the estimated seat position and specifies (for example, marks) a vacant seat or a seat where a person is seated on the map.

  In the terminal device 6 and the information presentation unit 14, for example, the terminal device 6 may function as a Web client, and the information presentation unit 14 may function as a Web server. In this case, the information presentation unit 14 may generate a Web page in response to a request from the terminal device 6 and may respond the generated Web page to the terminal device 6. The request may be, for example, an http (Hypertext Transfer Protocol) request for a predetermined URL (Uniform Resource Locator). Also, the web page may be generated by a markup language such as, for example, html (HyperText Markup Language).

  As described above, according to one embodiment, the server 4 detects the head of a person in the video of the monitoring camera 2 by the NN 13 and detects the position of the detected head by the static box determination unit 122 and the seat estimation. For example, deep learning may be performed by the unit 123.

  A place where a person's head frequently stays, for example, a stop can be estimated as the position of a seat. Therefore, by determining whether the head of the person detected from the image overlaps the estimated seat position, it can be determined whether there is a person in the seat and the congestion rate can be calculated.

  In addition, since the seat position can be estimated from the image of the monitoring camera 2 by the processing by the static box determination unit 122 and the seat estimation unit 123, the map of the seat position of the store 3 and the number of seats can not be acquired in advance. Also, the method according to one embodiment can be applied.

  For example, by performing a simple initial setting such as transmitting the video of the monitoring camera 2 to the server 4 in the store 3, the method according to the embodiment can be easily applied to the store 3, and the congestion rate of the seat with high accuracy It becomes possible to grasp In particular, when grasping the crowded situation of the seats of many stores 3, complicated initial setting and the like are not necessary, so that the photographing area of the monitoring camera 2 is set for each store as in the prior art. You can save a lot of costs.

  The method according to an embodiment can be applied or applied to, for example, the following situations.

Monitor and quantify the congestion status of the store 3 in real time.
・ Managers of facilities and stores 3 grasp congestion and use them in store strategy.
The user of the facility or store 3 checks the congestion status with the mobile application of the terminal device 6 and makes a decision on whether to go to the facility or store 3 or not.

[1-4] Operation Example Next, with reference to FIGS. 13 to 15, an operation example of the congestion degree estimation system 1 configured as described above will be described.

[1-4-1] Operation Example of Seat Position Estimation Phase First, an operation example of the seat position estimation phase will be described with reference to FIGS. 13 and 14.

  In the server 4, the information acquisition unit 121 of the control unit 12 acquires video data of the monitoring camera 2 installed in the store 3 and stores the video data as the video data 111 in the memory unit 11.

  As illustrated in FIG. 13, the NN 13 of the server 4 acquires frames at regular intervals (for example, every 5 minutes) from the video data 111 (step S1) and detects a box of the head of a person from the frame (step S2). ). The NN 13 may store the detected box as the detection information 112 in the memory unit 11.

  The static box determination unit 122 of the control unit 12 compares each box in the current (for example, latest) frame with a box in the past N (for example, N = 5) frames based on the detection information 112 (step S3) ).

  Note that comparisons between boxes include screening based on comparison of box size and / or distance between boxes, and comparison of overall scores of box similarity using color histograms, pixel differences, IoU, etc. May be included (see FIGS. 6 and 7).

  The static box determination unit 122 determines whether the box in the compared current frame is identical to any static box in the past frame (step S4).

  If the box is identical to any static box in the past frame (Yes in step S 4), the static box determination unit 122 adds the box to the data of the static box in the box information 113 ( Step S5), the process proceeds to step S8.

  On the other hand, if the box is not identical to any static box in the past frame (No in step S4), the non-static box identical to the box is added to the M (for example, M = 3) frame in the past N frames. It is determined whether it exists (step S6). When the condition of step S6 is not satisfied (No in step S6), the process proceeds to step S8.

  On the other hand, when the condition of step S6 is satisfied (Yes in step S6), the static box determination unit 122 manages these boxes and non-static boxes as new static boxes in the box information 113 (step S7). The process proceeds to step S8.

  In step S8, it is determined whether there is a static box without observation (ie, not detected) in the past M frames in a row. If there is no corresponding static box (No in step S8), the process proceeds to step S10.

  On the other hand, when the corresponding static box exists (Yes in step S8), the static box determination unit 122 writes the static box in a file and deletes the file from the memory (step S9). Then, the static box determination unit 122 determines whether all the boxes in the current frame have been compared (step S10).

  If all the boxes have not been compared (No in step S10), the process proceeds to step S3, and comparison is performed on the unexecuted boxes. On the other hand, when all the boxes have been compared (Yes in step S10), the static box determination unit 122 determines whether or not a predetermined period, for example, three days has elapsed since the storage of the box information 113 starts. Step S11).

  When the predetermined period has not elapsed (No in step S11), the process proceeds to step S1, and the above process is performed on a frame to be input next.

  On the other hand, when the predetermined period has elapsed (Yes in step S11), the process proceeds to step S12 in FIG. In this case, the NN 13 and the static box determination unit 122 may continue the processing based on the input video data 111 as a seat position estimation (update) phase.

In step S12 of FIG. 14, the seat estimation unit 123 sets representative values of each cluster, with each static box as an initial cluster. For example, the seat estimation unit 123 may calculate an average value (c _x , c _y ) of center coordinates of static boxes in a cluster, an average value “size” of box sizes, a number “n” of static boxes in a cluster, An array indicating the presence or absence of observation of the static box of, or the like may be set. In the arrangement, for example, “1” may be set for each frame if the static box is observed, and “0” if not.

Next, the seat estimation unit 123 calculates a distance index (d _ward ) between clusters for each pair of clusters (step S13). The distance index (d _ward ) may be calculated by the above equation (3). In calculating the distance index (d _ward ), a penalty obtained by the SUM (total value) of the AND results of the array of cluster pairs may be added (see FIG. 9).

The seat estimation unit 123 determines a cluster pair for which the distance index (d _ward ) is minimum (step S14), and determines whether the cluster pair satisfies the stop condition (step S15). The stop condition may be calculated by the above equation (4) (see FIG. 9).

  If the cluster pair does not satisfy the stop condition (No in step S15), the seat estimation unit 123 merges the cluster pair into a new cluster (step S16).

  The seat estimation unit 123 determines whether the number of remaining clusters is one (step S17). If the number of remaining clusters is not 1 (No in step S17), the process proceeds to step S13.

  If the cluster pair satisfies the stop condition in step S15 (Yes in step S15), or if the number of remaining clusters is 1 in step S17 (Yes in step S17), the process proceeds to step S18.

  In step S18, the seat estimation unit 123 deletes a cluster whose static box number in the cluster is equal to or less than a predetermined ratio (for example, 2%) of the static box number of other clusters.

  Then, the seat estimation unit 123 outputs the remaining cluster and the seat information 114, for example, stores it in the memory unit 11 (step S19), and the process of the seat position estimation phase ends.

[1-4-2] Operation Example of Congestion Situation Estimation Phase Next, an operation example of the congestion situation estimation phase will be described with reference to FIG.

  As illustrated in FIG. 15, the NN 13 acquires frames from the video data 111 at regular intervals (for example, every five minutes) (step S21), and detects a box of the head of a person from the frames (step S22). The NN 13 may store the detected box as the detection information 112 in the memory unit 11. Further, when the seat position estimation (update) phase is performed, the processing of steps S1 and S2 of FIG. 13 in the seat position estimation (update) phase may be replaced with the processing of steps S21 and S22 (steps S1 and S2 Processing may be omitted).

  The congestion degree calculation unit 124 of the control unit 12 calculates the distance between the box of the current (for example, the latest) frame in the detection information 112 and the seat position in the seat information 114 (step S23). The distance may be a distance between center coordinates or average coordinates of the box and the seat position.

  Next, the congestion degree calculation unit 124 assigns a box having a distance equal to or less than a threshold to the seat position in the order of closeness (step S24).

  Then, the congestion degree calculation unit 124 counts the number of seats allocated the box, in other words, where there are people (step S 25), and calculates “the number of seats where there are people / the total number of seats” as the congestion rate. (Step S26).

  The congestion degree calculation unit 124 determines whether the calculated congestion rate is less than “0.3” which is an example of the first threshold (step S27). If the congestion rate is less than "0.3" (Yes in step S27), the congestion degree calculating unit 124 estimates the congestion degree as "vacant" (step S28), and transmits the congestion degree information 115 to the memory unit 11. After storing, the process proceeds to step S32.

  When the congestion rate is “0.3” or more (No in step S27), the congestion degree calculation unit 124 determines whether the congestion rate is less than “0.6” which is an example of the second threshold ( Step S29). When the congestion rate is less than "0.6" (and "0.3" or more) (Yes in step S29), the congestion degree calculating unit 124 estimates the congestion degree as "normal" (step S30), and the congestion degree The information 115 is stored in the memory unit 11, and the process proceeds to step S32.

  On the other hand, when the congestion rate is "0.6" or more (No in step S29), the congestion degree calculating unit 124 estimates the congestion degree as "crowded" (step S31), and the congestion degree information 115 is stored in the memory unit In step S32, the process proceeds to step S32.

  In step S32, the information presentation unit 14 of the server 4 presents information on the degree of congestion of the store 3 based on the degree of congestion information 115. For example, the information presentation unit 14 may cause the terminal device 6 to display a Web page including information on the degree of congestion of the store 3 in response to a request from the terminal device 6.

  Thus, the process of the congestion state estimation phase ends.

[1-5] Hardware Configuration Example Next, a hardware configuration example of the server 4 according to an embodiment will be described with reference to FIG. Hereinafter, a hardware configuration example of the computer 10 will be described by taking the computer 10 as an example of the server 4 as an example. The terminal device 6 may have the same hardware configuration as that of the server 4.

  As illustrated in FIG. 16, the computer 10 illustratively includes a processor 10a, a memory 10b, a storage unit 10c, an IF (Interface) unit 10d, an I / O (Input / Output) unit 10e, and a reading unit 10f. Good.

  The processor 10a is an example of an arithmetic processing unit that performs various controls and calculations. The processor 10a may be communicably connected to each block in the computer 10 by a bus 10i. As the processor 10a, for example, an integrated circuit (IC) such as a CPU, an MPU, a DSP, an ASIC, or an FPGA may be used. CPU is an abbreviation for Central Processing Unit, and MPU is an abbreviation for Micro Processing Unit. DSP is an abbreviation for Digital Signal Processor, ASIC is an abbreviation for Application Specific Integrated Circuit, and FPGA is an abbreviation for Field-Programmable Gate Array.

  The memory 10 b is an example of hardware that stores information such as various data and programs. Examples of the memory 10 b include volatile memory such as RAM.

  The storage unit 10 c is an example of hardware that stores information such as various data and programs. Examples of the storage unit 10 c include various storage devices such as a magnetic disk device such as an HDD, a semiconductor drive device such as an SSD, and a non-volatile memory. Examples of the non-volatile memory include a flash memory, a storage class memory (SCM), and a read only memory (ROM).

  The memory unit 11 illustrated in FIG. 1 may be realized by, for example, at least one of the memory areas of the memory 10 b and the memory unit 10 c of the server 4.

  In addition, the storage unit 10 c may store a program 10 g for realizing all or part of various functions of the computer 10. The processor 10a can realize the function as the server 4 illustrated in FIG. 1 by developing the program 10g stored in the storage unit 10c in the memory 10b and executing the program 10g.

  The IF unit 10 d is an example of a communication interface that performs connection with the network 5 and control of communication. For example, the IF unit 10d may include an adapter conforming to a LAN or optical communication (for example, FC (Fibre Channel; Fiber Channel)). For example, the program 10g may be downloaded from the network 5 to the computer 10 via the communication interface and stored in the storage unit 10c.

  The I / O unit 10e includes one or both of an input unit such as a mouse, a keyboard or an operation button, and a touch panel display, a monitor such as an LCD (Liquid Crystal Display), an output unit such as a projector or a printer. Good.

  The reading unit 10 f is an example of a reader that reads information of data and programs recorded on the recording medium 10 h. The reading unit 10 f may include a connection terminal or device capable of connecting to or inserting the recording medium 10 h. Examples of the reading unit 10 f include an adapter conforming to a USB (Universal Serial Bus) or the like, a drive device for accessing a recording disk, a card reader for accessing a flash memory such as an SD card, and the like. The program 10g may be stored in the recording medium 10h, and the reading unit 10f may read the program 10g from the recording medium 10h and store the program 10g in the storage unit 10c.

  Examples of the recording medium 10 h include non-temporary recording media such as a magnetic / optical disk and a flash memory. Examples of the magnetic / optical disc include a flexible disc, a compact disc (CD), a digital versatile disc (DVD), a Blu-ray disc, and a holographic versatile disc (HVD). Examples of the flash memory include a USB memory and an SD card. In addition, as CD, CD-ROM, CD-R, CD-RW etc. are mentioned illustratively. Further, examples of the DVD include a DVD-ROM, a DVD-RAM, a DVD-R, a DVD-RW, a DVD + R, a DVD + RW, and the like.

  The hardware configuration of the computer 10 described above is an example. Therefore, hardware increase / decrease (for example, addition or deletion of arbitrary blocks), division, integration in any combination, addition or deletion of buses, etc. in the computer 10 in the server 4 may be appropriately performed. .

[2] Others The technology according to the above-described embodiment can be modified or changed as follows.

  For example, each functional block of the server 4 shown in FIG. 1 may be merged or divided in any combination. Further, each functional block of the control unit 12 shown in FIG. 1 may be merged or divided in any combination.

  Furthermore, the processor 10a of the computer 10 shown in FIG. 16 is not limited to a single processor or a single core processor, and may be a multiprocessor or a multi-core processor.

  In the above-described embodiment, the processing for calculating the degree of congestion from the input of the shop video has been described using a method for estimating the staying area based on the recognition result of the human head detection model for the video. However, the information of the estimated staying area is not limited to the use in the above-described aspect, and may be used for various analysis or estimation.

  In one embodiment, the seat estimation unit 123 performs hierarchical clustering, but the present invention is not limited to this. For example, when the number of seats provided in the store 3 is known in advance, the seat estimation unit 123 may perform non-hierarchical clustering so as to create the same number of clusters as the number of seats.

  The information on the number of seats may be acquired from the Internet or the like via the network 5 by the server 4 or a manager, for example, and stored in a database, for example, the memory unit 11. In this case, the information acquisition unit 121 may function as an example of an acquisition unit that acquires information on the number of seats in the imaging space 30 from the memory unit 11 when performing clustering by the seat estimation unit 123.

  The information on the number of seats of the store 3 can be easily acquired in many cases in the case of being posted on the website of the facility or store 3 or an introduction site on the Internet. Therefore, it is possible to accurately estimate the seat position based on the correct number of seats while suppressing the cost increase in the initial setting.

  Furthermore, in one embodiment, the server 4 estimates the seat position and the degree of congestion based on the video data from one monitoring camera 2, but the present invention is not limited to this.

  For example, when acquiring the video data from each of the plurality of monitoring cameras 2 provided in the store 3, the server 4 may estimate the seat position and the congestion degree for each of the monitoring cameras 2. Then, the server 4 may estimate the degree of congestion in the store 3 by, for example, averaging the estimated degree of congestion (or the degree of congestion) for each monitoring camera 2.

[3] Additional Notes The following additional notes will be disclosed regarding the above-described embodiment.

(Supplementary Note 1)
For the objects detected from each of the plurality of first images obtained by imaging the space, the plurality of the plurality of objects are detected based on position information of each detected object and a detection condition of each object in each of the plurality of first images. Group interrelated objects between first images,
Based on a result of the grouping, a staying area in which each object stays in the space is estimated
An estimation program that causes a computer to execute processing.

(Supplementary Note 2)
The congestion degree of the staying area is estimated based on position information of an object detected from a second image obtained by imaging the space, and information on the staying area in the space estimated.
The estimation program according to appendix 1, causing the computer to execute a process.

(Supplementary Note 3)
The predetermined number or more of objects determined to be present at the same position among a predetermined number or more of first images are managed in association with one static object.
Execute the process on the computer,
The grouping groups static objects that satisfy a condition regarding a distance among the plurality of first images as the correlated objects.
The estimation program according to Appendix 1 or 2.

(Supplementary Note 4)
The management manages, in association with the one static object, the predetermined number or more of objects determined that the image components of the objects match between the predetermined number or more of the first images.
A presumption program given in appendix 3.

(Supplementary Note 5)
The grouping is:
It is information which shows the detection condition of each object in each of a plurality of 1st pictures, and the object matched with the static object exists in each of a plurality of 1st pictures for every static object. Generate information indicating whether or not
Based on the generated information, excluding static objects whose corresponding objects are present in one first image from being grouped
Appendix 3. The estimation program according to Appendix 3 or 4.

(Supplementary Note 6)
The grouping performs hierarchical clustering,
The estimation program according to any one of appendices 1 to 5.

(Appendix 7)
Obtaining from the database the number of the stagnant areas in the space;
Execute the process on the computer,
The grouping performs non-hierarchical clustering so as to create a group of the acquired number of staying areas.
The estimation program according to any one of appendices 1 to 5.

(Supplementary Note 8)
The object is a human head,
The stagnant area is a seating area,
Detecting the object which is the head of a human body from each of the plurality of first images using a neural network that detects feature amounts of the image;
10. The estimation program according to any one of appendices 1 to 7, causing the computer to execute a process.

(Appendix 9)
For the objects detected from each of the plurality of first images obtained by imaging the space, the plurality of the plurality of objects are detected based on position information of each detected object and a detection condition of each object in each of the plurality of first images. A grouping unit that groups related objects among the first images;
An estimation unit configured to estimate a staying area in which each object stays in the space based on the grouping result;
An estimation system equipped with

(Supplementary Note 10)
A congestion degree estimation unit that estimates the congestion degree of the stagnant area based on position information of an object detected from a second image obtained by photographing the space and information on the stagnant area in the space estimated;
Appendix 9. The estimation system according to appendix 9.

(Supplementary Note 11)
A management unit that manages the predetermined number or more of objects determined to be present at the same position among the predetermined number or more of first images in association with one static object;
The grouping unit groups static objects satisfying a condition related to a distance as the interrelated objects among the plurality of first images.
The estimation system according to Supplementary Note 9 or Supplementary Note 10.

(Supplementary Note 12)
The management unit manages, in association with the one static object, the predetermined number or more of objects determined that the image components of the objects match between the predetermined number or more of the first images.
The estimation system according to appendix 11.

(Supplementary Note 13)
The grouping unit is
It is information which shows the detection condition of each object in each of a plurality of 1st pictures, and the object matched with the static object exists in each of a plurality of 1st pictures for every static object. Generate information indicating whether or not
Based on the generated information, excluding static objects whose corresponding objects are present in one first image from being grouped
The estimation system according to Supplementary Note 11 or 12.

(Supplementary Note 14)
The grouping unit performs the grouping by hierarchical clustering.
The estimation system according to any one of appendices 9 to 13.

(Supplementary Note 15)
An acquisition unit for acquiring from the database the number of the stagnant areas in the space;
The grouping unit performs the grouping by non-layered clustering so as to create a group of the acquired number of staying areas.
The estimation system according to any one of appendices 9 to 13.

(Supplementary Note 16)
The object is a human head,
The stagnant area is a seating area,
It is a neural network which detects the feature-value of a picture, provided with the above-mentioned neural network which detects the above-mentioned object which is a head of a human body from each of a plurality of above-mentioned 1st pictures, Estimated system described.

(Supplementary Note 17)
For the objects detected from each of the plurality of first images obtained by imaging the space, the plurality of the plurality of objects are detected based on position information of each detected object and a detection condition of each object in each of the plurality of first images. Group interrelated objects between first images,
An estimation method for estimating a staying area in which each object stays in the space based on the grouping result.

(Appendix 18)
The congestion degree of the staying area is estimated based on position information of an object detected from a second image obtained by imaging the space, and information on the staying area in the space estimated.
The estimation method according to appendix 17.

(Appendix 19)
Managing a predetermined number or more of objects determined to be present at the same position among a predetermined number or more of first images in association with one static object;
The grouping groups static objects that satisfy a condition regarding a distance among the plurality of first images as the correlated objects.
The estimation method as described in Supplementary Note 17 or 18.

(Supplementary Note 20)
The management manages, in association with the one static object, the predetermined number or more of objects determined that the image components of the objects match between the predetermined number or more of the first images.
An estimation method according to appendix 19.

DESCRIPTION OF SYMBOLS 1 crowded degree estimation system 2 surveillance camera 3 store 4 server 5 network 6 terminal device 10 computer 11 memory section 111 video data 112 detected information 113 box information 114 seat information 115 crowded degree information 12 control section 121 information acquisition section 122 static box determination Part 123 Seat estimation part 124 Congestion degree calculation part 13 NN (neural network)
14 Information presentation department

Claims (10)

For the objects detected from each of the plurality of first images obtained by imaging the space, the plurality of the plurality of objects are detected based on position information of each detected object and a detection condition of each object in each of the plurality of first images. Group interrelated objects between first images,
Based on a result of the grouping, a staying area in which each object stays in the space is estimated
An estimation program that causes a computer to execute processing. The congestion degree of the staying area is estimated based on position information of an object detected from a second image obtained by imaging the space, and information on the staying area in the space estimated.
The estimation program according to claim 1, which causes the computer to execute a process. The predetermined number or more of objects determined to be present at the same position among a predetermined number or more of first images are managed in association with one static object.
Execute the process on the computer,
The grouping groups static objects that satisfy a condition regarding a distance among the plurality of first images as the correlated objects.
The estimation program of Claim 1 or Claim 2. The management manages, in association with the one static object, the predetermined number or more of objects determined that the image components of the objects match between the predetermined number or more of the first images.
The estimation program according to claim 3. The grouping is:
It is information which shows the detection condition of each object in each of a plurality of 1st pictures, and the object matched with the static object exists in each of a plurality of 1st pictures for every static object. Generate information indicating whether or not
Based on the generated information, excluding static objects whose corresponding objects are present in one first image from being grouped
The estimation program of Claim 3 or Claim 4. The grouping performs hierarchical clustering,
The estimation program of any one of Claims 1-5. Obtaining from the database the number of the stagnant areas in the space;
Execute the process on the computer,
The grouping performs non-hierarchical clustering so as to create a group of the acquired number of staying areas.
The estimation program of any one of Claims 1-5. The object is a human head,
The stagnant area is a seating area,
Detecting the object which is the head of a human body from each of the plurality of first images using a neural network that detects feature amounts of the image;
The estimation program according to any one of claims 1 to 7, which causes the computer to execute a process. For the objects detected from each of the plurality of first images obtained by imaging the space, the plurality of the plurality of objects are detected based on position information of each detected object and a detection condition of each object in each of the plurality of first images. A grouping unit that groups related objects among the first images;
An estimation unit configured to estimate a staying area in which each object stays in the space based on the grouping result;
An estimation system equipped with For the objects detected from each of the plurality of first images obtained by imaging the space, the plurality of the plurality of objects are detected based on position information of each detected object and a detection condition of each object in each of the plurality of first images. Group interrelated objects between first images,
An estimation method for estimating a staying area in which each object stays in the space based on the grouping result.