JP7069667B2 - Estimating program, estimation system, and estimation method - Google Patents

Description

The present invention relates to an estimation program, an estimation system, and an estimation method.

As a method of grasping the congestion status of a store, for example, a method of grasping the waiting status of a user's service by analyzing the usage status of a ticket issuing machine by a user in a store such as a meal ticket system is known.

With the above method, it is possible to grasp the congestion status of the store, but it is difficult to grasp the congestion status of the seats, for example, the seating status or the vacant seat status of the seats.

As a method for grasping the congestion status of seats, a method of installing a pressure sensor in each seat or an infrared sensor in a store is known. In the method using the pressure sensor, for example, by attaching one pressure sensor to each seat, the seating status in each seat can be detected and the congestion status of the store can be grasped. Further, in the method using an infrared sensor, for example, an infrared sensor installed in a store can detect infrared rays from a human body to grasp the position and number of people.

However, with these methods, for example, the introduction cost such as installing one pressure sensor for each seat or setting the seat area where it is difficult to detect infrared rays from the human body in the analysis software, and for analysis Operation costs such as operation and maintenance of the software may be incurred.

Therefore, for example, in the case of a store having 100 or more seats such as a restaurant, the above-mentioned costs may be incurred for each store, and it is difficult to easily introduce the above-mentioned costs.

By the way, there is also known a method of grasping a congestion situation in a store by analyzing an image taken by a surveillance camera installed in the store.

Japanese Unexamined Patent Publication No. 2010-86300 JP-A-2017-156965

However, in the method of analyzing the image of the surveillance camera, the angle of the surveillance camera (for example, the angle of the shooting direction in the store and the angle of view), the resolution of the image, the shooting conditions, and the like have a great influence on the recognition accuracy. Further, for example, it is possible to specify and monitor the area to be monitored for the congestion situation, or to monitor by comparing the captured image with the vacant seat image. However, in order to perform these monitoring, various conditions such as store seat position information and surveillance camera settings / installation conditions must be set in the software.

As the information on the seat position, the seat layout diagram may be mentioned in the above example. The seat position (area) is an example of a "retention area" in which an object such as a person stays in the space.

In the case of a company having many stores, such software settings are made for each store, which may result in a large cost.

In one aspect, it is an object of the present invention to estimate the congestion status of a stagnant area of an object in space at low cost or with high accuracy.

In one aspect, the estimation program may cause the computer to perform the following processing. The processing is based on the position information of each detected object for the object detected from each of the plurality of first images obtained by photographing the space, and the detection status of each object in each of the plurality of first images. , Objects related to each other may be grouped among the plurality of first images. Further, in the process, the residence region where each object stays in the space may be estimated based on the result of the grouping. Further, in the process, the predetermined number or more of the objects determined to match the image components of the objects may be managed in association with one object among the predetermined number or more of the first images. Further, the grouping may include a process of grouping one object satisfying the condition regarding the distance between the plurality of first images as the mutually related object. Further, the grouping is information indicating the detection status of each object in each of the plurality of first images, and corresponds to the one object in each of the plurality of first images for each of the one objects. Information indicating whether or not the attached object exists is generated, and based on the generated information, one object whose corresponding object exists in one first image is excluded from the grouping target. Processing may be included.

In one aspect, the congestion status of the stagnant area of an object in space can be estimated at low cost or with high accuracy.

It is a block diagram which shows the configuration example of the congestion degree estimation system which concerns on one Embodiment. It is a figure which shows an example of the operation phase which concerns on one Embodiment. It is a figure which shows the example of the detection of the object by the server which concerns on one Embodiment. It is a figure which shows an example of the detection information. It is a figure which shows an example of a static box and a non-static box. It is a figure which shows an example of the similarity determination processing of the coordinates of a box. It is a figure which shows an example of the similarity determination process of a box itself. It is a figure which shows an example of the estimation process of a seat position. It is a figure which shows an example of the distance index of the layered clustering. It is a figure which shows an example of the estimation procedure of a seat position. (A) to (c) are diagrams showing an example of the estimated degree of congestion. It is a figure which shows an example of the presentation mode of the degree of congestion. It is a flowchart which shows the operation example of the seat position estimation phase which concerns on one Embodiment. It is a flowchart which shows the operation example of the seat position estimation phase which concerns on one Embodiment. It is a flowchart which shows the operation example of the congestion situation estimation phase which concerns on one Embodiment. It is a figure which shows the hardware configuration example of the computer which concerns on one Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the embodiments described below are merely examples, and there is no intention of excluding various modifications and applications of techniques not specified below. For example, the present embodiment can be variously modified and implemented without departing from the spirit of the present embodiment. In the drawings used in the following embodiments, the parts with the same reference numerals represent the same or similar parts unless otherwise specified.

[1] Embodiment As described above, in order to analyze the image taken by the surveillance camera and grasp the congestion status of the seats, the information of the seat layout and the setting / installation conditions of the surveillance camera are set in the software. As the number of stores increases, the cost can increase.

Therefore, in one embodiment, an estimation system capable of estimating the seat position based on the image taken by the surveillance camera will be described.

For example, the estimation system according to the embodiment may perform the following processing.

-For an object detected from each of a plurality of first images obtained by photographing a space, a plurality of first images are based on the position information of each detected object and the detection status of each object in each of the plurality of first images. 1 Group objects that are related to each other between images.

-Based on the grouping results, estimate the retention area where each object stays in space.

By the above processing, the estimation system according to the embodiment can estimate the retention area where each object stays, for example, the seat position where a person is present, based on a plurality of first images. The staying area means a specific area where a person stays, and may include not only a sitting area such as a chair or a cushion but also a standing area (space). As an example of a standing area, a standing restaurant (or called "tachigui") restaurant or a facility where a customer is provided with a service while standing is assigned to the customer by the store. The seat area can be mentioned.

By estimating the retention area in the space based on the first image obtained by photographing the space, for example, it is possible to easily estimate the congestion state of the retention area based on the image obtained by photographing the space and the estimated retention area. can.

For example, when estimating the seat congestion status based on the image of the surveillance camera, it is possible to eliminate the need for processing such as setting (or updating) conditions such as seat position information in the software, and realizing cost reduction. can.

In addition, when grasping the seat congestion situation using the information of the seat layout prepared in advance, the captured image is corrected based on the setting and installation conditions of the surveillance camera, and then the seat layout is used. It may not be possible to accurately estimate the congestion situation because it is compared with. On the other hand, in the estimation system according to one embodiment, processing using captured images is performed in both the estimation of the retention area and the estimation of the congestion status. Therefore, the influence of the setting / installation conditions of the surveillance camera can be reduced, and the congestion status can be estimated with high accuracy.

From the above, the estimation system according to the embodiment can estimate, for example, the usage status of the retention area in the space at low cost or with high accuracy based on the estimated information of the retention area.

It is highly possible that surveillance cameras that capture space are introduced and used in stores and the like for the purpose of crime prevention and the like. For this reason, existing equipment can be used by using a surveillance camera to estimate the seat congestion status, so it is better to newly introduce a pressure sensor, infrared sensor, ticketing machine, etc. to estimate the seat congestion status. However, the cost can be suppressed.

[1-1] Configuration Example of One Embodiment Hereinafter, a configuration example of one embodiment will be described. FIG. 1 is a block diagram showing a configuration example of a congestion degree estimation system 1 according to an embodiment. The congestion degree estimation system 1 is a system for estimating the congestion degree of the residence area in the facility to be estimated, and is an example of the above-mentioned estimation system. As shown in FIG. 1, the congestion degree estimation system 1 may optionally include one or more (plurality in the example of FIG. 1) surveillance cameras 2 and a server 4.

The surveillance camera 2 and the server 4 may be connected to each other so as to be able to communicate with each other by, for example, the network 5.

The network 5 may be, for example, a LAN (Local Area Network) or a WAN (Wide Area Network), or at least one of an Internet and an intranet including a combination thereof. Further, the network 5 may include a virtual network such as a VPN (Virtual Private Network). The network 5 may be formed by one or both of a wired network and a wireless network.

The surveillance camera 2 may shoot a space in a shooting direction and acquire a plurality of images (may be referred to as "frames") arranged in a time series such as an image series, for example, a moving image.

The surveillance camera 2 may be installed in, for example, a store 3, and as an example, the surveillance camera 2 is arranged so as to include customer seats (preferably all customer seats) existing inside and outside the store 3 in the shooting range. It's okay.

In order to include all the seats inside and outside the store 3 in the shooting range, for example, a plurality of surveillance cameras 2 are installed at positions that complement each other's blind spots (or some shooting ranges overlap). Alternatively, one or more movable surveillance cameras 2 may be installed. Alternatively, a combination of these may be adopted.

The surveillance camera 2 may transmit the acquired video to the server 4 via the network 5. For example, the surveillance camera 2 may store the acquired video in a recorder or the like (not shown) and transmit the data in the recorder to the server 4 at a predetermined timing. As the predetermined timing, various conditions such as the arrival of a predetermined time, the passage of a predetermined time, the storage capacity in the recorder, the number of storage frames, and the like may be used. Alternatively, the surveillance camera 2 may transmit the captured video to the server 4 without going through the recorder. When not via the recorder, for example, the surveillance camera 2 may transmit an image every one to several frames (in substantially real time).

Examples of the surveillance camera 2 include a box-type camera, a dome camera, a network camera, and the like. Examples of the network camera include an IP (Internet Protocol) camera and the like. Further, in a store 3 where the lighting inside or outside the store is dark, a night-vision camera such as an infrared camera may be used as the surveillance camera 2. As the surveillance camera 2, for example, cameras for various purposes such as a security camera, a surveillance camera, a street camera, and a fixed point camera may be used.

The server 4 estimates the seat position in the store 3 based on the image taken by the surveillance camera 2, and estimates the degree of congestion of the seat in the store 3 based on the estimated seat position and the image taken by the surveillance camera 2. You can do it. The details of the server 4 will be described later.

Information on the degree of seat congestion estimated by the server 4 may be provided to the terminal device 6 as shown in FIG. 1, for example. As an example, the server 4 may have a Web server function (for example, an information presentation unit 14 described later), and the Web server function allows a Web page indicating the degree of congestion of the store 3 to be terminal via the network 5. It may be displayed on the device 6.

The terminal device 6 is a computer that receives information on the degree of congestion of the store 3 estimated by the server 4. For example, the terminal device 6 may be a computer owned by a candidate for use of the store 3 who is interested in the congestion situation of the seats of the store 3. Examples of the terminal device 6 include various information processing devices such as a personal computer (PC) such as a desktop, laptop or mobile, a tablet, a smartphone, and a mobile phone.

The terminal device 6 transmits various communications such as transmission of a request for acquiring the congestion status of the seats of the store 3 and reception of a response regarding the estimation result of the congestion status with the server 4 via the network 5, for example. You may go.

As illustrated in FIG. 1, when the terminal device 6 is a PC, tablet, smart phone, mobile phone or the like that performs wireless communication, the connection with the network 5 is performed via the mobile network via the base station 7. You can do it.

Illustratively, the terminal device 6 may include means for inputting information (operation request) from the user, means for outputting information to the user, means for communicating with the server 4, and the like.

[1-2] Operation phase of the server 4 Next, the operation phase of the server 4 will be described.

As described above, the server 4 may estimate the seat position based on the image taken by the surveillance camera 2. Then, the server 4 may estimate the congestion status of the seat (for example, the latest) based on the estimated seat position and the (for example, the latest) image taken by the surveillance camera 2.

Various image recognition techniques may be used for estimating the seat position based on the image, but in one embodiment, by way of example, deep learning using a neural network (NN) is used. A detection model shall be used.

Although it depends on the service content and time zone provided by the store 3, it is considered that the period during which the customer uses the seat in the store 3 is a dozen minutes to several tens of minutes (or one hour or more). The surveillance camera 2 can capture image data of at least one frame (1 FPS; Frame Per Second) or more per second, but all frames are used to detect changes in the seat such as seating, leaving, and leaving the customer. It is inefficient to analyze the data of.

Therefore, in machine learning, the image data of a part of the captured image data may be used. For example, in one embodiment, frames acquired from the video at a predetermined sampling interval such as a 5-minute interval may be input to the neural network.

Further, in one embodiment, an estimation period of several hours to several days may be provided for estimating the seat position. In the following description, it is assumed that an estimated period (predetermined period) of 3 days is provided as an example.

From the above points, the method according to the embodiment may be carried out by dividing it into the phases illustrated in FIG. For example, as shown in FIG. 2, the store 3 in which the surveillance camera 2 is operated is set to transmit the image of the surveillance camera 2 to the server 4, and the seat position is set for 3 days as a predetermined period. An estimation phase may be provided.

After the seat position is estimated by the seat position estimation phase, the congestion situation estimation phase by the server 4 may be started next. In the congestion situation estimation phase, the server 4 may estimate the congestion situation at a predetermined timing based on the estimated seat position and the image data sent from the surveillance camera 2.

As described above, the seat position estimation phase may be positioned as the initial setting phase of the method according to the embodiment, and the congestion situation estimation phase may be positioned as the normal operation phase of the method according to the embodiment.

In the store 3, for example, there is a possibility that the layout of the store may be changed or the seats may be moved by a customer, an employee, or the like. Therefore, after the end of the seat position estimation phase, a seat position estimation (update) phase for updating the estimated seat position may be carried out also in the congestion situation estimation phase. Since the seat position estimation (update) phase can be performed in the same process as the seat position estimation phase, they are not distinguished in the following description and are simply referred to as the seat position estimation phase.

[1-3] Configuration Example of Server 4 Next, a configuration example of the server 4 will be described. The server 4 is an example of one or more computers installed in a store 3, a company having a store 3, a data center, or the like. Examples of the server 4 include various physical server devices and / or virtual server devices. At least a part of the functions of the server 4 may be realized by using, for example, resources, frameworks, applications and the like provided by the cloud service. Further, at least a part of the functions of the server 4 may be distributed or redundantly arranged in a plurality of computers.

As shown in FIG. 1, the server 4 may optionally include a memory unit 11, a control unit 12, an NN 13, and an information presentation unit 14.

The memory unit 11 stores various information used for the processing of the server 4. The information stored in the memory unit 11 will be described later in the description of the function of the server 4. The memory unit 11 includes a memory, for example, a volatile memory such as a RAM (Random Access Memory), and a storage unit, for example, a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). Both can be mentioned.

The control unit 12 controls the estimation of the seat position and the estimation of the congestion situation. As shown in FIG. 1, the control unit 12 may optionally include an information acquisition unit 121, a static box determination unit 122, a seat estimation unit 123, and a congestion degree calculation unit 124.

The information acquisition unit 121 may receive the video data transmitted from the surveillance camera 2 (store 3) and store it in the memory unit 11 as the video data 111. In one embodiment, the control unit 12 and the NN 13 may use image data (frames) arranged in time series taken at 5-minute intervals. The image data arranged in time series is an example of a plurality of first images.

For example, the information acquisition unit 121 may store the information obtained by extracting frames from the received video data at intervals of 5 minutes in the memory unit 11 as video data 111. Alternatively, the frame group extracted from the captured video data at 5-minute intervals in the store 3 or the surveillance camera 2 may be transmitted to the server 4 as video data.

The video data 111 may include image data of at least N (N is an integer; for example, “5”) frame. From the viewpoint of saving the storage capacity of the memory unit 11, the information acquisition unit 121 sets an upper limit on the data size or the number of frames of the video data 111, and for the frames exceeding the upper limit, the shooting date and time is in the video data 111 in the order of the past frames. You may delete the frame.

The static box determination unit 122 and the seat estimation unit 123, in cooperation with the NN 13 in the seat position estimation phase, estimate the seat position of the store 3 from the video data 111. In other words, the static box determination unit 122 and the seat estimation unit 123 are examples of the following grouping unit and estimation unit.

The grouping unit is based on the position information of each detected object for the object detected from each of the plurality of first images obtained by photographing the space, and the detection status of each object in each of the plurality of first images. , Objects that are related to each other may be grouped among a plurality of first images. The estimation unit may estimate the retention region in which each object stays in the space based on the result of grouping.

The congestion degree calculation unit 124 calculates the congestion degree of the store 3 based on the estimated seat position. In other words, the congestion degree calculation unit 124 estimates the congestion degree of the retention region based on the position information of the object detected from the second image of the space and the information of the retention region in the estimated space. This is an example of the congestion degree estimation unit.

The information presentation unit 14 may have, for example, a function of a Web server or a DB (Database) server, and the congestion degree calculation unit 124 calculates the congestion degree for the terminal device 6 in response to a request from the terminal device 6. Information on the degree may be presented.

Hereinafter, an example of the function and operation of the server 4 will be described.

[1-3-1] Explanation of NN First, NN13 will be described. The NN 13 detects an object from each of the plurality of first images taken by the surveillance camera 2 in the space. For example, the NN 13 may detect an object based on the video data 111 including a plurality of image data at 5-minute intervals stored in the memory unit 11.

The NN 13 may be a system in which machine learning is performed in advance so as to detect an object to be detected from image data. For example, a detection model for detecting the head of a person from image data may be applied to the NN 13. The head of a person is an example of an object to be detected.

FIG. 3 shows an example of detecting an object by NN13. As illustrated in FIG. 3, the NN 13 may calculate the position and score of the head of a person from the input image data. Note that FIG. 3 shows a state in which the position and score of the head detected by the NN 13 based on the image are applied to the image of the shooting space 30 taken by the surveillance camera 2. The image of the photographing space 30 may have a coordinate system in which the upper left of the image is (0,0) and the lower right of the image is (x, y). x may be a value indicating the size of the width of the image, and y may be a value indicating the size of the height of the image in terms of the number of pixels.

The position of the head may be specified, for example, as a rectangular area surrounding the head. Hereinafter, the rectangular shape surrounding the head may be referred to as a "box". The shape of the box is not limited to a rectangular shape, and may be a circular shape, an elliptical shape, a polygonal shape, or the like.

The score is an example of information indicating the certainty (likelihood) that the detected region is the head of the human body, with the maximum value being "1.000". The example of FIG. 3 is a frame taken at 12:00 (“12:00”).

By setting the object to be detected as the head of a person (human body), it is possible to capture a portion of the human body that easily moves to the surveillance camera 2, and the detection accuracy can be improved. Further, by detecting an object by deep learning, it is possible to detect a person's head in various postures or states in addition to the front face.

For example, in NN13, the head of a person has various postures such as the back surface (occipital region; see reference numeral A in FIG. 3), the upper surface (top of the head (downward posture); see reference numeral B), and the side surface (profile; see reference numeral C). Even so, the head can be detected accurately.

In addition, the NN13 is a state in which the head is hidden by another object, for example, a state in which the head is blocked by a chair (see reference numeral D), a state in which a cap is worn (see reference numeral E), or an obstacle or an obstacle in a crowded place. The head can be detected accurately even when it is blocked by an object.

The NN 13 may store the detected head position and score as the detection information 112 in the memory unit 11. The detection information 112 may be generated for each frame, or the frame identification information may be associated with the object in the detection information 112. The detection information 112 may be a file in various formats, and as an example, it may be a file in a format such as CSV (Comma-Separated Values).

FIG. 4 is an example of the detection information 112. As shown in FIG. 4, the detection information 112 may optionally include information of "No.", "x1", "y1", "x2", "y2", and "score". The example of FIG. 4 shows information on an object detected in one frame (image).

"No." is information for identifying the detected object. "No." may further include information for identifying the frame. “X1” and “y1” indicate the coordinates (x1, y1) of the upper left vertex of the detected head box, and “x2” and “y2” indicate the coordinates of the lower right vertex of the detected head box. Coordinates (x2, y2) may be indicated. The score may indicate the likelihood of the box represented by (x1, y1), (x2, y2). The box set in the detection information 112 by the NN 13 may be limited to a box having a score of "0.500" or more.

[1-3-2] Description of the static box determination unit The static box determination unit 122 sits on the seat based on the detection information 112, which is the detection result of the head of a person, as a preprocessing for estimating the seat position. You may determine the position of the head of the person who is doing it.

At least a part of the following processing by the static box determination unit 122 may be executed by deep learning using NN. For example, the static box determination unit 122 may be provided with an NN, or the NN 13 may be configured to further execute the following processing.

For example, the static box determination unit 122 compares the detection result of the head related to a certain frame with the detection result of the head related to the past several frames, and extracts the box remaining at the same position. Boxes that remain in the same position may include, for example, a person (head) seated in a seat at store 3. In other words, the static box determination unit 122 detects a stagnant object, including an object staying in the stagnant region, for each captured image from the captured image of the surveillance camera 2.

For the box identity determination between frames, the similarity determination of the box coordinates (for example, size and position) and the similarity determination of the box itself may be used.

Hereinafter, an example of a method of acquiring data on the head position of the “sitting person” by the static box determination unit 122 will be described. In the following description, the box at the head position of the sitting person is referred to as a "static box".

The static box determination unit 122 may manage the data of the detection result of the head related to the past frame by dividing it into the following two types of data.

(A) Static box data.

(B) Data of non-static boxes other than static boxes in the past N frames. It is assumed that N = 5.

The static box determination unit 122 may store, for example, the data of the static box and the non-static box in the memory unit 11 as the box information 113. The box information 113 may include, for example, as the data of the static box, the identification information of the box determined to correspond to the static box in a plurality of frames for each static box. Further, the box information 113 may include identification information of a box that is not a static box in a plurality of frames as data of a non-static box.

As described above, the static box determination unit 122 is an example of a management unit that manages a predetermined number or more of objects determined to exist at the same position between a predetermined number or more of the first images in association with one static object. be.

For example, the static box determination unit 122 performs the following processes (i) to (iii) on the image data at each time based on the position of the detected object and the detection status of each object in each frame. You may do it.

In the following description, the "frame of the current time" may be read as the frame to be determined. That is, the static box determination unit 122 may compare each object detected in the frame to be determined with N frames earlier than the frame. Further, when the static box determination unit 122 makes a determination, the next frame (after 5 minutes) may be set as the frame to be determined and compared with the N frames past the frame.

(I) The static box determination unit 122 compares the box detected in the frame of the current time with the static box of the frame of the past time, and adds the box determined to be the same to the data of the static box.

The static box determination unit 122 may acquire information on the box detected in each time frame from the detection information 112. The method for determining whether or not the static boxes are the same between frames will be described later.

(Ii) The static box determination unit 122 sets the boxes detected in the current time frame that are not determined to be static boxes as the data of non-static boxes other than the static boxes in the past N frames. Compare. Then, when the static box determination unit 122 detects a combination of boxes that satisfy the conditions of the static box, the static box determination unit 122 adds the box to the data of the new static box.

For example, if the M frame in the N (N = 5) frame contains boxes that can be determined to be identical to each other, the static box determination unit 122 newly sets a group of these boxes as a static box. You may judge. Note that M is an integer less than N and may be, for example, "3".

(Iii) The static box determination unit 122 writes the data of the box to the file of the storage unit, for example, for the static box that has not been observed (detected) continuously in M frames, and from the data of the static box in the memory. You may delete it. Examples of such a situation include an event in which a person who has been seated has left the seat.

As described above, the memory unit 11 may be one or both of the memory unit and the storage unit. Therefore, the process of (iii) above stores the data of the active static box detected in the M frame in the past N frames in the memory, and the data of the inactive static box not detected in the past M frame. Means to save to the storage unit.

FIG. 5 is a diagram showing an example of a static box and a non-static box in three frames of time “10:00”, “10:05”, and “10:10”. As shown in FIG. 5, for example, the box shown by the shaded up to the right and the box shown by the shaded down to the right, which are determined to be the same in the three frames, are static boxes (“static boxes”, respectively. 1 ”and“ static box 2 ”) are determined. On the other hand, a box with a white background in which the frames determined to be the same in the past 5 frames is less than 3 frames is determined to be a non-static box (“non-static box”).

Next, a method for determining the identity of boxes between frames will be described.

(Similarity judgment of box coordinates)
First, a method for determining the similarity of the box coordinates will be described. The static box determination unit 122 may compare the coordinates of the boxes between the frames, and if the comparison result exceeds the threshold value, determine both as "different boxes". The coordinates to be compared may include at least one of magnitude and position.

The static box determination unit 122 determines a pair of boxes whose sizes are significantly different and / or whose positions (in other words, distances) are significantly different depending on the similarity determination of the box coordinates. May be excluded from. As a result, the amount of calculation in the subsequent similarity determination of the box itself can be reduced.

FIG. 6 is a diagram showing an example of the similarity determination process of the coordinates of the box. As illustrated in FIG. 6, the static box determination unit 122 may execute at least one of the comparison of the size of the box between the frames and the comparison of the position of the box between the frames. In the example of FIG. 6, the current frame (f) and the past frame (f-1) are used as the two frames to be compared.

For example, as shown in the upper left of FIG. 6, the static box determination unit 122 determines whether or not the size ratio of the two boxes to be compared is a threshold value, for example, “1.3” or less. The box size ratio may be obtained by, for example, (larger box size) / (smaller box size). The box size may be, for example, the average of the width and height of the box and may be determined by (wise + height) / 2.

Further, for example, as shown in the upper right of FIG. 6, the static box determination unit 122 determines whether or not the distance between the two boxes to be compared is a threshold value, for example, “average box size × 1.5” or less. judge. The average box size may be the average box size of the two boxes or static boxes to be compared. Alternatively, the average box size may be the average box size of the boxes detected in the current frame, the past N frames, or all the frames analyzed so far.

When at least one of the box size ratio and the distance between the boxes is larger than the corresponding threshold value, the static box determination unit 122 determines the two boxes of the static box according to the above two determinations. It may be excluded from the judgment target.

(Similarity judgment of the box itself)
Next, the method of determining the similarity of the box itself will be described. The static box determination unit 122 may, for example, score the similarity of the cropped images of the two boxes to be compared and the overlap between the boxes, and calculate the total score of these scores. Then, if the total score is equal to or higher than the threshold value (for example, “1”), the static box determination unit 122 may determine that the two boxes are the same.

The crop image is an image obtained by cutting out an object (head in one embodiment) from image data. Obtaining the similarity score of the crop image may include, for example, obtaining the similarity score of the color histogram and obtaining the similarity score of the pixel difference.

FIG. 7 is a diagram showing an example of the similarity determination process of the box itself. As illustrated in FIG. 7, the static box determination unit 122 may calculate the following score.

(I) Color histogram similarity score: “sim_color”

Examples of the color histogram include a histogram of RGB (Red Green Blue) values.

(II) Similarity score by pixel difference: “sim_pixel”

Similarity due to pixel difference may be obtained by, for example, a method such as NRMSE (Normalized Root Mean-Squared Error). For the method of calculating the pixel difference by NRMSE, for example, [http: // scikit. image. org / docks / dev / api / skimage. measurement. html # skimage. measurement. compare_nrmse].

(III) Degree of overlap of boxes: IoU (Intersection over Union)

IoU is information indicating the degree of overlap of two boxes (in other words, regions). For example, as shown in FIG. 7, IoU indicates the ratio of the overlapping overlapping region (AND region) to the combined region (OR region) in which the two regions are overlapped when the two regions are overlapped in the coordinate system. ..

The scores of (I) to (III) may be calculated by a neural network.

The static box determination unit 122 calculates "sim_color * sim_pixel + IoU" as a total score based on the calculated scores (I) to (III). The overall score is an example of an index for determining whether or not the image components of an object match.

Then, the static box determination unit 122 determines whether or not the total score is equal to or higher than the threshold value (for example, “1”). When the total score is equal to or higher than the threshold value, the static box determination unit 122 may determine that the two boxes to be compared are the same box between frames.

The static box determination unit 122 may determine the similarity as follows in the comparison between the box of the current frame and the static box of the past frame in the process of (i) above. For example, the static box determination unit 122 calculates the total score of similarity between the box to be compared in the current frame and the static box to be compared in each of the plurality of past frames. good. Then, the static box determination unit 122 may calculate the average value of the calculated total scores of the similarity and determine whether or not the average value is larger than the threshold value (for example, “1”). If the mean value is greater than the threshold, the box to be compared may be added to the static box.

Further, the static box determination unit 122 may determine the similarity as follows in the comparison between the box of the current frame and the non-static box of the past frame in the process of (ii) above. .. For example, the static box determination unit 122 may calculate the similarity between the box to be compared in the current frame and the non-static box in each of the plurality of past frames. Then, in the static box determination unit 122, is the total score of similarity calculated in the M (for example, "3") frame in the past N (for example, "5") frame larger than the threshold value (for example, "1")? It may be determined whether or not. When the total score is larger than the threshold value in M frames or more, the box to be compared and the non-static box compared may be newly determined as a static box.

As described above, the static box determination unit 122 can reduce the probability of erroneous detection of the seat position by determining the similarity of the boxes themselves.

For example, a position in front of a cash register or a toilet is a place where customers may stay other than a seat. In such locations, boxes may be detected consecutively between frames at 5-minute intervals.

On the other hand, the static box determination unit 122 determines the similarity of the boxes themselves, so that, for example, when the boxes to be compared are the heads of people different from each other, the total score becomes equal to or less than the threshold value, and the static box determination unit 122 is static. Not judged as a box. As a result, even if the heads of different persons are detected over a plurality of frames in a place other than the seat such as in front of the cash register or in front of the toilet, the possibility that the place is detected as the seat position is reduced or It can be excluded and the seat position can be estimated accurately.

[1-3-3] Explanation of the seat estimation unit The seat estimation unit 123 estimates the seat position based on the information of the "sitting person" accumulated as the box information 113, and the seat estimation unit 123 uses the estimated seat position information as the seat. It is stored in the memory unit 11 as information 114.

In addition, at least a part of the following processing by the seat estimation unit 123 may be executed by deep learning using NN. For example, the seat estimation unit 123 may be provided with an NN, or the NN 13 may be further configured to perform the following processing.

For example, the seat estimation unit 123 uses the box information 113 that stores the position data (static box) of the "sitting person" determined by the static box determination unit 122 for a certain period (about 3 days in one embodiment). Then, the seat position and the number of seats may be estimated.

As an example, the seat estimation unit 123 may perform clustering based on the information of the static box, treat the estimated number of clusters as the number of seats, and treat the position of each cluster as the position information of each seat. The static box information includes a position, a size, and an observed time (frame). Further, the position of each cluster may be calculated based on the average value of the positions of the static boxes included in each cluster.

In one embodiment, the number of seats in the store 3 is not acquired or set for the sake of simplification of the initial setting. In this case, the clustering algorithm may be based on hierarchical clustering.

For example, as shown in FIG. 8, the seat estimation unit 123 sets the initial state in which all the observed static boxes are separate clusters (see “static boxes”), and sets the clusters that are close to each other as the same cluster. May be merged (see “clustering result”). By setting a threshold value for the distance to be merged, clusters separated by a distance remain without being merged. Therefore, the seat estimation unit 123 may estimate each remaining cluster as the final seat data (“estimated seat positions”). "reference).

Here, the cluster is a group of observed static boxes, and is an example of objects that are related to each other among a plurality of first images. Static boxes in the same cluster are considered to be observations of different people sitting in the same seat. That is, in one embodiment, the seat position is estimated by regarding the position of the finally remaining cluster as the position where the object stays and the position where the object stays as the seat position.

Hereinafter, a distance index for performing hierarchical clustering will be described with reference to FIG. 9. FIG. 9 is a diagram showing an example of a distance index of hierarchical clustering.

The seat estimation unit 123 may set a representative value of the cluster for distance calculation. In the example of FIG. 9, the seat estimation unit 123 sets the average value of the center coordinates of the static boxes in the cluster to (c _x , _cy ), the average value of the box sizes to size, and the average value of the box sizes in the cluster for each static box. The number of static boxes included may be n. Further, the seat estimation unit 123 arranges a binary array in which the time (frame) with observation is “1” and the time (frame) without observation is “0” for the static box included in the cluster. It may be generated and held in, for example, the memory unit 11. The array is an example of information indicating the detection status of each object in each of the plurality of first images.

The seat estimation unit 123 may obtain a distance index between clusters by calculating the following equations (1) to (3) as illustrated in FIG. 9 using the above representative values.

Here, (c _xA , _cyA ), size _A , and n _A indicate the coordinates, size, and number of static boxes of cluster A, respectively, and (c _xB , _cyB ), size _B , and n _B , respectively. , The coordinates of cluster B, the size, and the number of static boxes.

In the above equation (3), the portion other than "penalty" is the same as the distance index of the hierarchical clustering using Ward's method. On the other hand, in one embodiment, the accuracy of estimating the seat position and the number of seats is improved by adding "penalty" to the distance index of the Ward method.

For example, if a static box, in other words, a "sitting person", is observed between clusters A and B at the same time (same frame), these clusters have different seats from each other. Is expected.

Therefore, in the above equation (3), when a static box is observed between clusters A and B at the same time (same frame), the distance between the clusters is increased according to the number of observations. A penalty term has been added. For example, the seat estimation unit 123 may take an AND between arrays of observation times and apply the SUM (total value) of the array of AND results to the above equation (3) as a penalty.

In this way, the seat estimation unit 123 may generate information indicating whether or not an object associated with the static object exists in each of the plurality of first images for each static object. Then, the seat estimation unit 123 may exclude static objects whose corresponding objects exist in the first image from the grouping target based on the generated information.

The seat estimation unit 123 uses the _dward obtained by the above equation (3) as a distance index, and from the initial state in which the data of all static boxes are separate clusters (see "initial state" in FIG. 10), the following. Hierarchical clustering may be performed by the procedure of.

For example, the seat estimation unit 123 determines the cluster pair having the smallest distance index ( _dward ) between clusters, and determines whether or not the stop condition of the following equation (4) is satisfied.

When the stop condition of the above equation (4) is satisfied, the seat estimation unit 123 ends the hierarchical clustering and stores the remaining cluster information as the seat information 114 in the memory unit 11.

On the other hand, if the stop condition of the above equation (4) is not satisfied, the seat estimation unit 123 merges the cluster pairs into a new cluster.

The seat estimation unit 123 repeatedly executes the above process until the stop condition is satisfied or the number of clusters becomes 1.

In this way, the clusters remaining unmerged by the hierarchical clustering are estimated as seat positions (see “Seat position estimation result” in FIG. 10). The seat estimation unit 123 uses the average value (c _x , _cy ) of the center coordinates of the static boxes included in the cluster and the average value (size) of the box size as the data of the seat position and the reference size. good.

The seat estimation unit 123 may determine that the clusters having an extremely small number of samples (value of n) contained in the clusters among the clusters remaining without being merged are noise and exclude them from the seat information. .. The cluster with an extremely small number of samples may be, for example, a cluster having a sample number of several percent (exemplarily 2%) or less of the average number of samples contained in other clusters.

[1-3-4] Description of Congestion Degree Calculation Unit The congestion degree calculation unit 124 calculates the congestion degree using the head detection result and the seat information 114 in the congestion status estimation phase.

For example, the congestion degree calculation unit 124 calculates the distance between the detection result of the human head and each seat position of the seat information 114, and with respect to the seat position in order from the pair of the detection result and the seat position having the closest distance. Allocate the detection result of the head of a person who is at a distance below the threshold from the seat position.

As a result, information on the seat to which the detection result of the head is assigned (seat with a person) and information on the seat to which the detection result is not assigned (vacant seat) can be obtained. If the detection position of the head and the seat position are separated by a predetermined distance or more, the head may not be assigned to the seat position.

As the detection result of the human head, for example, the detection result of the object in the second image acquired in the congestion situation estimation phase may be used. As an example, the detection result of the human head is the object detected in the latest detection target frame (or the time requested by the terminal device 6) included in the detection information 112 stored in the memory unit 11. Information may be used.

The congestion degree calculation unit 124 may calculate the congestion rate based on the frame to be detected, for example, by calculating "the number of seats with people / the total number of seats". Further, the congestion degree calculation unit 124 is "vacant" / "normal" / "crowded" by setting threshold values such as "0.3" and "0.6" for the calculated congestion rate. The degree of congestion (discrete value) indicating the degree of congestion such as "" may be estimated. In the above example, the degree of congestion is estimated in three stages, but the number of stages of the degree of congestion may be increased or decreased by increasing or decreasing the number of threshold values.

As described above, when the seat position estimation (update) phase is executed during the congestion status estimation phase, the static box determination unit 122 and the seat estimation unit 123 are in the second (first) image. The seat information 114 may be estimated (updated) using the detection result of the object.

FIG. 11 shows an example of the estimation result of the degree of congestion by the degree of congestion calculation unit 124. In addition, FIG. 11 shows a state in which a seat with a person and a vacant seat detected by the congestion degree calculation unit 124 based on the image of the shooting space 30 taken by the surveillance camera 2 are fitted. show. In FIG. 11, the seat position is indicated by a circle symbol, and the seat position in which a person is seated is indicated by a square symbol.

FIG. 11A shows a case where the degree of congestion is “vacant”, for example, a case where the congestion rate is less than “0.3”. FIG. 11B shows a case where the degree of congestion is “normal”, for example, a case where the congestion rate is “0.3” or more and less than “0.6”. FIG. 11C shows a case where the degree of congestion is “crowded”, for example, a case where the congestion rate is “0.6” or more.

The congestion degree calculation unit 124 may store the estimated congestion degree information in the memory unit 11 as the congestion degree information 115.

As shown by reference numeral A in FIG. 11A, a seat does not have to be assigned to an object detected in an area where the seat position information does not exist, and in this case, the object is used for calculating the congestion rate. You don't have to consider it. As a result, it is possible to prevent the clerk, the moving customer, and the like from affecting the calculation of the congestion rate. In FIG. 11A, an object to which a seat is not assigned is indicated by a double-lined square symbol.

[1-3-5] Explanation of Information Presentation Unit The information presentation unit 14 may present information on the degree of congestion estimated by the control unit 12 to, for example, the terminal device 6.

FIG. 12 is a diagram showing an example of presentation of information to the terminal device 6 by the information presentation unit 14. For example, as shown in FIG. 12, the information presenting unit 14 responds to a request from the terminal device 6 with respect to the display device 60 of the terminal device 6, and the congestion degree information 115 which is an estimation result by the congestion degree calculation unit 124. The degree of congestion of the store 3 based on the above may be presented.

The information presented may be information about one store 3 or information about each of the plurality of stores 3 as shown in FIG. Further, as shown in FIG. 12, the information presenting unit 14 may be able to browse the vacant seat status of the store 3. The vacant seat information may be, for example, information that generates a map based on the estimated seat position and identifies (for example, marks) a vacant seat or a seat in which a person is seated on the map.

As an example of the terminal device 6 and the information presentation unit 14, the terminal device 6 may function as a Web client and the information presentation unit 14 may function as a Web server. In this case, the information presenting unit 14 may generate a Web page in response to a request from the terminal device 6, and may respond to the generated Web page to the terminal device 6. The request may be, for example, a http (Hypertext Transfer Protocol) request for a predetermined URL (Uniform Resource Locator). Further, the Web page may be generated by a markup language such as html (HyperText Markup Language).

As described above, according to one embodiment, the server 4 detects the head of a person in the image of the surveillance camera 2 by the NN 13, and estimates the position of the detected head by the static box determination unit 122 and the seat. The unit 123 may be used for learning, for example, by deep learning.

The location where the person's head frequently stays, for example, stops, can be presumed to be the position of the seat. Therefore, by determining whether or not the head of the person detected from the video overlaps with the estimated seat position, it is possible to determine whether or not there is a person in the seat, and the congestion rate can be calculated.

Further, since the seat position can be estimated from the image of the surveillance camera 2 by the processing by the static box determination unit 122 and the seat estimation unit 123, it is not necessary to acquire the map of the seat position of the store 3 and the number of seats in advance. Also, the method according to one embodiment can be applied.

For example, in the store 3, by performing simple initial settings such as transmitting the image of the surveillance camera 2 to the server 4, the method according to one embodiment can be easily applied to the store 3, and the seat congestion rate can be achieved with high accuracy. Can be grasped. In particular, when grasping the congestion status of the seats of many stores 3, complicated initial settings and the like are not required, so the shooting area and the like of the surveillance camera 2 are set for each store as in the conventional case. Many costs can be reduced.

The method according to one embodiment can be applied or applied to, for example, the following situations.

・ Monitor and quantify the congestion status of store 3 in real time.
・ The managers of facilities and stores 3 grasp the congestion situation and utilize it for store strategy.
-The user of the facility or store 3 confirms the congestion status with the mobile application of the terminal device 6 or the like, and makes a decision as to whether or not to go to the facility or store 3.

[1-4] Operation Example Next, an operation example of the congestion degree estimation system 1 configured as described above will be described with reference to FIGS. 13 to 15.

[1-4-1] Operation example of the seat position estimation phase First, an operation example of the seat position estimation phase will be described with reference to FIGS. 13 and 14.

In the server 4, the information acquisition unit 121 of the control unit 12 acquires the video data of the surveillance camera 2 installed in the store 3 and stores it in the memory unit 11 as the video data 111.

As illustrated in FIG. 13, the NN13 of the server 4 acquires frames from the video data 111 at regular intervals (for example, every 5 minutes) (step S1), and detects a box of a person's head from the frames (step S2). ). The NN 13 may store the detected box as the detection information 112 in the memory unit 11.

The static box determination unit 122 of the control unit 12 compares each box in the current (for example, the latest) frame with the box in the past N (for example, N = 5) frame based on the detection information 112 (step S3). ).

For comparison between boxes, screening by comparison of box size and / or distance between boxes and comparison of total scores of box similarity using color histograms, pixel differences, and IoU, etc. are performed. May include (see FIGS. 6 and 7).

The static box determination unit 122 determines whether or not the box in the compared current frame is the same as any static box in the past frame (step S4).

If the box is the same as any static box in the past frame (Yes in step S4), the static box determination unit 122 adds the box to the data of the static box in the box information 113 (Yes). Step S5), the process proceeds to step S8.

On the other hand, when the box is not the same as any static box in the past frame (No in step S4), the same non-static box as the box is in the M (for example, M = 3) frame in the past N frames. It is determined whether or not it exists (step S6). If the condition of step S6 is not satisfied (No in step S6), the process proceeds to step S8.

On the other hand, when the condition of step S6 is satisfied (Yes in step S6), the static box determination unit 122 manages these boxes and non-static boxes as new static boxes in the box information 113 (step S7). , The process proceeds to step S8.

In step S8, it is determined whether or not there is a static box that has not been observed (that is, has not been detected) in the past M frames in a row. If the corresponding static box does not exist (No in step S8), the process proceeds to step S10.

On the other hand, when the corresponding static box exists (Yes in step S8), the static box determination unit 122 writes the static box to a file and deletes it from the memory (step S9). Then, the static box determination unit 122 determines whether or not all the boxes have been compared in the current frame (step S10).

If all the boxes have not been compared (No in step S10), the process proceeds to step S3, and the boxes for which comparison has not been performed are compared. On the other hand, when all the boxes are compared (Yes in step S10), the static box determination unit 122 determines whether or not a predetermined period, for example, 3 days has elapsed since the accumulation of the box information 113 was started (Yes). Step S11).

If the predetermined period has not elapsed (No in step S11), the process proceeds to step S1, and the above process is executed for the frame to be input next.

On the other hand, when the predetermined period has elapsed (Yes in step S11), the process proceeds to step S12 in FIG. In this case, the NN 13 and the static box determination unit 122 may continue the processing based on the input video data 111 as the seat position estimation (update) phase.

In step S12 of FIG. 14, the seat estimation unit 123 sets the representative value of each cluster with each static box as the initial cluster. For example, the seat estimation unit 123 has an average value (c _x , _cy ) of the center coordinates of the static boxes in the cluster, an average value of the box size “size”, the number of static boxes in the cluster “n”, and the cluster. An array indicating the presence or absence of observation of the static box of is set. In the array, for example, "1" may be set for each frame if a static box is observed, and "0" may be set if the static box is not observed.

Next, the seat estimation unit 123 calculates a distance index ( _dward ) between clusters for each pair of clusters (step S13). The distance index ( _dward ) may be calculated by the above equation (3). In the calculation of the distance index ( _dward ), a penalty obtained by the SUM (total value) of the AND result between the arrays of the cluster pair may be added (see FIG. 9).

The seat estimation unit 123 determines the cluster pair having the smallest distance index ( _dward ) (step S14), and determines whether or not the cluster pair satisfies the stop condition (step S15). The stop condition may be calculated by the above equation (4) (see FIG. 9).

When the cluster pair does not satisfy the stop condition (No in step S15), the seat estimation unit 123 merges the cluster pair into a new cluster (step S16).

The seat estimation unit 123 determines whether or not the number of remaining clusters is 1 (step S17). When the number of remaining clusters is not 1 (No in step S17), the process proceeds to step S13.

If the cluster pair satisfies the stop condition in step S15 (Yes in step S15), or if the number of remaining clusters is 1 in step S17 (Yes in step S17), the process proceeds to step S18.

In step S18, the seat estimation unit 123 deletes a cluster in which the number of static boxes in the cluster is equal to or less than a predetermined ratio (for example, 2%) of the number of static boxes in other clusters.

Then, the seat estimation unit 123 stores the remaining clusters in the seat information 114 and output, for example, the memory unit 11 (step S19), and the processing of the seat position estimation phase is completed.

[1-4-2] Operation example of congestion status estimation phase Next, an operation example of the congestion status estimation phase will be described with reference to FIG.

As illustrated in FIG. 15, the NN 13 acquires frames from the video data 111 at regular intervals (for example, at intervals of 5 minutes) (step S21), and detects a box of a person's head from the frames (step S22). The NN 13 may store the detected box as the detection information 112 in the memory unit 11. Further, when the seat position estimation (update) phase is performed, the processing of steps S1 and S2 in FIG. 13 in the seat position estimation (update) phase may be replaced with the processing of steps S21 and S22 (steps S1 and S2). Processing may be omitted).

The congestion degree calculation unit 124 of the control unit 12 calculates the distance between the box of the current (for example, the latest) frame in the detection information 112 and the seat position in the seat information 114 (step S23). The distance may be the distance between the center coordinates or the average coordinates of the box and the seat position.

Next, the congestion degree calculation unit 124 assigns a box having a distance equal to or less than a threshold value to the seat position in the order of the closest distance (step S24).

Then, the congestion degree calculation unit 124 counts the number of seats to which the box is assigned, in other words, the number of seats with people (step S25), and calculates "the number of seats with people / the total number of seats" as the congestion rate. (Step S26).

The congestion degree calculation unit 124 determines whether or not the calculated congestion rate is less than “0.3”, which is an example of the first threshold value (step S27). When the congestion rate is less than "0.3" (Yes in step S27), the congestion degree calculation unit 124 estimates that the congestion degree is "vacant" (step S28), and transfers the congestion degree information 115 to the memory unit 11. After storing, the process proceeds to step S32.

When the congestion rate is "0.3" or more (No in step S27), the congestion degree calculation unit 124 determines whether or not the congestion rate is less than "0.6", which is an example of the second threshold value (No). Step S29). When the congestion rate is less than "0.6" (and "0.3" or more) (Yes in step S29), the congestion degree calculation unit 124 estimates the congestion degree to be "normal" (step S30) and determines the congestion degree. The information 115 is stored in the memory unit 11, and the process proceeds to step S32.

On the other hand, when the congestion rate is "0.6" or more (No in step S29), the congestion degree calculation unit 124 estimates that the congestion degree is "crowded" (step S31), and stores the congestion degree information 115 in the memory unit. Stored in 11, the process proceeds to step S32.

In step S32, the information presentation unit 14 of the server 4 presents the congestion degree information of the store 3 based on the congestion degree information 115. For example, the information presenting unit 14 may display a Web page including information on the degree of congestion of the store 3 on the terminal device 6 in response to a request from the terminal device 6.

As a result, the processing of the congestion status estimation phase is completed.

[1-5] Hardware Configuration Example Next, a hardware configuration example of the server 4 according to the embodiment will be described with reference to FIG. Hereinafter, a computer 10 will be taken as an example of the server 4, and a hardware configuration example of the computer 10 will be described. The terminal device 6 may also have the same hardware configuration as the server 4.

As shown in FIG. 16, the computer 10 is exemplified by a processor 10a, a memory 10b, a storage unit 10c, an IF (Interface) unit 10d, an I / O (Input / Output) unit 10e, and a reading unit 10f. good.

The processor 10a is an example of an arithmetic processing unit that performs various controls and operations. The processor 10a may be connected to each block in the computer 10 so as to be communicable with each other by the bus 10i. As the processor 10a, for example, an integrated circuit (IC) such as a CPU, MPU, DSP, ASIC, or FPGA may be used. CPU is an abbreviation for Central Processing Unit, and MPU is an abbreviation for Micro Processing Unit. DSP is an abbreviation for Digital Signal Processor, ASIC is an abbreviation for Application Specific Integrated Circuit, and FPGA is an abbreviation for Field-Programmable Gate Array.

The memory 10b is an example of hardware that stores information such as various data and programs. Examples of the memory 10b include volatile memories such as RAM.

The storage unit 10c is an example of hardware that stores information such as various data and programs. Examples of the storage unit 10c include a magnetic disk device such as an HDD, a semiconductor drive device such as an SSD, and various storage devices such as a non-volatile memory. Examples of the non-volatile memory include flash memory, SCM (Storage Class Memory), ROM (Read Only Memory) and the like.

The memory unit 11 shown in FIG. 1 may be realized by, for example, at least one storage area of the memory 10b and the storage unit 10c of the server 4.

Further, the storage unit 10c may store a program 10g that realizes all or a part of various functions of the computer 10. The processor 10a can realize the function as the server 4 shown in FIG. 1 by expanding and executing the program 10g stored in the storage unit 10c in the memory 10b.

The IF unit 10d is an example of a communication interface that controls connection and communication with the network 5. For example, the IF unit 10d may include a LAN or an adapter compliant with optical communication (for example, FC (Fibre Channel)). For example, the program 10g may be downloaded from the network 5 to the computer 10 via the communication interface and stored in the storage unit 10c.

The I / O unit 10e includes one or both of an input unit such as a mouse, a keyboard, or an operation button, and an output unit such as a touch panel display, a monitor such as an LCD (Liquid Crystal Display), a projector, or a printer. good.

The reading unit 10f is an example of a reader that reads data and program information recorded on the recording medium 10h. The reading unit 10f may include a connection terminal or device to which the recording medium 10h can be connected or inserted. Examples of the reading unit 10f include an adapter compliant with USB (Universal Serial Bus), a drive device for accessing a recording disk, a card reader for accessing a flash memory such as an SD card, and the like. The program 10g may be stored in the recording medium 10h, or the reading unit 10f may read the program 10g from the recording medium 10h and store it in the storage unit 10c.

Examples of the recording medium 10h include a non-temporary recording medium such as a magnetic / optical disk or a flash memory. Examples of the magnetic / optical disk include flexible discs, CDs (Compact Discs), DVDs (Digital Versatile Discs), Blu-ray discs, HVDs (Holographic Versatile Discs), and the like. Examples of the flash memory include a USB memory and an SD card. Examples of the CD include a CD-ROM, a CD-R, a CD-RW, and the like. Examples of the DVD include a DVD-ROM, a DVD-RAM, a DVD-R, a DVD-RW, a DVD + R, a DVD + RW, and the like.

The hardware configuration of the computer 10 described above is an example. Therefore, in the server 4, the increase / decrease of hardware (for example, addition or deletion of arbitrary blocks), division, integration in any combination, addition or deletion of buses, etc. in the computer 10 may be performed as appropriate. ..

[2] Others The technique according to the above-described embodiment can be modified or modified as follows.

For example, each functional block of the server 4 shown in FIG. 1 may be merged or divided in any combination. Further, each functional block of the control unit 12 shown in FIG. 1 may be merged or divided in any combination.

Further, the processor 10a of the computer 10 shown in FIG. 16 is not limited to a single processor or a single core processor, and may be a multiprocessor or a multicore processor.

In the above-described embodiment, the process of calculating the degree of congestion from the input of the store image is described by using the method of estimating the retention area based on the recognition result of the detection model of the human head for the image. However, the estimated retention region information is not limited to the use in the above-described embodiment, and may be used for various analyzes or estimations.

Further, in one embodiment, the seat estimation unit 123 performs layered clustering, but is not limited to this. For example, when the number of seats provided in the store 3 is known in advance, the seat estimation unit 123 may perform non-layered clustering so as to create clusters having the same number of seats.

The information on the number of seats may be acquired from the Internet or the like via the network 5 by, for example, a server 4 or an administrator, and stored in a database, for example, a memory unit 11. In this case, the information acquisition unit 121 may function as an example of an acquisition unit that acquires information on the number of seats in the shooting space 30 from the memory unit 11 during clustering by the seat estimation unit 123.

Information on the number of seats in the store 3 is often posted on the facility, the homepage of the store 3, the introduction site, or the like on the Internet, and can be easily obtained. Therefore, it is possible to accurately estimate the seat position based on the accurate number of seats while suppressing the increase in cost in the initial setting.

Further, in one embodiment, the server 4 estimates the seat position and the degree of congestion based on the video data from one surveillance camera 2, but the present invention is not limited to this.

For example, when the server 4 acquires video data from each of the plurality of surveillance cameras 2 provided in the store 3, the server 4 may estimate the seat position and the degree of congestion for each surveillance camera 2. Then, the server 4 may estimate the degree of congestion in the store 3 by, for example, averaging the estimated degree of congestion (or the degree of congestion) for each surveillance camera 2.

[3] Additional notes The following additional notes will be further disclosed with respect to the above embodiments.

(Appendix 1)
For the objects detected from each of the plurality of first images obtained by photographing the space, the plurality of objects are based on the position information of each detected object and the detection status of each object in each of the plurality of first images. Grouping objects that are related to each other between the first images
Based on the result of the grouping, the residence area where each object stays in the space is estimated.
An estimation program that lets a computer perform processing.

(Appendix 2)
The degree of congestion of the stagnant region is estimated based on the position information of the object detected from the second image obtained by photographing the space and the estimated information of the stagnant region in the space.
The estimation program according to Appendix 1, which causes the computer to execute the process.

(Appendix 3)
A predetermined number or more of objects determined to exist at the same position among a predetermined number or more of the first images are managed in association with one static object.
Let the computer perform the process
In the grouping, static objects satisfying the condition regarding distance are grouped as the interconnected objects among the plurality of first images.
The estimation program described in Appendix 1 or Appendix 2.

(Appendix 4)
The management manages the predetermined number or more of objects determined to match the image components of the objects among the predetermined number or more of the first images in association with the one static object.
The estimation program described in Appendix 3.

(Appendix 5)
The grouping is
Information indicating the detection status of each object in each of the plurality of first images, and for each of the static objects, there is an object associated with the static object in each of the plurality of first images. Generates information indicating whether or not
Based on the generated information, static objects whose corresponding objects exist in one first image are excluded from the grouping target.
The estimation program according to Appendix 3 or Appendix 4.

(Appendix 6)
The grouping performs hierarchical clustering.
The estimation program according to any one of Supplementary Provisions 1 to 5.

(Appendix 7)
The number of the stagnant areas in the space is obtained from the database.
Let the computer perform the process
The grouping performs non-hierarchical clustering so as to create a group of the number of acquired residence regions.
The estimation program according to any one of Supplementary Provisions 1 to 5.

(Appendix 8)
The object is the head of the human body
The stagnation area is a seating area.
Using a neural network that detects the feature amount of an image, the object that is the head of the human body is detected from each of the plurality of first images.
The estimation program according to any one of Supplementary note 1 to 7, wherein the processing is executed by the computer.

(Appendix 9)
For the objects detected from each of the plurality of first images obtained by photographing the space, the plurality of objects are based on the position information of each detected object and the detection status of each object in each of the plurality of first images. A grouping unit that groups objects that are related to each other between the first images,
Based on the result of the grouping, an estimation unit that estimates the retention area where each object stays in the space, and an estimation unit.
Estimating system with.

(Appendix 10)
A congestion degree estimation unit that estimates the degree of congestion in the residence area based on the position information of the object detected from the second image obtained by photographing the space and the estimated information on the retention area in the space.
The estimation system described in Appendix 9 is provided.

(Appendix 11)
It is equipped with a management unit that manages the predetermined number or more of objects determined to exist at the same position among the predetermined number or more of the first images in association with one static object.
The grouping unit groups static objects satisfying the condition of distance between the plurality of first images as the mutually related objects.
The estimation system according to Appendix 9 or Appendix 10.

(Appendix 12)
The management unit manages the predetermined number or more of objects determined to match the image components of the objects among the predetermined number or more of the first images in association with the one static object.
The estimation system according to Appendix 11.

(Appendix 13)
The grouping unit
Information indicating the detection status of each object in each of the plurality of first images, and for each of the static objects, there is an object associated with the static object in each of the plurality of first images. Generates information indicating whether or not
Based on the generated information, static objects whose corresponding objects exist in one first image are excluded from the grouping target.
The estimation system according to Appendix 11 or Appendix 12.

(Appendix 14)
The grouping unit performs the grouping by hierarchical clustering.
The estimation system according to any one of Supplementary note 9 to 13.

(Appendix 15)
It is equipped with an acquisition unit that acquires the number of the retention areas in the space from the database.
The grouping unit performs the grouping by non-hierarchical clustering so as to create a group of the number of acquired residence regions.
The estimation system according to any one of Supplementary note 9 to 13.

(Appendix 16)
The object is the head of the human body
The stagnation area is a seating area.
In any one of the appendices 9 to 15, the neural network that detects the feature amount of the image and that detects the object that is the head of the human body from each of the plurality of first images. The estimation system described.

(Appendix 17)
For the objects detected from each of the plurality of first images obtained by photographing the space, the plurality of objects are based on the position information of each detected object and the detection status of each object in each of the plurality of first images. Grouping objects that are related to each other between the first images
An estimation method for estimating a retention region in which each object stays in the space based on the result of the grouping.

(Appendix 18)
The degree of congestion of the stagnant region is estimated based on the position information of the object detected from the second image obtained by photographing the space and the estimated information of the stagnant region in the space.
The estimation method described in Appendix 17.

(Appendix 19)
The predetermined number or more of the objects determined to exist at the same position among the predetermined number or more of the first images are managed in association with one static object.
In the grouping, static objects satisfying the condition regarding distance are grouped as the interconnected objects among the plurality of first images.
The estimation method according to Appendix 17 or Appendix 18.

(Appendix 20)
The management manages the predetermined number or more of objects determined to match the image components of the objects among the predetermined number or more of the first images in association with the one static object.
The estimation method described in Appendix 19.

1 Congestion degree estimation system 2 Surveillance camera 3 Store 4 Server 5 Network 6 Terminal equipment 10 Computer 11 Memory unit 111 Video data 112 Detection information 113 Box information 114 Seat information 115 Congestion degree information 12 Control unit 121 Information acquisition unit 122 Static box judgment Part 123 Seat estimation part 124 Congestion degree calculation part 13 NN (Neural network)
14 Information presentation section

Claims (8)

For the objects detected from each of the plurality of first images obtained by photographing the space, the plurality of objects are based on the position information of each detected object and the detection status of each object in each of the plurality of first images. Grouping objects that are related to each other between the first images
Based on the result of the grouping, the residence area where each object stays in the space is estimated.
Let the computer do the processing,
Among the first images having a predetermined number or more, the objects having the predetermined number or more determined to match the image components of the objects are managed in association with one object.
Further processing is performed by the computer,
The grouping includes a process of grouping one object satisfying the condition regarding distance between the plurality of first images as the mutually related object.
The grouping further
Information indicating the detection status of each object in each of the plurality of first images, and for each of the one object, there is an object associated with the one object in each of the plurality of first images. Generates information indicating whether or not
Including a process of excluding one object whose corresponding object exists in one first image from the target of grouping based on the generated information.
Estimate program. The degree of congestion of the stagnant region is estimated based on the position information of the object detected from the second image obtained by photographing the space and the estimated information of the stagnant region in the space.
The estimation program according to claim 1, wherein the processing is executed by the computer. The management manages an object that is determined to exist at the same position among the predetermined number or more of the first images and that the image components are determined to match in association with the one object.
The estimation program according to claim 1 or 2. The grouping performs hierarchical clustering.
The estimation program according to any one of claims 1 to 3 . The number of the stagnant areas in the space is obtained from the database.
Let the computer perform the process
The grouping performs non-hierarchical clustering so as to create a group of the number of acquired residence regions.
The estimation program according to any one of claims 1 to 3 . The object is the head of the human body
The stagnation area is a seating area.
Using a neural network that detects the feature amount of an image, the object that is the head of the human body is detected from each of the plurality of first images.
The estimation program according to any one of claims 1 to 5 , which causes the computer to execute the process. For the objects detected from each of the plurality of first images obtained by photographing the space, the plurality of objects are based on the position information of each detected object and the detection status of each object in each of the plurality of first images. A grouping unit that groups objects that are related to each other between the first images,
Based on the result of the grouping, an estimation unit that estimates the retention area where each object stays in the space, and an estimation unit.
It is an estimation system that has
The estimation system further includes a management unit that manages the predetermined number or more of objects determined to match the image components of the objects among the predetermined number or more of the first images in association with one object.
In the grouping, one object satisfying the condition regarding the distance is grouped as the mutually related object among the plurality of first images.
The grouping further
Information indicating the detection status of each object in each of the plurality of first images, and for each of the one object, there is an object associated with the one object in each of the plurality of first images. Generates information indicating whether or not
Based on the generated information, one object whose corresponding object exists in one first image is excluded from the grouping target.
Estimating system. For the objects detected from each of the plurality of first images obtained by photographing the space, the plurality of objects are based on the position information of each detected object and the detection status of each object in each of the plurality of first images. Grouping objects that are related to each other between the first images
Based on the result of the grouping, the residence area where each object stays in the space is estimated.
The computer executes the process,
Among the first images having a predetermined number or more, the objects having a predetermined number or more determined to match the image components of the objects are managed in association with one object.
Further processing is performed by the computer,
In the grouping, one object satisfying the condition regarding the distance is grouped as the mutually related object among the plurality of first images.
The grouping further
Information indicating the detection status of each object in each of the plurality of first images, and for each of the one object, there is an object associated with the one object in each of the plurality of first images. Generates information indicating whether or not
Based on the generated information, one object whose corresponding object exists in one first image is excluded from the grouping target.
Estimating method.

Images