JP2023159972A - Stay state presentation device and stay state presentation method - Google Patents

Abstract

To enable information having higher reliability about a congestion state to be presented to a user by predicting a person leaving a facility in near feature on the basis of movement of each person.SOLUTION: Persons existing in a facility are detected, persons whose existing time becomes a prescribed threshold or greater are extracted from among existing persons as prediction objects, leaving of persons of the predicted objects is predicted on the basis of a prescribed leaving sign movement, and at least the number of existing persons (the number of attending persons) and the number persons predicted to leave (the number of leaving-predicted persons) are displayed on a guide screen of a user terminal. Also, time of updating the number of existing persons and the number of persons predicted to leave is displayed on the guide screen of the user terminal on the basis of the latest processing result of processing of detecting the person of existing persons and processing of predicting leaving. Also, the number of users currently browsing the guide screen displayed on the user terminal is displayed on the user terminal in response to access from the user terminal.SELECTED DRAWING: Figure 8

Description

The present invention relates to a stay situation presentation device and a stay situation presentation method that generate information regarding a person's stay situation at a facility based on images taken of the facility and present the information to a user.

Convenience for users can be improved by distributing the current congestion status of facilities such as restaurants to user terminals in real time and presenting the current congestion status to users who are about to use the facility. On the other hand, since the congestion situation of the facility changes when the person staying at the facility leaves the facility, it is good to predict the future congestion situation and present the prediction result to the users.

Conventionally, as a technology to predict the congestion situation of a facility and present it to users, the number of people present is measured based on the detection results of sensors (motion sensors, cameras, etc.). There is a known technology that uses a prediction model generated by machine learning based on information about past congestion to predict future congestion and present the prediction results to users (see Patent Document 1). .

Japanese Patent Application Publication No. 2021-189708

According to conventional technology, future congestion conditions can be predicted by using a prediction model. However, since the conventional technology is machine learning based on information regarding past congestion status of the entire facility, it is not possible to predict the trend of congestion status of the facility, that is, how the congestion status will change from a statistical perspective. It is something to do. Therefore, in reality, congestion often does not resolve as predicted, and it is desirable to present more reliable information regarding the congestion situation to users.

SUMMARY OF THE INVENTION The present invention provides a stay status presentation device and a stay status presentation device that can present users with more reliable information regarding crowding status by predicting who will leave the facility in the near future based on the behavior of each person. The main purpose is to provide a method for presenting the stay status.

A stay situation presentation device of the present invention is a stay situation presentation device in which a processor executes a process of generating display information regarding the degree of congestion of a facility based on a video shot of a person's stay area in the facility, the processor comprising: A person staying in the stay area is detected, a person whose stay time is equal to or more than a predetermined threshold is extracted as a prediction target from among the people staying in the stay area, and based on a predetermined departure sign movement, the The present invention is configured to predict the departure of the person who is the prediction target, and to generate the display information including at least the number of people staying in the residence and the number of people expected to leave.

Further, the stay status presentation method of the present invention is a stay status presentation method in which a processor executes a process of generating display information regarding the degree of congestion of a facility based on a video shot of a person's stay area in the facility, the method comprising: Detect people staying in the area, extract people whose stay time exceeds a predetermined threshold as prediction targets from among the people staying in the area, and select the prediction targets based on predetermined departure sign movements. The display information is configured to predict the departure of the person who has been staying, and to generate the display information including at least the number of people staying in the residence and the number of people predicted to leave.

According to the present invention, the number of people staying and the number of people expected to leave are presented to the user. By considering the number of people expected to leave, the user can correct and judge the degree of congestion of the facility based on the number of people staying. Thereby, more reliable information regarding the congestion situation of the facility can be presented to the user. In addition, the process of predicting when a person will leave is executed for people whose presence time exceeds a predetermined threshold, and the process is narrowed down to those who are likely to leave. The load can be reduced.

Overall configuration diagram of the stay situation presentation system according to this embodiment Explanatory diagram showing the camera shooting situation at the facility Block diagram showing the schematic configuration of the server Explanatory diagram showing the registered contents of the people present database managed by the server Explanatory diagram illustrating an example of the type of departure warning behavior and the predicted score obtained for each type of departure warning behavior Explanatory diagram showing an overview of the vacancy prediction process performed by the server A flow diagram showing an example of the steps of presence detection processing, prediction target extraction processing, and absence prediction processing performed by the server. Explanatory diagram showing the guidance screen displayed on the user terminal Explanatory diagram showing the settings screen displayed on the administrator terminal

A first invention made to solve the above problem is a stay situation presentation device in which a processor executes a process of generating display information regarding the degree of congestion of a facility based on a video shot of a person's stay area in the facility. The processor detects people staying in the stay area, extracts people whose stay time exceeds a predetermined threshold as prediction targets from among the people staying in the stay area, and detects a person who is staying in the stay area as a prediction target. Based on the movement, the departure of the person targeted for prediction is predicted, and the display information is generated including at least the number of people staying in the residence and the number of people predicted to leave.

According to this, the number of people staying and the number of people predicted to leave are presented to the user. By considering the number of people expected to leave, the user can correct and judge the degree of congestion of the facility based on the number of people staying. Thereby, more reliable information regarding the congestion situation of the facility can be presented to the users. In addition, the process of predicting when a person will leave is executed for people whose presence time exceeds a predetermined threshold, and the process is narrowed down to those who are likely to leave. The load can be reduced.

Further, in a second aspect of the present invention, the stay area is a presence detection area for detecting a person present in the seat, and the leaving sign operation is a leave sign movement performed by the person in the seat immediately before leaving the seat. The configuration is as follows.

According to this, it is possible to present appropriate information to the user regarding the congestion status of the facility where the seats are provided.

Further, in a third aspect of the present invention, the processor is configured to transmit the display information to the terminal device in response to an access from a terminal device operated by a user.

According to this, users can easily view information regarding the congestion status of the facility.

Further, in a fourth aspect of the invention, the processor is configured to set the threshold in response to an operation for specifying the threshold performed by a facility manager.

According to this, the administrator of the facility can arbitrarily designate the threshold value, taking into consideration the characteristics of the facility.

Further, in a fifth invention, the processor predicts the number of people staying and whether the person will leave based on the latest processing results of the process of detecting the person staying and the process of predicting leaving. The display information is configured to generate the display information including the time at which the number of people who will be interviewed is updated.

According to this, the user can easily understand how new the displayed information is.

Further, in a sixth invention, the processor generates the display information including the number of users viewing the display information transmitted to the terminal device in response to an access from a terminal device operated by a user. composition.

According to this, in addition to the number of people who are expected to leave, users can adjust the congestion level of the facility based on the number of people staying at the facility by considering the number of users who are viewing the displayed information. You can make a judgment.

Further, in a seventh aspect of the present invention, the moving-out warning motion is at least one of the following: an action of putting on clothes, an action of holding luggage, an action of finishing handling of electronic devices, an action of putting away the goods, and an action of storing the goods in a bag. The configuration is as follows.

According to this, it is possible to accurately predict who will leave the facility in the near future.

Further, an eighth invention is a stay status presentation method in which a processor executes a process of generating display information regarding the degree of congestion of a facility based on a video shot of a person's stay area in the facility, the method comprising: Detecting a person inside, extracting a person whose length of stay is equal to or greater than a predetermined threshold from among the people staying there as a prediction target, and selecting the person as a prediction target based on a predetermined exit sign movement. The present invention is configured to predict the departure of a person, and generate the display information including at least the number of people staying in the residence and the number of people predicted to leave.

According to this, similar to the first invention, by predicting who will leave the facility in the near future based on the movements of each person, it is possible to present users with more reliable information regarding the congestion situation. can.

Embodiments of the present invention will be described below with reference to the drawings.

(First embodiment)
FIG. 1 is an overall configuration diagram of a stay situation presentation system according to this embodiment.

This system presents users with the congestion status (stay status) of target facilities (restaurants, etc.). This system includes a camera 1, a server 2 (stay status presentation device), a user terminal 3 (terminal device), and an administrator terminal 4 (terminal device). Camera 1, server 2, user terminal 3, and administrator terminal 4 are connected via a network.

The camera 1 is installed at a target facility and photographs the inside of the facility.

The server 2 acquires a video shot inside the facility from the camera 1, and generates information regarding the congestion situation of the facility based on the video. The server 2 has the function of a web server, and distributes information regarding the congestion situation of the facility to the user terminal 3. The server 2 may be installed in a target facility and perform processing for one facility, or may be configured as a cloud computer and perform processing for multiple facilities.

The user terminal 3 is operated by a user (planned user) who is about to use the facility. The user terminal 3 displays a guidance screen (see FIG. 8) regarding the congestion status of the facility distributed from the server 2. Users can grasp the congestion status of the facility by viewing the information screen.

The administrator terminal 4 is operated by a facility administrator. The administrator terminal 4 displays a settings screen (see FIG. 9) for setting conditions for processing performed by the server 2, and the like. The administrator can perform an operation to specify processing conditions on the setting screen. The server 2 sets processing conditions according to the administrator's operations.

Note that the target facility in this embodiment is not particularly limited, but may be a restaurant or a company cafeteria that is mainly used by individuals or a small number of people. Further, the target facility may be an office that employs a free address system in which users can freely select seats. It may also be a sports gym or entertainment facility where visitors gather during a specific time slot.

Next, an overview of the processing performed in this system will be explained. FIG. 2 is an explanatory diagram showing a photographing situation of the camera 1 in a facility.

Target facilities (restaurants, etc.) are equipped with seats (chairs) and tables. A camera 1 is installed on the ceiling of the facility. A camera 1 photographs a person staying in the facility.

In this embodiment, a presence detection area for each seat is set in advance on the image of the camera 1, and based on the image of this presence detection area, it is determined whether the person is present, that is, whether the person is sitting in the seat or not. is detected and the current number of people present is presented to the user. This allows users to grasp the congestion situation of the facility.

On the other hand, even if the number of people currently in attendance is large, if there are people who finish using the seats and leave the seats in the near future, the actual degree of congestion can be considered to be low. It is not possible to present information that accurately reflects the situation.

Therefore, in this embodiment, motion prediction technology is used to predict which people will leave their seats in the near future (leaving prediction processing), and to predict who will leave their seats in the near future (leaving prediction processing), and to predict who will leave their seats in the near future. The number of people who will leave their seats in the near future (predicted number of people who will leave their seats) is presented in addition to the number of people currently present, distinguishing them from those who will leave their seats in the future. This allows users to more accurately grasp the congestion situation of the facility.

Furthermore, it is a heavy processing load to execute the vacancy prediction process for all the people present at the seat. Therefore, in this embodiment, since it is unlikely that a person who has just been present will leave the seat in the near future, the presence time (the elapsed time from the start time of presence) is calculated for each person who is present. Then, a person whose presence time is equal to or greater than a predetermined threshold is extracted as a prediction target (target of the absence prediction process), and the absence prediction process is executed. As a result, the targets of the vacancy prediction process are narrowed down to those who are likely to be vacated, thereby reducing the load of the vacancy prediction process.

Here, the user's presence time (staying time) differs depending on the facility. For example, in the case of a store that primarily serves drinks, such as a cafe, the seating time is relatively short, and in the case of a store that primarily provides food, such as a family restaurant, the seating time is relatively long. Therefore, it is desirable to set a threshold value for each facility based on the trend of attendance time at each facility. For example, the threshold value is set to 10 minutes in a facility where the occupancy time tends to be short, and the threshold value is set to 30 minutes in a facility where the occupancy time tends to be long.

In addition, in this embodiment, the administrator of the target facility can specify the threshold of attendance time, and the threshold of attendance time is set individually for each facility. It may be set according to the type of facility. That is, statistical processing regarding the past presence time of each person may be performed for each facility type, and the threshold of presence time may be set for each facility type.

Furthermore, in this embodiment, the usage status (occupancy status) of seats provided in the facility is presented to the user as the congestion status of the facility for facilities such as restaurants and offices. It may also be something that shows users the congestion status of facilities that do not have seats, such as apparel stores. In this case, in order to avoid confusion between people moving within the facility, it is preferable to track the person on the video of the camera 1.

Further, it is preferable that the camera 1 is a box camera having a predetermined angle of view. Thereby, it is possible to obtain a video that allows highly accurate seat-absence prediction processing.

On the other hand, an omnidirectional camera that takes images in all directions may be used as a part of the camera 1. Omnidirectional cameras can obtain images suitable for presence detection processing, but since images from omnidirectional cameras are taken from directly above the person, the person's movements cannot be properly recognized. It is not suitable for use in seat prediction processing. For this reason, a box camera may be used to acquire an image for performing the vacancy prediction process, and an omnidirectional camera may be used to acquire an image for performing the presence detection process. In this case, it is preferable to perform a process (calibration) that associates the position on the image of the box camera with the position on the image of the omnidirectional camera.

Next, a schematic configuration of the server 2 will be explained. FIG. 3 is a block diagram showing a schematic configuration of the server 2. As shown in FIG.

The server 2 includes a communication section 21, a storage section 22, and a processor 23.

The communication unit 21 communicates with the camera 1, the user terminal 3, and the administrator terminal 4 via the network.

The storage unit 22 stores programs executed by the processor 23 and the like. Furthermore, the storage unit 22 stores images received from the camera 1. The storage unit 22 also stores registration information (presence start time, presence time, etc.) of the persons present database.

The processor 23 performs various processes by executing programs stored in the storage unit 22. In this embodiment, the processor 23 performs a presence detection process, a prediction target extraction process, an absence prediction process, a statistical process, a display process, and the like.

In the presence detection process (staying detection process), the processor 23 detects a person present on the basis of the image of the camera 1. Specifically, a presence detection area for each seat is set in advance on the image of camera 1, an image of the presence detection area is cut out from the image of camera 1, and a person is detected based on the image of the presence detection area. Determine whether the person is sitting in the seat. Note that when a person present at the table is detected, a person ID is assigned to that person.

In the prediction target extraction process, the processor 23 extracts a person to be a prediction target (target of the absence prediction process) from among the people whose presence was detected in the presence detection process. Specifically, the presence time of each person is compared with a predetermined threshold, and a person whose presence time is equal to or greater than the threshold is extracted as a prediction target.

In the leaving prediction process (leaving prediction process), the processor 23 predicts, based on the image of the camera 1, whether the person targeted for prediction will leave the table in the near future (within a predetermined time from the current time). The vacancy prediction process detects that the person targeted for prediction has performed a predetermined vacancy foreshadowing motion (leaving foreshadowing motion). A departure warning motion is an action performed by a person who is about to leave the seat (an action performed immediately before leaving the seat), and when this departure warning motion is detected, the person who performed the departure warning motion will be notified in the near future. It can be predicted that the participants will leave their seats.

Here, leaving the seat is an act of leaving the seat to leave the store. For example, actions that indicate that you will be away from your seat include putting on clothes (such as a jacket or hat), holding luggage (such as a bag), or ending handling of electronic devices (such as a laptop, smartphone, or tablet). , putting away items that were used (books, tableware, etc.), getting up from a seat, etc. Note that the leaving-seat precursor motion is a motion performed by a person who is about to leave his or her seat, and can be distinguished from a motion performed by a person when temporarily leaving his or her seat to use the restroom or other errands while leaving behind luggage or the like.

In addition, an action anticipation technique is used for the separation prediction process. This motion prediction technique predicts motions that will be performed at a future time (when a predetermined amount of time has elapsed from the present) based on images from the past that are a predetermined period of time back to the present.

The absence prediction process includes a video extraction process, a feature extraction process, a predicted score acquisition process, and an absence determination process.

In the video extraction process, the processor 23 extracts a clip video based on the current time from the stored video. A clip video is a video included in a target period from the past a predetermined period of time back to the present. In the video extraction process, the processor 23 extracts a unit video at each time from the clip video. A unit video is a set (video) of a predetermined number (for example, six) of past captured images (frames).

In the feature extraction process, the processor 23 extracts feature information at each time from the unit video at each time. In this embodiment, spatial feature information, motion feature information, and object feature information are extracted. The spatial feature information is information regarding the positional relationship of objects appearing in the video. The motion feature information is information regarding the motion that appears in the video. Object feature information is information regarding the type of object that appears in the video. Note that any one or two of spatial feature information, motion feature information, and object feature information may be extracted.

Further, in the feature extraction process, a feature extraction model (machine learning model) generated by machine learning such as deep learning can be used. In this case, the unit video is input to the feature extraction model to obtain feature information output from the feature extraction model.

In the prediction score acquisition process, the processor 23 acquires a prediction score for each type of vacancy precursor movement (action of putting on a jacket, movement of holding a bag, etc.) based on the feature information at each time obtained in the feature extraction process. do. The prediction score is an evaluation value that represents the probability that the person has performed the vacancy foreshadowing action.

Further, in the predicted score acquisition process, a predicted score acquisition model (machine learning model) generated by machine learning such as deep learning can be used. In this case, the feature information is input to the predictive score acquisition model, and a predictive score for each type (motion class) of the departure sign behavior outputted from the predictive score acquisition model is acquired. Note that, for example, a neural network may be used as the prediction score acquisition model. Alternatively, a prediction score acquisition model may be provided for each type of feature information (spatial feature information, motion feature information, object feature information), and a prediction score may be obtained for each type of feature information.

In the away from seat determination process, the processor 23 determines whether the person to be predicted will be away from the seat in the near future (within a predetermined time from the current time) based on the predicted score for each type of the away from seat precursor behavior acquired in the predicted score acquisition process. Determine whether or not to do so. Specifically, if the highest prediction score among the prediction scores for each type of vacancy precursor behavior is equal to or higher than a predetermined threshold, it is determined that the person to be predicted will leave the vacancy in the near future. .

In addition, in the absence determination process, an absence determination model (machine learning model) generated by machine learning such as deep learning can be used. In this case, the predicted score for each type of the away-from-seat predictive motion is input to the away-from-seat determination model, and the away from-seat determination result outputted from the away-from-seat determination model is obtained. Note that, for example, a support vector machine (SVM) may be used as the absence determination model.

In the statistical processing, the processor 23 generates statistical information to be presented to the administrator. The statistical information is provided to the administrator as a reference when specifying the threshold for presence time when extracting the person to be predicted (absence prediction processing target) from among the people present. Presented.

In the display process, the processor 23 generates display information regarding the guide screen (see FIG. 8) to be displayed on the user terminal 3. Further, in the display process, the processor 23 generates display information regarding the setting screen (see FIG. 9) to be displayed on the administrator terminal 4.

Next, the database of people present managed by the server 2 will be explained. FIG. 4 is an explanatory diagram showing the registered contents of the person present database managed by the server 2. As shown in FIG.

The server 2 performs a process of detecting a person present at the seat (presence detection process) based on the image of the camera 1. In the presence detection process, when a person present is detected, a person ID is assigned to that person, and the person ID and presence start time are registered in the presence person database. In addition, the server 2 calculates the presence time (elapsed time from the presence start time to the current time) for each person present, and registers the presence time in the presence person database. Note that in this example, the current time is 13:50.

Further, the server 2 performs a process (prediction target extraction process) of extracting a person who is a prediction target (target of the absence prediction process) from among the people whose presence was detected in the presence detection process. In the prediction target extraction process, the presence time of each person is compared with a predetermined threshold with reference to the presence person database, and a person whose presence time is equal to or greater than the threshold is extracted as a prediction target. In this example, since the person IDs are 11 and 12 and their presence time is equal to or greater than the threshold (30 minutes), these persons are extracted as prediction targets.

Next, the departure sign operation will be explained. FIG. 5 is an explanatory diagram illustrating an example of the type of leaving-seat-predicting motion and the predicted score obtained for each type of leaving-seat-predicting motion.

The server 2 performs a process of acquiring a prediction score representing the probability that the person performed the movement (prediction score acquisition process) for each type of leaving-the-seat precursor movement (motion class).

The leaving-the-seat-predicting motion includes the motion of putting on a jacket. In addition to this, actions of putting on clothing, such as putting on a hat, are included in the leaving-predicting actions. Further, the leaving-the-seat-predicting motion includes the motion of holding a bag. In addition to this, the motion of holding baggage such as a cart is included in the motion that indicates leaving the seat. Further, the leaving-the-place warning motion includes an action of closing a laptop computer, an action of putting the laptop computer into a bag, and an action of putting a smartphone into a bag. In addition to this, actions for terminating the handling (use) of various electronic devices are included in the leaving-predictive actions. Further, the leaving-the-seat-predicting motion includes the motion of closing a book. In addition, actions such as stacking tableware and other actions to put away items that have been used are included in the leaving-predicting actions. Further, the leaving-the-seat-predicting motion includes the motion of putting a book into a bag. In addition to this, the action of storing various items placed on the table into a bag is included in the leaving-predicting action. In addition, if a person leaves the seat and the type of behavior that indicates the person leaves the seat has not been specified, a new type can be set by storing the behavior immediately before the person leaves the seat and performing additional learning. It may be added as a type of seat indication motion.

Next, the leaving prediction process performed by the server 2 will be explained. FIG. 6 is an explanatory diagram showing an overview of the seat absence prediction process.

The server 2 performs a process of predicting whether or not the person to be predicted will leave the seat in the near future (within a predetermined time from the current time) (leaving prediction process). In the example shown in FIG. 6, time t ₁₄ is the current time, and time t ₁₅ is the predicted time. α is a parameter that determines how far in the future from the current time the time is predicted, and the predicted time is set according to α. For example, if α=10s, predict 10 seconds ahead.

The server 2 first performs a process of cutting out a clip video based on the current time from the stored video, and extracting unit videos V ₁ to V ₁₄ at each time t ₁ to t ₁₄ from the clip video ( video extraction processing). A clip video is a video included in a target period from the past a predetermined period of time back to the present. The unit video V _t at time t is a set (video) of a predetermined number (for example, six) of past captured images (frames) with respect to time t.

Next, the server 2 performs a process of extracting feature information at each time t ₁ to t ₁₄ from the unit videos V ₁ to V ₁₄ at each time t ₁ to t ₁₄ (feature extraction process).

Next, the server 2 performs a process of acquiring a prediction score for each type of departure sign motion as a prediction result based on the characteristic information at each time t ₁ to t ₁₄ (prediction score acquisition process).

In the example shown in FIG. 6, first, from time t ₁ to time t ₅ , only the process of extracting feature information at each time from unit videos V ₁ to V ₅ at each time is performed. At time _t6 , feature information is extracted from unit video _V6 , and _the prediction result at time t6 is calculated based on the feature information at each time extracted from unit video _{V1 to V6} at times t1 to _t6 . is obtained. Thereafter, prediction results for times t ₇ to t ₁₄ are obtained in the same manner. At this time, at each time, the previous prediction result is sequentially updated, and the prediction result at that time is acquired.

Next, the server 2 determines whether the person to be predicted will leave the seat in the near future (within a predetermined time from the current time), based on the prediction score for each type of vacancy precursor motion, which is the prediction result at the current time _t14 . A process of determining whether or not to do so is performed (seating-away determination process). In the example shown in FIG. 6, based on the prediction result at time _t14 , it is determined whether the person to be predicted will leave the seat at time _t15 .

Next, the procedures of the presence detection process, the prediction target extraction process, and the absence prediction process performed by the server 2 will be described. FIG. 7 is a flow diagram illustrating an example of the procedures of the presence detection process, the prediction target extraction process, and the absence prediction process. Note that these flows are executed periodically.

As shown in FIG. 7A, in the presence detection process, the server 2 first acquires an image captured by the camera 1 (ST101). Next, the server 2 cuts out an image of the presence detection area for each seat from the image taken by the camera 1, and detects a person present in the presence detection area based on the image of the presence detection area. detection processing) (ST102).

Next, the server 2 determines whether a person present in the presence detection area is present (presence determination process) (ST103). At this time, if the person in the presence detection area is detected continuously a predetermined number of times or more, it is determined that the person present in the presence detection area is present. For example, if images captured by camera 1 are acquired at 10 second intervals, and a person is detected in the presence detection area six times in a row, that is, a person exists in the presence detection area for one minute continuously. is determined to be present. In addition, in the presence determination process, it is possible to use a classifier that has previously learned two classes, a presence state and an absence state, by deep learning. During the presence determination process, the discriminator calculates the likelihood (likelihood) of the presence state and the likelihood of the absence state for the image of the presence detection area, and compares the two likelihoods. By doing so, it is possible to determine whether the person is present or absent.

Here, if it is determined that the person present in the presence detection area is present (Yes in ST103), the server 2 stores the person ID assigned to the person determined to be present and the presence start time. It is registered in the person present database (ST104).

As shown in FIG. 7B, in the prediction target extraction process, the server 2 first selects a person to be determined from among the people whose presence was detected in the presence detection process (ST201).

Next, the server 2 determines whether the presence time of the person targeted for determination is equal to or greater than a threshold value (ST202).

Here, if the presence time of the person targeted for determination is equal to or greater than the threshold value (Yes in ST202), the server 2 sets the person as a prediction target (target for the absence prediction process) (ST203).

As shown in FIG. 7(C), in the separation prediction process, the server 2 first extracts a clip video based on the current time from the stored video, and extracts a unit video of each time from the clip video. Extract (video extraction process) (ST301).

Next, the server 2 uses a feature extraction model (machine learning model) to extract feature information from the unit video (feature extraction process) (ST302).

Next, using a prediction score acquisition model (machine learning model), the server 2 acquires a prediction score for each type of departure sign behavior based on the characteristic information of each time included in the target period (prediction score acquisition processing) (ST303).

Next, the server 2 determines whether or not the person to be predicted will leave the seat in the near future (within a predetermined time from the current time) based on the predicted score for each type of the leave-away sign motion (leave the seat). determination process) (ST304).

Here, if it is determined that the person to be predicted will leave the seat in the near future (Yes in ST304), that person is set as a measurement target for the predicted number of people leaving the seat (ST305).

Next, the guide screen 31 displayed on the user terminal 3 will be explained. FIG. 8 is an explanatory diagram showing the guide screen 31 displayed on the user terminal 3.

On the user terminal 3, a guide screen 31 is displayed that guides the user to the congestion situation of the store. By viewing the guide screen 31, the user can grasp the current congestion situation of the store he/she wishes to use.

The guide screen 31 is provided with a connection destination display section 32 . By accessing the server 2 by specifying the viewing address (URL) of the specific store to be accessed on the user terminal 3, a guide screen 31 (web page) will be displayed on the user terminal 3, and the connection destination will be displayed. The address for viewing the store is displayed on the display unit 32.

Further, the guide screen 31 is provided with a facility information display section 33. The facility information display section 33 displays information regarding the target facility (for example, name, location, etc.). Facility information is registered in advance by the administrator. Note that the facility information may include a privacy protection image (person mask image) generated from a live image inside the store.

Further, the guidance screen 31 is provided with a display section 34 for displaying the number of people present and a display section 35 for displaying the number of people expected to be absent. The number of people present display section 34 displays the number of people present, that is, the number of people present in the facility. The predicted number of absentees display section 35 displays the predicted number of absentees, that is, the number of people present in the facility who are predicted to be absent in the near future. Thereby, the user can correct and judge the degree of congestion of the facility based on the number of people present by considering the predicted number of people who will be absent. Specifically, even in a crowded state where there are many people in the seats, if the predicted number of people leaving the seats is large, the user can judge that the congestion will be alleviated soon. The number of people present is acquired by presence detection processing. The predicted number of absentees is obtained by the absentee prediction process.

Further, the guide screen 31 is provided with an update time display section 36. In the update time display section 36, the number of people present and the predicted number of people absent are updated based on the update time of the information displayed on the guide screen 31, that is, the latest processing results of the presence detection process and the absence prediction process. displayed. This allows the user to understand how new the displayed information (number of people present and expected number of people absent) is.

Note that the display information is updated periodically at predetermined intervals (for example, every 15 minutes). Furthermore, the display information may be updated at intervals longer than the intervals (for example, one minute) at which the presence detection process and the absence prediction process are executed. Further, the frequency of updating the display information may be changed depending on the characteristics of the facility and the congestion situation.

Further, the guide screen 31 is provided with a display section 37 for displaying the number of people currently viewing the page. The number of users currently browsing display section 37 displays the number of users currently browsing, that is, the number of users currently viewing the guide screen 31. As a result, the user can correct and judge the degree of congestion of the facility based on the number of people present by considering the number of people currently viewing the site in addition to the number of people expected to be absent. Specifically, the user can judge that, for example, if the number of people expected to leave the site is large but the number of people currently browsing is large, the congestion will not be alleviated much as users visit one after another. The number of people currently browsing is acquired based on the number of user terminals 3 that have transmitted the display information of the guide screen 31 in response to access from the user terminals 3 .

Next, the setting screen 41 displayed on the administrator terminal 4 will be explained. FIG. 9 is an explanatory diagram showing the setting screen 41 displayed on the administrator terminal 4.

The administrator terminal 4 displays a setting screen 41 for setting conditions for processing performed by the server 2 in response to operations by the administrator. The setting screen 41 displays information regarding the presence time when extracting a person who is a prediction target (target of the absence prediction process) from among the people present in the prediction target extraction process based on the length of the presence time. A threshold value can be specified by the administrator. Further, on the setting screen 41, the administrator is presented with statistical information that can be used as a reference when the administrator determines a threshold value regarding the presence time.

The setting screen 41 is provided with a connection destination display section 42 . By accessing the server 2 by specifying the address (URL) for settings of a specific target store on the user terminal 3, the setting screen 41 (web page) will be displayed on the user terminal 3, and the connection destination will be displayed. The address for setting the store is displayed on the display unit 42.

Further, the setting screen 41 is provided with a statistical graph display section 43. The statistical graph display section 43 displays a histogram representing the distribution of attendance times as a statistical graph. In the histogram, the vertical axis represents the number of people (frequency), and the horizontal axis represents the range of attendance time (less than 15 minutes, 15 minutes or more but less than 30 minutes, 30 minutes or more but less than 45 minutes, 60 minutes or more). The statistical graph is generated based on statistical information obtained through statistical processing.

Further, the setting screen 41 is provided with a display period designation section 44 . In the display period designation section 44, the administrator can input the display period, that is, the target period (start date and time and end date and time) of the statistical graph to be displayed on the statistical graph display section 43. Note that the administrator may be able to select a preset period (for example, the entire past period, the past one year, the past three months, etc.).

Further, the setting screen 41 is provided with a recommended value display section 45. The recommended value display section 45 displays a recommended value regarding the threshold of presence time. In the server 2, a recommended value regarding the threshold value is calculated by statistical processing regarding the past presence time of each person stored in the storage unit 22. For example, the most frequent value of the attendance time (the attendance time when the number of people (frequency) is maximum) is set as the recommended value. Further, the average value of the attendance time is set as the recommended value.

Further, the setting screen 41 is provided with a threshold input section 46 and an OK button 47. In the threshold input section 46, the administrator can input a threshold for the presence time. The administrator can input an independently determined threshold value with reference to the statistical graph displayed on the statistical graph display section 43. Further, the administrator can adopt the recommended value of the threshold value displayed on the recommended value display section 45. For example, if the threshold value input section 46 is left blank and the administrator operates the OK button 47, the server 2 sets the threshold value using the value displayed on the recommended value display section 45. Thereby, it is possible to set a threshold according to the usage status of the target store (facility). In addition, for statistical information, multiple statistical graphs can be displayed for various periods, such as morning, afternoon, weekdays, holidays, etc., and recommended threshold values can be displayed and threshold input can be performed for each period. It is also possible to accept the request and set a threshold that reflects the situation in each period.

As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in the present disclosure is not limited to this, and can also be applied to embodiments in which changes, replacements, additions, omissions, etc. are made. Furthermore, it is also possible to create a new embodiment by combining the components described in the above embodiments.

The stay situation presentation device and the stay situation presentation method according to the present invention present users with more reliable information regarding crowding situations by predicting who will leave the facility in the near future based on the movements of each person. The present invention is useful as a stay situation presentation device and a stay situation presentation method that generates information about a person's stay situation in a facility based on images taken of the facility and presents it to the user.

1 Camera 2 Server (stay status presentation device)
3 User terminal (terminal device)
4 Administrator terminal (terminal device)
22 Storage unit 23 Processor 31 Guide screen 34 Number of people in attendance display unit 35 Predicted number of people away from seats display unit 36 Update time display unit 37 Number of people currently viewing display unit 41 Setting screen 46 Threshold input unit

Claims (8)

A stay situation presentation device that uses a processor to generate display information regarding the degree of congestion of a facility based on a video shot of a person's stay area in the facility,
The processor includes:
Detecting a person staying in the stay area,
Extracting people whose staying time is equal to or more than a predetermined threshold from among the people staying there as prediction targets;
Predicting the departure of the person targeted for prediction based on a predetermined departure sign behavior,
A stay status presentation device characterized in that the display information is generated including at least the number of people staying and the number of people expected to leave. The stay area is a presence detection area that detects a person present,
2. The stay status presentation device according to claim 1, wherein the leaving sign motion is a leaving sign motion performed by a person present in the seat immediately before leaving the seat. The processor includes:
The stay status presentation device according to claim 1, wherein the display information is transmitted to the terminal device in response to access from a terminal device operated by a user. The processor includes:
2. The stay status presentation device according to claim 1, wherein the threshold value is set in response to an operation for specifying the threshold value performed by a facility manager. The processor includes:
Based on the latest processing results of the process of detecting the person staying and the process of predicting departure, the number of people staying and the time when the number of people predicted to leave are updated. The stay status presentation device according to claim 1, wherein the device generates display information. The processor includes:
The stay according to claim 1, wherein the display information including the number of users who are viewing the display information transmitted to the terminal device in response to access from a terminal device operated by a user is generated. Situation presentation device. The move-out warning motion is characterized in that it is at least one of the following actions: putting on clothes, holding luggage, finishing handling electronic devices, putting away items, and storing items in a bag. The stay status presentation device according to claim 1. A stay situation presentation method in which a processor executes a process of generating display information regarding the degree of congestion of a facility based on a video shot of a person's stay area in the facility, the method comprising:
Detecting a person staying in the stay area,
Extracting people whose stay time is equal to or greater than a predetermined threshold from among the people staying there as prediction targets;
Predicting the departure of the person targeted for prediction based on a predetermined departure sign movement,
A stay situation presentation method characterized by generating the display information including at least the number of people who are staying and the number of people who are expected to leave.

Images