JP2022178389A - Information processing system, control method thereof, electronic device, information processing device, program, and recording medium - Google Patents

Abstract

To enable a user to more effectively utilize a space to be used.SOLUTION: An information processing system includes acquisition means for acquiring information indicating a staying state of a detection target in a specific space, receiving means for receiving a specific instruction from a user, and display control means for performing control to display, on a display object representing the specific space, a staying state of the detection target for each position in the specific space from when the specific instruction is received until the latest display is updated, on the basis of the information indicating the staying state which is acquired by the acquisition means.SELECTED DRAWING: Figure 3

Description

The present invention relates to a technique for performing display based on the staying state of a detection target in space.

Techniques have been proposed for notifying a situation in which cleaning is required. In Patent Document 1, by monitoring the signal of a human body detection sensor, it is possible to check the number of users of the toilet room, predict the dirtiness even in cases other than regular cleaning, and clean it. is suggested to be kept clean. More specifically, when 100 people use the toilet, the manager outputs a cleaning instruction and a check signal for consumables from the terminal device to the terminal device in the toilet room, and performs cleaning work or replenishes consumables. etc. is proposed. In addition, it is proposed to notify the toilet congestion status and empty toilets.

Also, there is known a technique for generating a heat map image that visualizes a person's activity status in order to analyze the usage status of a store. In Patent Document 2, temporal statistical processing is performed on the position information of the person detected from the video of the monitoring area, statistical information on the staying state of the person is acquired according to the setting of the target period of the statistical processing, and statistical information is obtained. It has been proposed to generate a heat map image that visualizes the information.

Japanese Patent Application Laid-Open No. 2003-242230 JP 2016-48834 A

In recent years, due to epidemics of infectious diseases, etc., it is becoming increasingly important to use a clean environment free from stains such as adhesion of droplets.

In the prior art described in Patent Document 1, users of the toilet (the space to be used) can know the current congestion status and availability of the toilet, but do not know where people have stayed so far. Therefore, the user cannot avoid the position where the person has stayed until then. In addition, it is not possible for the cleaner to perform efficient and/or reliable cleaning by focusing on cleaning the positions where people have stayed so far.

In the prior art described in Patent Document 2, only a specific time period of stay status is known, and cleaning and dirt status are not taken into consideration. It is not possible to know where the Therefore, the user cannot use the system without avoiding a position where a person may stay and be contaminated. In addition, it is not possible for the cleaner to perform efficient and/or reliable cleaning by focusing on cleaning the positions where people have stayed so far.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a technology that enables a user to more effectively utilize a space to be used.

In order to solve the above problems, the information processing system of the present invention includes:
Acquisition means for acquiring information indicating the residence status of a detection target in a specific space;
receiving means for receiving specific instructions from the user;
Based on the information indicating the stay status acquired by the acquisition means, stay status by position in the specific space of the detection target from the reception of the specific instruction to the latest display update, and display control means for controlling to display on a display object representing the specific space.

ADVANTAGE OF THE INVENTION According to this invention, a user can utilize the space of utilization object more effectively.

It is a system configuration diagram of the present embodiment. 2 is a hardware block diagram of terminal device 101 and server device 105. FIG. It is a display example of a cleaning assistance screen. 8 is a flowchart of cleaning support screen display processing in the terminal device 101. FIG. 10 is a flow chart of staying situation analysis processing in the server device 105. FIG. (a) is a photographed image. (b) is a person region extraction image. (c) Stay map. (d) A composite image of the captured image and the stay map. (e) A composite image of a schematic diagram of a room and a stay map.

Preferred embodiments of the present invention will be described below with reference to the drawings.

In the present embodiment, a cleaning support (contamination status guide) system will be described, in which the need for cleaning is displayed based on the staying status of people who have used the room, so that the cleaning staff can perform accurate cleaning at a suitable timing. As a countermeasure against infectious diseases, it is effective to wipe off droplets, sterilize, disinfect, and clean the areas where droplets emitted from a person's mouth adhere. For example, in rooms (shared spaces) such as conference rooms and dining rooms used by unspecified people, it is preferable that the next person uses them after cleaning the droplets from the previous person. . In particular, it is desirable that the position where the droplets from the person who used it before adhered is accurately cleaned. However, it is desirable to perform cleaning as efficiently as possible because of the limited resources available for cleaning, and the increased time spent cleaning reduces the available time for the room. In view of such circumstances, in the cleaning support system of this embodiment, by showing the cleaning staff where the users of the room have stayed since the previous cleaning, the cleaning staff can perform more efficient cleaning and/or cleaning. Or enable more accurate cleaning. In addition, the user of the room should be able to use the room while avoiding the contaminated area. Retention points are locations where users of rooms that have taken appropriate measures against infectious diseases, such as masks, simply pass through to move (i.e., locations that exist but are not retained), where droplets adhere. This is because the probability is low and the need for cleaning and avoidance of use is low.

FIG. 1 shows a system configuration diagram of an information processing system (cleaning support system, stay situation guide system) of the present embodiment. Rooms 102a to 102c are examples of shared spaces, and are schematic diagrams showing the shape of the room, the layout of tables (desks) illustrated by rectangles, and the layout of seats illustrated by circles. Doors 103a to 103c are provided in the rooms 102a to 102c, respectively. Cameras 104a to 104c are installed in the rooms 102a to 102c, respectively. The cameras 104a to 104c are network cameras (surveillance cameras), one for each room or a plurality of network cameras for each room, capable of photographing the entire room, and function as sensors for detecting the presence of people in each room. do. Although three rooms are described as an example in this embodiment, the cleaning support system of this embodiment can be applied to any number of rooms, from one room to a plurality of rooms.

The terminal devices 101a to 101c are electronic devices for displaying the staying state of people in the rooms 102a to 102c. Stationary PCs (personal computers) are shown in FIG. 1 as the terminal devices 101a to 101c. When the terminal devices 101a to 101c are of the stationary type, it is preferable that the person who uses each room or the cleaning staff can check the display before entering the room or immediately after entering the room. It is preferable to install near the doorway of each room. In that case, the terminal devices 101a to 101c may correspond to the rooms 102a to 102c, respectively, and may be configured to exclusively display the stay status of the corresponding rooms. Note that the terminal devices 101a to 101c may be portable electronic devices such as smartphones and tablet terminals instead of stationary devices. In this case, the terminal devices 101a to 101c do not have to be associated with each room, and the number of terminals may be set regardless of the number of rooms, and may be mobile terminals owned by cleaning staff or users.

The server device 105 is an electronic device that can be connected to the cameras 104a to 104c by wire or wirelessly, and can acquire images captured by the cameras 104a to 104c. Further, the server device 105 can be connected to the terminal devices 101a to 101c by wire or wirelessly. Send to A connection method between the server device 105 and the cameras 104a to 104c and a connection method between the server device 105 and the terminal devices 101a to 101c may be any method such as connection via a LAN (Local Area Network) or the Internet. Although one server device 105 is illustrated, a plurality of devices on the network may work together to execute the functions of the server device 105 .

FIG. 2 shows a hardware block diagram of each of the terminal devices 101a to 101c and the server device 105. As shown in FIG.

FIG. 2 shows an example of the configuration of an electronic device 200 as an example of each of the terminal devices 101a to 101c and the server device 105. As shown in FIG.

2, a CPU 201, a memory 202, a nonvolatile memory 203, an image processing unit 204, a display 205, an operation unit 206, a recording medium I/F 207, an external I/F 209, and a communication I/F 210 are connected to an internal bus 250. ing. Each unit connected to the internal bus 250 is capable of exchanging data with each other via the internal bus 250 .

The memory 202 is, for example, a RAM (a volatile memory using a semiconductor element, etc.). The CPU 201 controls each unit of the electronic device 200 using the memory 202 as a work memory, for example, according to a program stored in the nonvolatile memory 203 . The nonvolatile memory 203 stores image data, audio data, other data, various programs for the CPU 201 to operate, and the like. The nonvolatile memory 203 is composed of, for example, a hard disk (HD) or ROM.

Under the control of the CPU 201, the image processing unit 204 processes images stored in the nonvolatile memory 203 and the recording medium 208, video signals obtained via the external I/F 209, and images obtained via the communication I/F 210. , various image processing is performed on the captured image. The image processing performed by the image processing unit 204 includes A/D conversion processing, D/A conversion processing, image data encoding processing, compression processing, decoding processing, enlargement/reduction processing (resize), noise reduction processing, color conversion, and so on. processing, etc. The image processing unit 204 may be composed of a dedicated circuit block for performing specific image processing. Further, depending on the type of image processing, the CPU 201 can perform image processing according to a program without using the image processing unit 204 .

The display 205 displays images, GUI screens that form a GUI (Graphical User Interface), and the like, under the control of the CPU 201 . CPU 201 controls each unit of electronic device 200 to generate a display control signal according to a program, generate a video signal to be displayed on display 205 , and output the video signal to display 205 . A display 205 displays an image based on the output image signal. Note that the electronic device 200 itself includes an interface for outputting a video signal for display on the display 205, and the display 205 may be configured by an external monitor (such as a television).

An operation unit 206 is an input device for receiving user operations, including a character information input device such as a keyboard, a pointing device such as a mouse and a touch panel, buttons, dials, joysticks, touch sensors, and touch pads. It should be noted that the touch panel is an input device that is superimposed on the display 205 and configured in a two-dimensional manner so as to output coordinate information according to the touched position.

A recording medium I/F 207 can be loaded with a recording medium 208 such as a memory card, a CD, or a DVD. I do. The external I/F 209 is an interface for connecting with an external device by a wired cable or wirelessly and inputting/outputting a video signal and an audio signal. The communication I/F 210 is an interface for communicating with an external device, the Internet 111, etc., and transmitting/receiving various data such as files and commands.

The camera unit 112 is a camera unit including an image sensor (image sensor) including a CCD or CMOS element for converting an optical image into an electric signal.

FIGS. 3A to 3E show display examples of cleaning support screens on the display 205 of the terminal device 101 (symbols of the terminal devices 101a to 101c; the same shall apply hereinafter). Common display elements in FIGS. 3A to 3E are given the same reference numerals.

FIG. 3A is a display example immediately after the cleaning support screen is started. When the cleaning assistance application software in the terminal device 101 is started or the cleaning assistance screen is opened by the WEB browser, the cleaning assistance screen 300 is displayed. The cleaning support screen 300 includes an area 301 for displaying a list of rooms, and an area 310 for displaying details such as the stay status of the selected room/whether cleaning is necessary or not. Area 301 displays selection items 302a to 302c as a list of selectable rooms. Selection items 302a-302c are options for selecting rooms 102a-102c, respectively. Selectable items 302a to 302c display outline information such as the corresponding room name (identification information for the rooms illustrated as rooms A to C), room capacity, and whether or not cleaning is necessary. The user can select one of the selection items 302a to 302c using a mouse, touch panel, or the like, and when one of the selection items 302a to 302c is selected, detailed information on the selected room is displayed in the area 310. . An end button 304 is an operation icon for closing (hiding) the cleaning assistance screen.

FIG. 3B is a display example when a selection item 302a (corresponding to room A, room 102a) is selected. A selection frame 303 is a selection cursor indicating that the selection item 302a is selected. A dialog 311, a stay status display area 312, and a usage history 313 are displayed in the area 301, and detailed information corresponding to the room A is displayed. Also, a cleaning completion button 314, which is an operation button for instructing that the cleaning of the room A has been completed, is displayed. In the dialog 311, the photographing date and time of the photographed image (image photographed by the camera 104a) of the room A displayed in the stay status display area 312, the room name (room identification information), the current number of people in the room, the current Cleaning necessity information 311a indicating whether cleaning is necessary or not is displayed. A captured image of the room A (image captured by the camera 104a) and the staying status of the person are displayed in the staying status display area 312 . The usage history 313 displays, as the usage history of the room A, the actual number of users of the room A for each time period, and information indicating the time when the cleaning completion button 314 was operated. Note that FIG. 3B is an example of the display immediately after the system is started at 7:00 in the morning. is not operated. Therefore, the cleaning necessity information 311a indicates that cleaning is not required (that is, no indication that cleaning is required), and the stay status display area 312 does not display any stay status (that is, that there is no stay). is displayed). In addition, the usage history 313 does not display that there was a record of the number of users (that is, it displays that the number of users was 0), and information indicating the time when the cleaning completion button 314 was operated is displayed. not

FIG. 3(c) shows a case where the selection item 302a is selected as in FIG. 3(b), but the cleaning assistance at a point in time (for example, around 11:01) after that in FIG. 3(b). It is a display example of a screen. At this point, display elements 321 to 330 representing the stay status are displayed in the stay status display area 312 superimposed on the captured image of the room A (the latest image captured by the camera 104a). The display elements 321 to 330 are display elements that have different display forms (colors in this embodiment) depending on the stay status. Colors by stay status will be described later using the colored stay map in FIG. 5(c).

Display elements 321, 330, and 331 are displayed in a color indicating that there is a retention that requires cleaning. The user (cleaner or the next user of the room) sees this display and decides that the positions of the display elements 321, 330, and 331 in the room A should be cleaned or should not be used (for example, (e.g., use a seat in the

Of the display elements 321 to 330, for the display elements other than the display elements 321, 330, and 331, the person who was in the room A was present but did not stay, or even if he stayed, it was very short. It is the time and indicates the place where the possibility of contamination by droplets is low. This indicates that although the person was detected at that position based on the person's detection status, the person simply passed by during the process of movement, and did not remain so long as to require cleaning. The user (cleaner or the next user of the room) sees this display, and in room A, the priority of cleaning and the priority of avoiding use is the second highest after the positions of the display elements 321, 330, and 331. I know the location. Conversely, it can also be regarded as a position where there is a history of the presence of a person, but where there is a low need for cleaning or where there is a low need for avoiding use. Of the display elements 321 to 330, the display elements other than the display elements 321, 330, and 331 are not essential for the user to determine whether cleaning is necessary or to determine the position to avoid, so they may not be displayed. .

Further, in the cleaning necessity information 311a in FIG. 3(c), it is displayed that cleaning is necessary. This is a display prompting the cleaning staff to clean, and from this, the cleaning staff can understand that it is better to clean. The usage history 313 displays the number of users by time zone up to that day. This also serves as reference information for the user to determine whether cleaning is necessary or whether to avoid using the room. Also, an indicator 315 indicating that the cleaning completion button 314 was operated once (cleaning was performed) at 7:15 is displayed. The indicator 315 indicates the time when the cleaning completion button 314 was operated by displaying the time by numbers and by the position of the vertical bar-shaped indicator superimposed on the usage history for each time period. Either one may be used to display the time.

In FIG. 3C, the fact that the selected items 302a and 302b also need cleaning is indicated by a character string ("cleaning required") and the color of the selected items 302a and 302b (cleaning required). The selected item 302c is displayed in an identifiable display format). As a result, even in the area 301 that displays the room list, it is possible to know whether or not there is a stay of people and whether or not cleaning is required. In particular, knowing that the room B indicated by the selection item 302b that is not selected in the selection frame 303 also needs cleaning, it is understood that it is better to select the selection item 302b (room B) and check the details, which is preferable. .

FIG. 3(d) shows a point in time (for example, 12:03) after a while after the state of FIG. ) is a display example of the cleaning support screen for the room A. The image captured by the camera 104a displayed in the stay status display area 312 does not show a person, but the position where there has been a stay to the extent that cleaning is required is also displayed in FIG. 3(c). Elements 321, 330 and 331 remain visible in FIG. 3(d). On the other hand, among the display elements 321 to 330 displayed in FIG. 3C, display elements other than the display elements 321, 330, and 331 are not displayed. Display elements that were displayed in positions where it was not determined that there was a level of retention that required cleaning will change over time to a display mode (color) with a lower degree of retention, and eventually disappear. becomes. As a result, the display elements remain displayed up to the point where there is no retention to the extent that cleaning is required, and it is possible to prevent the retention status display area 312 from being filled with the display elements and becoming difficult to see. Further, in FIG. 3D, the cleaning necessity information 311a continues from FIG. be.

FIG. 3(e) is a display example when room A is cleaned after the state of FIG. 3(d) and the cleaning completion button 314 is operated. When the cleaning completion button 314 is operated, the display elements 321, 330, and 331 indicating positions requiring cleaning in the stay status display area 312 are hidden. That is, when the cleaning completion button 314 is operated while the display elements 321, 330, and 331 indicating that there is a stagnation are displayed, the display indicating that there is a stagnation is initialized, and a state in which there is no stagnation. (cleaning not required state). Further, the cleaning necessity information 311a is updated (initialized) to indicate that cleaning is not required, and the selection item 302a is also updated (initialized) to display (character string, color) indicating that cleaning is not required. Further, the usage history 313 displays an indicator 316 indicating that the cleaning completion button 314 was operated (cleaning was performed) at 12:10. After that, the stay status of room A is continuously monitored and displayed, and regarding the stay status after 12:10, the need for cleaning and the position that needs to be cleaned (the stay position) are displayed. It will be. That is, in the stay status display area 312, for each room A, from the time when the cleaning completion button 314 was last operated to the time of the latest display update (when the display was last updated), Stay status is displayed.

FIG. 4 shows a flowchart of the cleaning support screen display process, which is the process performed by the terminal device 101 to display the cleaning support screen described with reference to FIGS. 3(a) to 3(e). This processing is realized by developing a program stored in the non-volatile memory 203 of the terminal device 101 into the memory 202 of the terminal device 101 and executing it by the CPU 201 of the terminal device 101 . When the cleaning support application software in the terminal device 101 is activated or the cleaning support screen is opened by the WEB browser, the process of FIG. 4 is started.

In S401, the CPU 201 communicates with the server device 105 via the communication I/F 210 and acquires room information regarding rooms that can be displayed in the cleaning support screen process. Specifically, the name (identification information) of the room displayed in the room list in the area 301, the number of people accommodated in each room, and the state (whether or not cleaning is required) are acquired.

In S402, the CPU 201 displays a room list in the area 301 on the display 205 based on the information acquired in S401. As a result, selection items 302a to 302c are displayed as a room list, as in the display example of FIG. 3(a).

In 403, the CPU 201 determines whether or not a room selection operation has been performed to select one of the multiple rooms displayed in the room list. When one of the selection items 302a to 302c is selected by the operation unit 206 (clicked with a mouse or touched with a touch panel), it is determined that a room selection operation has been performed. If there is a room selection operation, the process proceeds to S405; otherwise, the process proceeds to S404.

In S404, the CPU 201 determines whether or not there is an end event such as the end button 304 being operated. If there is no end event, the process proceeds to S403, and if there is an end event, the cleaning support screen is hidden, and the cleaning support screen display processing of FIG. 4 ends.

In S<b>405 , the CPU 201 communicates with the server device 105 via the communication I/F 210 and acquires detailed information and a composite image of the selected room from the server device 105 . The composite image is an image displayed in the stay status display area 312, and is a captured image of the selected room (a captured image captured by the camera of the selected room among the cameras 104a to 104c) and the stay status. 4 is a composite image with display elements shown. In a situation where the display element indicating the staying state is not displayed, only the captured image without the display element indicating the staying state is obtained. The detailed information acquired from the server device 105 includes the photographing date and time of the photographed image used for the synthesized image to be acquired, room identification information, the number of people in the room at the acquired photographing date and time (photographed images are acquired in real time). current number of people in the room), and information on whether or not cleaning is necessary. In this embodiment, the cleaning necessity information is a cleaning required flag related to the selected room, which will be described later. Further, the detailed information acquired from the server device 105 includes information indicating the number of users of the selected room for each time period and the time when the cleaning completion button 314 was operated.

In S406, the CPU 201 performs various displays in the area 301 as detailed information of the selected room on the display 205 based on the detailed information and the synthesized image acquired in S405. That is, the dialog 311, the stay status display area 312, and the usage history 313 are displayed as described with reference to FIGS. 3(b) to 3(e). The displayed contents differ depending on the situation (contents of the information acquired in S405), as described with reference to FIGS. 3(b) to 3(e).

In S407, the CPU 201 determines whether or not the cleaning required flag acquired in S405 is 1, that is, whether information indicating that cleaning is required has been acquired. If the cleaning required flag is 1 (cleaning required), the process proceeds to S409; otherwise, the process proceeds to S408.

In S408, the CPU 201 displays that the selected room does not require cleaning. As a result, as shown in FIGS. 3B and 3E, the cleaning necessity information 311a displays that cleaning is unnecessary.

In S409, the CPU 201 displays that the selected room needs cleaning. Accordingly, as shown in FIGS. 3(c) and 3(d), the necessity of cleaning is displayed in the cleaning necessity information 311a.

In S410, the CPU 201 determines whether or not the cleaning completion button 314 has been operated (there has been an operation to issue a cleaning completion instruction, which is an instruction to the effect that cleaning has been completed). When the cleaning completion button 314 is clicked with the mouse included in the operation unit 206 or touched via the touch panel, it is determined that the cleaning completion instruction has been operated, and the cleaning completion instruction is accepted. If there is an operation to instruct completion of cleaning, the process proceeds to S411; otherwise, the process proceeds to S412.

In S411, the CPU 201 communicates with the server device 105 via the communication I/F 210, and notifies the server device 105 that there has been a cleaning completion instruction operation.

In S412, the CPU 201 determines whether or not information update is necessary. When the information is updated periodically, it is determined that the information update is necessary when the information update cycle has passed. Further, when communicating with the server device 105 via the communication I/F 210 and receiving from the server device 105 a notification (push notification) to the effect that the detailed information of the selected room and the composite image will be updated, the information is updated. is necessary. Further, it is determined that the information update is necessary when the user instructs to update the information. If it is determined that the information needs to be updated, the process advances to S405 to obtain the latest information, and the processing of S406 to S409 is performed based on the obtained information to update the display. At this time, if information indicating that the cleaning necessity of the unselected rooms is to be updated is obtained, the display of the selection items 302a to 302c is also updated based on the obtained information. In this embodiment, an example of automatically updating the display periodically (regularly) or based on a push notification from the server device 105 has been described. It may be updated only when For example, the update may be performed in response to an update instruction from the user at the timing when the meeting in the target room is over, the timing when the room is about to be entered, or the timing just before cleaning. By doing so, the communication load can be reduced compared to the case of automatic updating.

In S413, the CPU 201 determines whether or not there has been an operation to select another room. If there is an operation to select a room different from the currently selected room among the selection items 302a to 302c, the process advances to S405 to display the newly selected room. If another room has not been selected, the process proceeds to S414.

In S414, the CPU 201 determines whether or not there is an end event such as the end button 304 being operated. If there is no end event, the process proceeds to S410, and if there is an end event, the cleaning support screen is hidden, and the cleaning support screen display processing of FIG. 4 ends.

FIG. 5 shows a flow chart of the staying state analysis process, which is the process performed by the server device 105 to display the cleaning support screen described with reference to FIGS. 3(a) to 3(e). This process is realized by deploying a program stored in the non-volatile memory 203 of the server device 105 in the memory 202 of the server device 105 and executing it by the CPU 201 of the server device 105 . When the start event of the staying state analysis process occurs in the server device 105, the process of FIG. 5 is started. This start event is different from the operation of opening the cleaning assistance screen on the terminal device 101, and is generated by, for example, the operator of the server device 105 performing a start instruction operation, automatic activation at a predetermined time every morning, or the like.

In S<b>501 , the CPU 201 initializes various variables held in the memory 202 . The variables to be initialized here include:
- Room variable R: A variable indicating which of a plurality of rooms is to be processed. Initialized to 1.
Staying degree S: A variable that differs for each block obtained by dividing the image of the monitored area of each room (the image captured by the cameras 104a to 104c) into a plurality of blocks. 0≦S≦1, where 0 is a state in which there is no retention, and 1 is a state in which there is retention to the extent that cleaning is required. Each room and each block has a staying degree S. In this embodiment, one pixel of the image of the monitoring target area (images captured by the cameras 104a to 104c) is defined as one block, but a wider range may be defined as a single block. The retention degree S becomes 0 when initialized.
- Cleaning required flag: A variable indicating whether or not the room needs to be cleaned. A value of 1 indicates that cleaning is required, and a value of 0 indicates that cleaning is not required. It is a variable that each room has. Initialized to 0.
• Block variable B: A variable indicating which of a plurality of blocks is to be processed. Initialized to 1.

In this embodiment, when the staying state analysis process of FIG. 5 is activated in S501, the staying degree S and the cleaning required flag are also initialized. However, the present invention is not limited to this. The residence degree S and the cleaning required flag are not initialized. It may be set. By doing so, for example, it is possible to check the staying situation and the necessity of cleaning analyzed on the previous day in the morning of the day, and make use of it for cleaning and using the room.

In S502, the CPU 201 acquires the latest captured image and information on the shooting date and time from the network camera (one of the cameras 104a to 104c) in one of the plurality of rooms, and records them in the memory 202 or the nonvolatile memory 203. .

In S503, the CPU 201 determines whether or not captured images and shooting dates and times have been acquired for all rooms. If the images for all rooms have not been acquired, the process returns to S502 to acquire the captured images and the shooting date and time for the next room. If the photographed images and photographing dates and times for all rooms have been obtained, the process advances to step S504.

In S503, the CPU 201 initializes the room variable R=1. Hereinafter, the processing of S510 to S526 is processing for each room, and is processing for the room corresponding to the room variable R.

In S510, CPU201 determines whether the notification of the completion of cleaning was received. When the cleaning completion notification is transmitted from the terminal device 101 to the server device 105 in S411 of FIG. That is, an instruction to the effect that cleaning has been completed is received from the user based on an operation performed on the terminal device 101 . If the cleaning completion notification has been received, the process proceeds to S511; otherwise, the process proceeds to S512.

In S511, the CPU 201 initializes the retention degree S of all blocks of the image of the monitoring target area of the room corresponding to the room variable R to zero. Also, the cleaning required flag is initialized to 0.

In S511, the CPU 201 performs person detection (subject detection) from the captured image corresponding to the room variable R among the captured images acquired in S502. For example, a person is detected from the photographed image shown in FIG. Through this processing, it is specified which range of the photographed image is the region of the person. Also, how many people are currently in the target room (number of people in the room) is specified. The number of people in the room is stored in the memory 202, and history information for displaying the aforementioned usage history is generated. Also, at the timing of updating the information, the usage history information and the latest number of people in the room are transmitted to the terminal device 101 as the detailed information of the selected room.

In S512, the CPU 201 generates an extracted image by extracting the area of the detected person based on the detection result of the person detection in S512. For example, an extracted image shown in FIG. 6B is generated by extracting a region of a person from the photographed image shown in FIG. 6A. In the extracted image of FIG. 6B, the white portion is the person's area (person area), and the black portion is the area other than the person (non-person area).

In S513, the CPU 201 initializes the book variable B to one. Below, S515 to S518 are processing for each block.

In S514, it is determined whether or not the retention degree S of the block corresponding to the block variable B is one. If it is 1, proceed to S523; otherwise, proceed to S516. If the staying degree S is 1, the block will continue to need cleaning until the cleaning completion notification is received, so the staying degree S will not be lowered until the cleaning completion notification is received. As a result, as described above, the display elements 321, 330, and 331 indicating the positions requiring cleaning that were displayed in FIG. 3(c) remain displayed in FIG. 3(d) as well. Moreover, the retention degree S has an upper limit of 1. Furthermore, the fact that there is a block with a retention degree S of 1 means that the cleaning required flag is already 1. Therefore, if the staying degree S is 1, the process proceeds to S523 without performing the processes of S516 to S520.

In S516, the CPU 201 determines whether or not the block corresponding to the block variable B is included in the person area of the person extraction image generated in S513. If it is determined to be included in the person area (is the person area), the process proceeds to S517; otherwise, that is, if the area is the non-person area, the process proceeds to S521.

In S517, the CPU 201 sets the retention degree S of the block corresponding to the block variable B to S+(1/m). That is, (1/m) is added to the last staying degree S. m is a value of 1 or more and is a constant. In addition, m can be adjusted by the designer, and if the predetermined condition until cleaning is required is designed to require a long residence time until cleaning is required, m is increased to shorten the residence time. In the case of requiring cleaning at , m should be reduced. In the periodical processing, if the block is a person area m times in a row, S=1 for that block, which requires cleaning. As will be described later in S522, the retention degree S may be subtracted, so there may be cases where cleaning is required even if the cleaning is not performed m consecutive times. In this embodiment, as an example, the processing of S502 to S529 is performed 30 times per minute (processing is performed at a cycle of 1/30 (minute)), and m is set to 15. In this way, if a person stays in the same place for 30 seconds, cleaning is required.

In S518, the CPU 201 determines whether S=1. If S=1 as a result of the addition in S517, the process proceeds to S519; otherwise, the process proceeds to S523.

In S<b>519 , the CPU 201 determines whether or not the cleaning required flag of the room corresponding to the room variable R is “1”. If it is 1, the process proceeds to S523, and if it is 0, the process proceeds to S520.

In S<b>520 , the CPU 201 sets the cleaning required flag of the room corresponding to the room variable R to “1”. As a result, the room corresponding to the room variable R changes from a state in which it did not require cleaning to a new state in which cleaning is required. Further, in S520, the CPU 201 notifies the terminal device 101 via the communication I/F 210 that the cleaning required flag of the room corresponding to the room variable R has become 1. As a result, the state of the corresponding room among the selection items 302a to 302c is updated to indicate that cleaning is required.

In S521, the CPU 201 determines whether or not the staying degree S=0. When the degree of retention S=0, it is the lower limit value, and since it is not decreased any more, the process proceeds to S523. If the residence degree S is not 0, the process proceeds to S522.

In S522, the CPU 201 sets the retention degree S of the block corresponding to the block variable B to S-(1/m). That is, (1/m) is subtracted from the last staying degree S. In this embodiment, the addition and subtraction of the residence degree S are performed at the same pace, but in S522, S=S−(1/n) may be set (m<n) so as to moderate the pace of subtraction. . Alternatively, the subtraction in S522 may be omitted. By doing this, regardless of the residence time of the same person, multiple people exist for a short time, respectively, and locations where the cumulative time is high (such as routes frequently passed by multiple people) need cleaning. It becomes easy to do. Therefore, more reliable cleaning becomes possible.

In S523, the CPU 201 determines whether or not the block variable B is the maximum value Bmax. If it is not Bmax, that is, if all blocks have not been processed, the process proceeds to S524, the block variable B is incremented, and the next block is processed. If it is not Bmax, that is, if the processing for all blocks is completed, the process proceeds to S525.

In S<b>525 , the CPU 201 creates a colored stay map and records it in the memory 202 . In the colored stay map, each block of the image of the monitored area is colored according to the stay degree S. FIG. FIG. 6(c) shows an example of a colored stay map. In FIG. 6(c), the colors are colored such as transparent (black portion in the drawing), blue, light blue, green, yellow, and red in ascending order of retention. A transparent portion (black portion in the drawing) corresponds to a retention degree S=0, and a red portion corresponds to a retention degree S=1. This colored stay map represents the stay status of the room corresponding to the room variable R from the last reception of the cleaning completion notification in S510 to the present (when the latest captured image is acquired).

In S526, the CPU 201 acquires the captured image acquired in S502 and the colored stay map generated in S525, and generates a composite image of these. Specifically, the latest photographed image of the room corresponding to the room variable R acquired in S502 is combined with the colored stay map generated in S525. For example, the colored stay map shown in FIG. 6(c) is superimposed on the photographed image shown in FIG. 6(a) to generate a composite image shown in FIG. 6(d). A colored retention map may be synthesized semi-transparently so that the photographed image can be confirmed. The composite image generated in this manner is transmitted to the terminal device 101 and displayed in the stay status display area 312 . By generating such a composite image, it is controlled to display the staying state for each position of the room corresponding to the room variable R on the photographed image which is the display object representing the room corresponding to the room variable R. ing. An update instruction (the push notification described in S412) may be notified to the terminal device 101 at the timing when a new composite image is generated.

In S527, the CPU 201 determines whether or not the room variable R is the maximum value Rmax. If not Rmax, that is, if processing for all rooms has not been completed, the process proceeds to S526, increments the room variable R, and proceeds to S510 to perform processing for the next room. If it is not Rmax, that is, if the processing for all rooms is completed, the process proceeds to S529.

In S529, the CPU 201 determines whether or not an end event of the stay status analysis process has occurred. For example, it can be designed so that it ends at a regular timing such as 22:00 every night. If no end event has occurred, the process advances to S502 to perform processing in the next period. In this way, the staying state analysis process is periodically repeated. When the end event is first voiced, the staying state analysis process of FIG. 5 is ended. At this time, the retention degree S and the cleaning required flag may be recorded in the non-volatile memory 203 so that they can be used when the retention state analysis processing is performed next time.

According to the present embodiment described above, the stay status display area 312 of the specific space (room) to be monitored displays the stay status of the detection target (person) by position in the room on the display object representing the space. (on the captured image). This display allows the user (user of the space, cleaner) to recognize the position in the space where the object to be detected has stayed. This allows the user to more effectively utilize the space to be used. More specifically, the user of the room can determine whether or not to use the room, and if so, which position to use, considering the position where the person stayed. For example, the room can be used in other positions by avoiding positions requiring cleaning where there is a possibility that droplets are attached. In addition, the cleaner can focus on cleaning the position where the person has stayed, and can perform cleaning more reliably and/or efficiently. With these, more reliable and/or efficient measures against infectious diseases can be taken.

In this embodiment, an example of displaying a composite image of a photographed image of a room and a colored stay map in the stay status display area 312 has been described. It is not limited to this as long as the display indicates For example, as shown in FIG. 6(e), the stay status for each position may be mapped and displayed on a schematic diagram (on a map) of the room. This is a representation of the target space (room), in which the same retention situation as in FIG. 3C is mapped and displayed on the schematic diagram of the room 102a described in FIG. Even with such a display, the same effects as those described above can be obtained.

In addition, in the present embodiment, an example was described in which the cameras 104a to 104c, which are imaging devices, are used as sensors for detecting the stay state of a person (stay state detection target). sensors may be used. For example, a temperature sensor may be used in place of or in combination with a camera to detect the position of a person based on the temperature of each position in space.

Further, in the present embodiment, an example has been described in which it is determined whether or not cleaning is required based on the stay status and is displayed. It may be determined and displayed. For example, if the latest number of people in the room detected in S512 exceeds the threshold for the number of people in the room (for example, half of the number of people in the room assuming that every other seat is used), The cleaning required flag may be set to 1 to indicate that cleaning is required regardless of the staying state.

Furthermore, a second-stage cleaning-required notification may be issued based on the number of people who are required to be cleaned based on the stay status. For example, if cleaning is required for the same number of people as the number of people in the room, if each user avoids the location where cleaning is required, all seats in that room will need cleaning. , it is assumed that it will not be possible to avoid the location where cleaning is required any more. Therefore, there is a possibility that immediate cleaning is required instead of regular cleaning, or that countermeasures such as limiting the use of the room or issuing a warning are required. Therefore, second information such as "immediate cleaning required", "unusable until cleaning", etc. is displayed, which is different from the display indicating that cleaning is required as described above with reference to FIGS. do it.

In addition, the display elements 321, 330, and 331 indicating the positions requiring cleaning described in FIGS. It may be displayed to indicate the range. In particular, considering that the droplets fly in front of the person, based on the direction of the detected person, a relatively wide range may be displayed in front of the person (the direction in which the face is facing). In order to do this, for example, in generating a person extraction image in S513, a slightly wider range than the area in which the person is detected is extracted as the person area. Furthermore, based on the direction of the detected person, a relatively wide area in front of the person (the direction in which the face is facing) may be extracted as the person area from the area where the person is detected.

In addition, considering that people who have taken measures against droplets, such as wearing a mask, are unlikely to be exposed to droplets, or that the range of droplets is small, the following measures can be taken based on whether a mask is worn or not. good. That is, among the display elements 321, 30, and 331 indicating positions requiring cleaning described with reference to FIGS. do. For example, for a person detected as a face image without a mask, a display element indicating that cleaning is required is displayed in a wider range than the range in which the person is detected as described above, particularly in a wider front range. display. On the other hand, for a person detected as a face image wearing a mask, the display element indicating that cleaning is required is not displayed in a range expanded from the range in which the person is detected, or the display element may be expanded. Also make the proportion of dilation smaller than that of a person without a mask.

In addition, considering that people who have taken measures against droplets, such as wearing a mask, are unlikely to be exposed to droplets or have a narrow range of droplets, cleaning is more necessary than those who are judged not to be wearing a mask. You may make it lengthen the residence time until it determines with. This is done, for example, so that the above-mentioned constant m is different between a block corresponding to a person determined to be wearing a mask and a block corresponding to a person determined not to be wearing a mask. should be processed. Specifically, a constant m1 may be used for a block corresponding to a person wearing a mask, and a constant m2 (<m1) may be used for a block corresponding to a person not wearing a mask.

In addition, considering that there is a high possibility that droplets will fly when you speak, sneeze, or cough, and that droplets are likely to fly far away, Different processing may be performed. That is, for a person detected to have made at least one of voice, sneeze, and cough, the residence time until it is determined that cleaning is required may be set longer than for a person who is not. In addition, for a person detected to have uttered at least one of utterance, sneeze, or cough, the range indicated by the display element indicating the position to be cleaned is greater than that of other persons when utterance, sneeze, or cough. It may be extended so that it is wider in front of the face of the Vocalization, sneezing, and coughing can be detected by detecting the shape of the face based on the captured image. may be detected.

In addition, the residence time until a person's hand is detected and the part where the hand was present is determined to require cleaning is made shorter than the part other than the hand in the person's area (constant m is set to a small value). can be Furthermore, by using object detection from the image, it is determined that the part where the hand was present and where the object was detected as having touched the object is judged to require cleaning (S=1). or it may be determined that cleaning is required even if there is no retention. By doing so, it becomes possible to preferentially avoid the use of parts touched by a person's hand, and to encourage cleaning, which makes it possible to take more reliable measures against infectious diseases.

Also, in the present embodiment, an example of notifying the completion of cleaning by the cleaning completion button 314 for each room has been described, but it is also possible to notify the completion of partial cleaning instead of the entire cleaning. For example, an instruction to complete cleaning for each seat or an instruction to complete cleaning for each area may be received. When such a partial cleaning completion notification is received, the server device 105 initializes the staying degree S of the block corresponding to the part (seat, partial area, etc.) for which the cleaning completion notification has been received, and is not initialized. By doing so, for example, it is possible to use it by cleaning only one seat in a hurry and making it available for another person to use.

Also, the cleaning completion notification may be notified according to the determination that the cleaning is completed, instead of the notification based on the user's manual operation. For example, the cleaning staff and cleaning equipment are detected by IC tags or image recognition, and when it is detected that there has been movement assumed to have cleaned the target room, it is determined that the cleaning has been completed (cleaning completed can be automatically detected).

Furthermore, when the cleaning completion notification is manually performed by the user, a method of directly operating the operation unit 206 of the terminal device 101 like the operation of the cleaning completion button 314 has been described, but the method is not limited to this. The cleaning completion notification (cleaning completion instruction) may be performed by a method other than directly operating the operation unit 206 of the terminal device 101 . For example, when the cleaning person completes cleaning, the camera of the room that has been cleaned out of the cameras 104a to 104c or the camera unit 212 provided in the terminal device 101 may be instructed to complete cleaning by making a specific gesture. . In addition, when the cleaning person completes cleaning, a specific identification image such as a QR code (registered trademark) or a bar code (which belongs to the cleaning person) is sent to the cleaned room or the camera of the terminal device 101. The cleaning completion instruction may be given by photographing a printed or displayed object. When the server device 105 detects a specific gesture or a specific identification image from the images captured by the cameras 104a to 104c, it can be accepted as a cleaning completion instruction from the user (cleaner). In addition, when the terminal 101 detects a specific gesture or a specific identification image from an image captured by the camera unit 212 provided in the terminal device 101, it accepts a cleaning completion instruction from the user (cleaner) and completes cleaning. It is possible to notify notifications. Furthermore, the cleaning completion instruction (cleaning completion notification) may be instructed (notified) by holding a specific card held by the cleaning person over a card reader or the like provided in the room for determining the room entry authority. . Furthermore, a cleaning completion instruction (cleaning completion notification) can be instructed (notified) by the user issuing a specific voice command to a microphone (not shown) provided in the room or a microphone (not shown) of the terminal device 101. It can be a thing. By doing so, it is possible to prevent a person who is not a cleaning person from erroneously notifying the completion of cleaning in a state in which cleaning is not performed. Also, contamination due to multiple persons touching the operation unit 206 of the terminal device 101 can be prevented.

In the present embodiment, an example of displaying the stay status and whether or not cleaning is required based on the stay status of a person has been described, but the detection target of the stay status is not limited to the person. For example, based on the staying state of the animal, the staying state and whether or not cleaning is required may be displayed. In this case, the person detection described in S512 may be replaced with the detection of a specific animal, and the subsequent processing may be performed regarding the detection target animal instead of the person. In this way, cleaning can be efficiently performed, for example, in consideration of the stay situation of livestock in the livestock barn. This can contribute to measures against infectious diseases in livestock. In addition, since it is assumed that there will be a large amount of excrement, hair loss, and the like to be cleaned in the stagnant position, it is possible to support efficient cleaning not only for the purpose of countermeasures against infectious diseases. Moreover, it is also possible to use it not only for the purpose of cleaning, but also for assisting layout design such as installation of a feeding station and a bed.

Also, in the present embodiment, an example has been described in which a room or barn is used as the target space for detecting the staying state. The present invention is not limited to this, and can be applied to various spaces including outdoors and indoors. For example, it can be applied to parks, grounds, stadiums, viewing spaces, gymnasiums, fitness gyms, sports gyms, assembly halls, halls, public halls, theaters, concert halls, waiting spaces, trains, buses, dining spaces, audience seats, etc. be.

Further, in the present embodiment, it was explained that the terminal device 101 and the server device 105 cooperated to perform processing, but the present invention is not limited to this. It is also possible to perform the processing described in . Conversely, more hardware (for example, multiple processors or circuits) may share the processing to control the entire system.

Moreover, although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention can be applied to the present invention. included. Furthermore, each embodiment described above merely shows one embodiment of the present invention, and it is also possible to combine each embodiment as appropriate.

(Other embodiments)
The present invention is also realized by executing the following processing. That is, the software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads the program code. This is the process to be executed. In this case, the program and the storage medium storing the program constitute the present invention.

Claims (20)

Acquisition means for acquiring information indicating the residence status of a detection target in a specific space;
receiving means for receiving specific instructions from the user;
Based on the information indicating the stay status acquired by the acquisition means, stay status by position in the specific space of the detection target from the reception of the specific instruction to the latest display update, and display control means for controlling display on a display object representing the specific space. The display control means initializes the display indicating that there was a stay when the specific instruction is received while the display object indicates that there was a stay as the stay situation, 2. The information processing system according to claim 1, wherein the display is controlled so as to correspond to a state in which there is no stagnation. 3. The information processing system according to claim 1, wherein said display object is a photographed image of said specific space or a diagram representing said specific space. 4. The information processing system according to any one of claims 1 to 3, wherein the specific instruction is an instruction that cleaning has been completed. The display control means displays a display prompting cleaning based on the information indicating the stay status, and performs control so as to hide the display prompting cleaning when the specific instruction is given. The information processing system according to any one of claims 1 to 4. The receiving means can receive a partial instruction for a part of the specific space rather than the entire space,
The display control means responds to the partial instruction in response to receiving the partial instruction while the display indicating that there was a stay as the stay situation is displayed on the display object. 6. The method according to any one of claims 1 to 5, characterized in that the display indicating that there has been a stay at a part of the position where the user is in contact is initialized so that the display corresponds to a state in which there is no stay. information processing system. The display control means performs display in a first display form indicating that the detection target has been detected at a position on the display object as the stay status based on the detection status of the detection target, and then Displaying in a second display form indicating that there was a stay satisfying the predetermined condition in response to the stay satisfying the predetermined condition,
The display in the first display mode is hidden in response to the detection of the detection target being stopped before the predetermined condition is satisfied, and
1. Control is performed so that the display in the second display mode is not hidden when the detection target is no longer detected after the predetermined condition is satisfied and the detection target is no longer detected. 7. The information processing system according to any one of 6. The detection target is a person, and the display control means expands and displays the range in front of the detected person when displaying the range indicating that there was a stay as the stay situation. 8. The information processing system according to any one of claims 1 to 7, wherein the information processing system controls. The display control means varies the stay status to be displayed according to whether or not the person detected as the detection target is wearing a mask,
Narrowing the range of display indicating that there was a dwell when wearing a mask compared to when not wearing a mask,
Increase the amount of time before an indication that there has been a dwell is made when wearing a mask compared to when not wearing a mask;
If you are wearing a mask, there is no indication that there was a stay,
9. The information processing system according to any one of claims 1 to 8, wherein control is performed so that at least one of: The display control means varies the staying state to be displayed according to the detection state of the hand of the person detected as the detection target,
For at least one of the position where the hand is detected and the position where the contact between the hand and the object is detected, the residence time is shorter than the positions other than the hand among the positions where the person is detected. 10. The information processing system according to any one of claims 1 to 9, wherein the information processing system is controlled so as to display a message indicating that the operation has been completed. The specific spaces include rooms, barns, parks, grounds, stadiums, viewing spaces, gymnasiums, fitness gyms, sports gyms, assembly halls, halls, public halls, theaters, concert halls, waiting spaces, trains, buses, eating and drinking. 11. The information processing system according to any one of claims 1 to 10, wherein the information processing system is at least one of a space and audience seats. 12. The information processing system according to any one of claims 1 to 11, wherein the detection target is at least one of a person and an animal. further comprising analysis means for analyzing a staying state of the detection target in the specific space based on a detection state of the detection target in the specific space;
13. The information processing system according to any one of claims 1 to 12, wherein said acquisition means acquires the result of analysis by said analysis means as information indicating said stay status. 14. The apparatus according to any one of claims 1 to 13, further comprising detecting means for detecting said detection target based on a photographed image of said specific space photographed by an imaging means or a temperature of said specific space detected by a temperature sensor. The information processing system according to any one of the items. Acquisition means for acquiring information indicating the residence status of a detection target in a specific space;
receiving means for receiving specific instructions from the user;
Based on the information acquired by the acquisition means, the stay status of the detection target by position in the specific space from the reception of the specific instruction to the present is displayed on a display representing the specific space. and control means for controlling to generate a composite image combined with the above. A control method for an information processing system,
an acquisition step of acquiring information indicating a staying state of a detection target in a specific space;
a receiving step for receiving specific instructions from a user;
Based on the information indicating the staying situation acquired in the acquiring step, the staying situation of the detection target by position in the specific space from the reception of the specific instruction to the latest display update, A control method for an information processing system, comprising: a display control step of controlling to display on a display object representing the specific space. A control method for an information processing device,
an acquisition step of acquiring information indicating a staying state of a detection target in a specific space;
a receiving step for receiving specific instructions from a user;
Based on the information acquired in the acquisition step, the stay status of the detection target by position in the specific space from the time when the specific instruction was received to the present is displayed on a display object representing the specific space. A control method for an information processing device, comprising: a generating step of generating a synthesized image synthesized with the above; An electronic device comprising each means of the information processing system according to any one of claims 1 to 14. A program for causing a computer to function as each means of the information processing system according to any one of claims 1 to 14. A computer-readable storage medium storing a program for causing a computer to function as each means of the information processing system according to any one of claims 1 to 14.

Images