JP2023111478A - Information processing device, information processing method, and program - Google Patents

Abstract

To make it possible to execute a notification about failure in operating a device in a manner suitable for the situation.SOLUTION: An information processing device comprises a control unit for performing control to notify a user that the user has forgotten to activate a monitoring target device based on action information on the user related to the monitoring target device that is determined based on sensing data, operation information on the monitoring target device, and presence/absence information on the user.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing device, an information processing method, and a program.

In recent years, life support devices have been proposed that support life by managing the operation of equipment.

For example, Patent Literature 1 below proposes a system that notifies a user when a device is not operating at a time scheduled by the user.

JP 2016-95619 A

However, in Patent Document 1, based on the amount of power used by the device, it is detected whether or not the device is operating at the scheduled time. However, the actual situation of the user, such as whether or not he/she intends to operate the system, is not sufficiently considered.

SUMMARY OF THE INVENTION Accordingly, the present invention has been made in view of the above problems, and an object of the present invention is to provide a new and improved information device capable of notifying a device of forgetting to operate in accordance with the situation. An object is to provide a processing device, an information processing method, and a program.

In order to solve the above problems, according to an aspect of the present invention, behavior information related to a monitored device of a user determined based on sensing data, operation information of the monitored device, and presence/absence of the user are provided. and a control unit that performs control for notifying the user that the monitoring target device has forgotten to operate based on the information.

When the user's situation based on the behavior information satisfies a predetermined condition, the user is present in the room, and the device to be monitored is not in operation within a preset estimated device operation time , control may be performed to notify the user that the monitored device has forgotten to operate.

The control unit determines a line-of-sight behavior of the user toward the monitored device as the behavior information, and further, based on the determined line-of-sight behavior, determines whether or not the user has an intention to operate the monitored device. , it may be determined that the predetermined condition is satisfied when the user has no intention of operating the monitored device.

The control unit may determine, as the determination of the line-of-sight behavior, whether or not the line-of-sight dwells within a specific range including the monitoring target device, and based on the determination result of the line-of-sight dwell, the presence or absence of the intention to operate the device may be estimated.

The control unit may count the number of times the line of sight stays within the specific range, and when the number of times the line of sight stays is less than a threshold, it may be estimated that the device has no intention to operate.

The control unit determines whether or not the user is performing a specific action that requires operation of the monitored device, and satisfies the predetermined condition if the user is performing the specific action. can be judged.

The control unit may determine whether or not the specific action is being performed based on sensing data of the tool used for the specific action.

The estimated device operation time may be a time zone in which it is assumed that the user performs an operation start operation of the monitoring target device.

When notifying the user that the monitoring target device has forgotten to start operation, the control unit performs an operation to start operation of the monitoring target device, which is set in advance from a predetermined starting time based on the behavior of the user. It may be performed at the timing when the optimal device operating grace time, which is the optimal grace time until the user performs the action, has passed.

The optimal device operation grace period may be set by machine learning based on the user's action history and the operation start time history of the monitoring target device.

The predetermined starting time based on the behavior of the user may be the time of returning home.

The action history of the user may include the time when the user went out and the time when the user came home.

The control unit may perform notification control of forgetting to operate the monitoring target device according to inventory information of deliverables obtained from the monitoring target device.

The control unit may perform control to notify that the monitoring target device has forgotten to operate when there is no inventory of the product obtained from the monitoring target device.

In the notification, the control unit may also present information on a substitute product for the deliverable.

When the notification is given at the timing when the optimal equipment operation grace period has passed, the control unit presents information on substitute products for deliverables obtained from the monitoring target equipment to the user when the estimated equipment operation time has passed. may

The device to be monitored may be a home electric device, and the presence or absence of the user may be the presence or absence of the user at home where the home electric device is placed.

In order to solve the above problems, according to another aspect of the present invention, a processor comprises behavior information related to a user's monitored device determined based on sensing data, and operation information of the monitored device. and information on the presence/absence of the user, performing control for notifying the user that the monitored device has forgotten to operate.

In order to solve the above problems, according to another aspect of the present invention, a computer is provided with behavior information related to a monitored device of a user determined based on sensing data, and operation information of the monitored device. and the presence/absence information of the user.

As described above, according to the present invention, it is possible to make a notification that a device has forgotten to operate, in a manner suitable for the situation.

BRIEF DESCRIPTION OF THE DRAWINGS It is a figure explaining the outline|summary of the notification system which concerns on this embodiment. 1 is a block diagram showing an example of configuration of a notification system according to a first embodiment; FIG. It is a figure explaining the setting of the apparatus operation assumption time which concerns on 1st Embodiment. It is a figure which shows an example of the notification screen of operation forgetting which concerns on 1st Embodiment. 6 is a flowchart showing an example of the flow of notification processing according to the first embodiment; 7 is a flow chart showing an example of the flow of count processing of the number of times of gaze retention according to the first embodiment. FIG. 4 is a diagram specifically explaining determination of gaze retention according to the first embodiment; It is a block diagram which shows an example of a structure of the notification system which concerns on 2nd Embodiment. FIG. 11 is a diagram illustrating an example of object motion determination according to the second embodiment; FIG. 11 is a flow chart showing an example of the flow of notification processing according to the second embodiment; FIG. It is a block diagram showing an example of a configuration of a notification system according to a third embodiment. It is a figure which shows an example of the data stored in action-learning DB which concerns on 3rd Embodiment. It is a figure which shows an example of clustering used for machine learning based on 3rd Embodiment. FIG. 13 is a flow chart showing an example of the flow of notification processing according to the third embodiment; FIG. FIG. 13 is a flow chart showing an example of the flow of notification processing according to the third embodiment; FIG. It is a block diagram which shows an example of a structure of the notification system which concerns on 4th Embodiment which concerns on 4th Embodiment. FIG. 11 shows an example of data stored in an inventory/substitute product DB according to the fourth embodiment; FIG. FIG. 14 is a flowchart showing an example of the flow of notification processing according to the fourth embodiment; FIG. FIG. 16 is a flow chart showing an example of the flow of inventory/substitute product management processing according to the fourth embodiment; FIG. It is a block diagram showing an example of hardware constitutions of an information processor concerning this embodiment.

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

<<1. Overview＞＞
FIG. 1 is a diagram illustrating an outline of a notification system according to this embodiment. As shown in FIG. 1, the notification system 1 according to the present embodiment includes an information processing device 10 that performs control for notifying the user that a monitoring target device 40 that is the monitoring target of this system has forgotten to operate, and a user behavior and It includes a motion detection device 20 that detects the motion of an object, a human detection sensor 30 that detects the presence or absence of a user, and a user terminal 50 that is an information processing terminal used by the user.

The monitored device 40 may be, for example, a household electrical device. Household electric appliances include, for example, rice cookers, washing machines, dishwashers, bath water heaters, and the like. Further, the monitoring target device 40 may be assumed to be a device affected by a time difference due to forgetting to operate. For example, in the case of a rice cooker, if the user forgets to operate the rice cooker, the rice will not be cooked at the scheduled meal time, making it impossible to eat. In the case of a washing machine, if the user forgets to start washing, the laundry cannot be dried at the scheduled time. Also, in the case of a bath water heater, if the user forgets to start the water heating, he or she will not be able to take a bath at the scheduled time. In this way, especially in the case of devices that provide products with a time lag, there is a possibility that the user will not immediately notice that the device has forgotten to start, so notification of the forgotten to start is important.

Also, the human detection sensor 30 is realized by, for example, a camera or an infrared sensor. The human detection sensor 30 detects the presence or absence of the user (whether or not the user is in the room). Also, the human detection sensor 30 may be provided around the monitored device 40 to detect whether or not the user is in the same space as the monitored device 40 . Also, as an example, the motion detection device 20 is assumed to be a camera (infrared stereo camera, visible light monocular camera) that detects the line of sight of the user and the motion of a tool. Also, the user terminal 50 is assumed to be a mobile terminal that can be possessed by the user, such as a smart phone, a tablet terminal, a smart watch, or a glasses type device.

The information processing apparatus 10 determines behavior information related to the monitoring target device 40 of the user based on the sensing data acquired from the motion detection device 20, and obtains the operation information of the monitoring target device and the human detection sensor 30. Based on the presence/absence information of the user based on the sensing data, control is performed so that the user terminal 50 notifies the user that the monitoring target device 40 has forgotten to operate. The operation information of the monitored device is information indicating whether or not the monitored device is in operation. The operation information of the monitored device 40 may be determined, for example, based on the amount of current flowing through the outlet to which the monitored device 40 is connected.

Here, in the conventional technology, based on the power consumption of the device, it is detected whether or not the device is operating at the scheduled time. However, the actual situation of the user, such as whether or not he/she intends to operate the system, is not sufficiently considered.

Therefore, in view of the above circumstances, the present invention proposes an information processing apparatus, an information processing method, and a program that enable notification of forgetting to operate a device in accordance with the situation.

Specifically, the information processing device 10 determines whether or not the user's situation satisfies a predetermined condition based on behavior information obtained from sensing data acquired from the motion detection device 20 . Then, the information processing apparatus 10 determines that the user's situation satisfies a predetermined condition, that the user is at home, and that the device operation start operation is assumed to be performed by the user. If the monitoring target device is not in operation within the assumed time, it is highly possible that the monitoring target device has forgotten to start operation. The estimated device operation time is set by the user, for example.

In this embodiment, as an example, the user terminal 50 is used for notification, but the present invention is not limited to this. The notification of forgetting to operate may be made from a display device (wall display, television device, projector, etc.) installed in the room or a sound device. Also, the motion detection device 20 and the human detection sensor 30 may be realized by one device, or may be mounted on one device. Also, the motion detection device 20 and the human detection sensor 30 may be mounted on the user terminal 50 . For example, the user terminal 50 may be realized by a glasses-type device, and the motion detection device 20 and the human detection sensor 30 may be realized by an outward/inward-facing camera, an infrared sensor, or the like mounted on the glasses-type device.

The outline of the notification system according to the present embodiment has been described above. Next, each embodiment of the notification system 1 according to this embodiment will be described.

<<2. First Embodiment>>
First, the notification system 1a according to the first embodiment will be described with reference to FIGS. 2 to 7. FIG. FIG. 2 is a block diagram showing an example of the configuration of the notification system 1a according to the first embodiment.

As shown in FIG. 2 , the notification system 1 a includes an information processing device 10 a , a user terminal 50 , a line-of-sight sensor 20 a that is an example of the motion detection device 20 , and a human detection sensor 30 . The information processing device 10a communicates with the user terminal 50, the line-of-sight sensor 20a, and the human detection sensor 30 via a network. The network may be, for example, a home network formed by a wireless LAN (Local Area Network) installed in the home.

(Line-of-sight sensor 20a)
The line-of-sight sensor 20a detects a user's line-of-sight behavior, which is an example of user behavior, and outputs line-of-sight information, which is an example of behavior information, to the information processing apparatus 10a. The line-of-sight sensor 20a may be implemented by, for example, an infrared stereo camera. In this case, the user's line-of-sight information (line-of-sight position) is output as a two-dimensional coordinate position on the two-dimensional image. A two-dimensional image is an image including at least the monitoring target device 40 in an angle of view, and may be acquired in advance by another camera. Further, the line-of-sight sensor 20 a may capture an image of the user in the room while being installed near the monitored device 40 , such as on the side or above the monitored device 40 . Further, the positional relationship between the line-of-sight sensor 20a and the monitoring target device 40 is known, and the line-of-sight sensor 20a detects the line-of-sight direction and distance of the user with respect to the line-of-sight sensor 20a on a two-dimensional image including the monitoring target device 40 within the angle of view. You may calculate the viewpoint position (two-dimensional coordinate position) in .

The line-of-sight sensor 20a measures the two-dimensional coordinate position P(x, y) of the line of sight for each unit time sufficient to estimate the user's situation, and outputs it to the information processing device 10a. For example, the detection time unit is Δt, and the two-dimensional coordinate position of the line of sight (line-of-sight position) obtained at each time is expressed as the two-dimensional coordinate position Pn (xn, yn) after the elapse of Δt×n time. That is, when the line-of-sight sensor 20a measures up to Δt×t time (measurement time t), the information processing apparatus 10 can obtain the following two-dimensional coordinate position array as the line-of-sight information of the user.
P1 (x1, y1), P2 (x2, y2)... Pt (xt, yt)

(Human detection sensor 30)
The human detection sensor 30 has a function of detecting the position of the user in space. For example, the human detection sensor 30 two-dimensionally monitors the space, two-dimensionally detects the user's position, and detects that the user has stayed in a specific area for a certain period of time or longer. The human detection sensor 30 is realized by, for example, a camera, and can detect the user's position by recognizing an object by analyzing a captured image. Here, as an example, it has been described that it has a function of detecting the position of the user in the space. A sensor may be used.

(User terminal 50)
The user terminal 50 has an input unit 510 realized by a touch panel and buttons, and an output unit 520 realized by a display unit and an audio output unit. The input unit 510 and the output unit 520 can be controlled by a control unit (not shown) built into the user terminal 50 . In addition, a dedicated application may be downloaded to the user terminal 50 in conjunction with the information processing apparatus 10a to realize the notification of forgetting to operate. In this case, the control unit performs various controls according to instructions from the dedicated application. Specifically, the control unit displays on the display unit (input unit 510) an input screen for inputting the equipment to be monitored and the estimated operating time of the equipment according to instructions from the dedicated application, and transmits the input information to the communication unit (not shown). ) to the information processing apparatus 10a. In addition, the control unit controls the output unit 520 to notify that the monitoring target device has forgotten to operate in accordance with the notification execution instruction transmitted from the information processing apparatus 10a. Notification methods include screen display (text or illustration), voice output, vibration, light emission, and the like.

(Information processing device 10a)
The information processing apparatus 10a performs control for notifying the user that the monitoring target device has forgotten to operate in consideration of the user's actual situation obtained from the sensing data. A specific configuration example of the information processing apparatus 10a will be described below.

<2-1. Configuration of Information Processing Device 10a>
The information processing device 10a has a communication unit 110, a control unit 120a, and a storage unit . The information processing device 10a can be implemented by any mobile terminal such as a smartphone, tablet terminal, or wearable device.

(Communication unit 110)
The communication unit 110 has functions as a transmitting unit and a receiving unit that transmit and receive information to and from other devices. For example, the communication unit 110 can transmit and receive data to and from the line-of-sight sensor 20a, the human detection sensor 30, and the user terminal 50 via a network. The network may be realized by a wired or wireless LAN (Local Area Network). Also, the network may be realized by a wireless communication network such as Wi-Fi (registered trademark) or Bluetooth (registered trademark).

(Control unit 120a)
The control unit 120a is mainly composed of, for example, a CPU (Central Processing Unit) or the like, and generally controls the operation of the information processing device 10a. Specifically, based on various information acquired by the communication unit 110, the control unit 120a determines whether or not to notify the user that the monitored device has forgotten to operate. is transmitted from the communication unit 110 to the user terminal 50 .

More specifically, the control unit 120 a can function as a time management unit 121 , a sensor management unit 122 , an operation intention determination unit 123 , an operation monitoring unit 124 and a notification control unit 125 .

The time management unit 121 manages various set times. In this embodiment, the time management unit 121 outputs information on the estimated device operation time to the sensor management unit 122 and the notification control unit 125 according to the estimated device operation time stored in the time storage unit 131 . The estimated device operation time is the time at which the user wishes to start the operation of the monitoring target device, and may be set as a time period with a certain time span (eg, 30 minutes, 1 hour, etc.). . The estimated device operation time may be set by the user or may be prepared in advance. FIG. 3 is a diagram for explaining setting of estimated device operation time according to the first embodiment. As shown in FIG. 3, for example, an input screen 521 for an estimated device operation time is displayed on the display unit 520a of the user terminal 50, and the user can set any time. Although not shown in FIG. 3, selection of the monitoring target device 40 to be associated may be performed in advance. In addition, as an arbitrary setting by the user, as shown in FIG. 3, it may be possible to set the range of line-of-sight stagnation used when estimating the user's intention to work (setting of a specific range when determining stagnation). . The user terminal 50 transmits information such as the input estimated device operation time to the information processing apparatus 10a. Note that the time management unit 121 may output to the sensor management unit 122 and the notification control unit 125 that the current time has reached the estimated device operation time.

The sensor management unit 122 has a function of managing various sensors. For example, the sensor management unit 122 appropriately controls ON/OFF of the line-of-sight sensor 20a and the human detection sensor 30 according to the estimated device operation time. Specifically, the sensor management unit 122 may perform control to turn on the line-of-sight sensor 20a for a certain period of time from a predetermined period of time before the estimated device operation time. Further, the sensor management unit 122 may perform control to turn on the human detection sensor 30 when the estimated device operation time has come.

The operation intention determination unit 123 has a function of determining whether or not the user has an intention to operate the monitoring target device 40 based on the line-of-sight information obtained from the line-of-sight sensor 20a. The line-of-sight information is an example of action information related to the monitoring target device 40 of the user, including information about the user's line of sight toward the monitoring target device 40 . When the user is looking at the monitored device 40, the operation intention determination unit 123 estimates that the user has the intention to operate the monitored device 40 (remembers to operate the monitored device 40). When the line of sight is not turned to 40, it is assumed that the monitored device 40 has no intention to operate (has forgotten to operate).

More specifically, the line-of-sight information used in this embodiment is a two-dimensional coordinate position (line-of-sight position) on a two-dimensional image including the monitored device 40 within the angle of view. The working intention determination unit 123 may estimate the presence or absence of the working intention based on the number of times the user's line of sight stays within the specific range. Staying of the line of sight is a state in which the line of sight position (gazing point) is positioned within a specific range on the two-dimensional image. Such a specific range may be arbitrarily set by the user in advance (see FIG. 3). The operation intention determination unit 123 determines whether or not the user's line of sight is kept by the line-of-sight retention determination unit 1231. Then, when the number of times of line-of-sight retention (an example of the determination result of line-of-sight retention) is below a threshold, the operation intention is determined by the intention estimation unit 1232. It is assumed that there is no intention to operate, and if the threshold is exceeded, it is assumed that there is an intention to operate.

In this way, by determining whether or not the user has the intention to operate based on the sensed line-of-sight information of the user, it is possible to notify the user of forgetting to operate according to the actual situation of the user. According to the present embodiment, it is possible to reduce the possibility of notifying the user of forgetting to operate despite the intention to operate.

Here, as an example, it has been described that the presence or absence of the working intention is estimated based on the number of gaze retention times, but the present embodiment is not limited to this. Presence or absence of intention may be estimated. The number of times of residence and the count of residence time are reset when the assumed operation time of the equipment has passed. Further, in the present embodiment, it has been described that the intention to work is estimated based on line-of-sight information, but the present invention is not limited to this.

The operation monitoring unit 124 has a function of monitoring whether the monitored device 40 is in operation. Although the monitoring method is not particularly limited, for example, information on the amount of current flowing through the outlet to which the monitored device 40 is connected may be obtained for determination. Further, when the monitored device 40 is an IoT device (Internet of Things) and is connected to a network, the information processing apparatus 10a may acquire operation information of the monitored device 40 via the network. In addition, the operation monitoring unit 124 analyzes the captured image of the monitoring target device 40, and detects the state of the light emitting unit provided in the monitoring target device 40, which indicates the operating state of the monitoring target device 40 by color or blinking, and the monitoring target device. It may be determined based on 40 movements.

The notification control unit 125 determines whether or not to notify that the operation has been forgotten, and if so, performs control to transmit a notification execution instruction to the user terminal 50 via the communication unit 110 . Specifically, if the user's situation satisfies a predetermined condition, the monitoring target device 40 is not operating within a preset estimated device operating time, and the user is present, the notification control unit 125 notifies judge to do

The estimated device operation time is a time period during which the user desires to start operation of the monitoring target device, and for example, the same time period is set every day. As an example, an estimated device operation time is set for a device that is basically used every day, such as a rice cooker and a water heater for a bath, and the information processing device 10a forgets to operate such a device during the estimated device operation time. is appropriately notified according to the user's actual situation, the user's life can be supported.

Also, whether or not the user is present can be detected by the human detection sensor 30 . The human detection sensor 30 may be provided around the monitored device 40 together with the line-of-sight sensor 20a. When a user is present around the monitored device 40 within the estimated device operation time, it is highly likely that the user will use the monitored device 40, and the failure to start operation can be appropriately notified. In addition, the notification control unit 125 may notify that the user forgot to operate when the user is in a specific place. In addition, the human detection sensor 30 is provided in a plurality of locations (each room) in the home, and it is not limited to the case where the user is in the vicinity of the monitoring target device 40, but when the user is at home, the notification control unit 125 can be prevented from forgetting to operate. Notification may be made. In either case, when it is assumed that the user is out and does not use the device 40 to be monitored, it is possible to avoid notifying the user that he forgot to operate.

Further, the predetermined condition is that the user has no intention of operating the monitoring target device 40 in this embodiment. The notification control unit 125 determines that the user's situation satisfies a predetermined condition when the operation intention determination unit 123 determines that the user does not have the operation intention. In the present embodiment, the user's line-of-sight position based on line-of-sight information (more specifically, the determination result of line-of-sight retention such as the number and time of line-of-sight retention) is used as the "user situation (based on behavior information)". be done.

In the user terminal 50 that has received the notification execution instruction, the output unit 520 outputs a notification that the operation has been forgotten. FIG. 4 is a diagram showing an example of an operation forgetting notification screen according to the present embodiment. As shown in FIG. 4 , a notification screen 522 indicating that the estimated device operation time has come may be displayed on the display unit 520a of the user terminal 50 .

(storage unit 130)
The storage unit 130 has a function of temporarily or permanently storing various data such as data relating to processing of the control unit 120a and operation program files of the control unit 120a. The storage unit 130 also has a time storage unit 131 that stores information on the estimated device operation time.

An example of the configuration of the information processing apparatus 10a according to the first embodiment has been described above. The information processing device 10a can be implemented by, for example, a PC (personal computer), a dedicated device, a server, or the like. At least part of the functions of the control unit 120a of the information processing device 10a may be provided in another device. For example, the function of the operating intention determination unit 123 may be provided in the line-of-sight sensor 20a, and the function of the notification control unit 125 (the function of determining whether or not to perform notification) may be provided in the user terminal 50. . The information processing device 10a may be realized by a plurality of devices.

Further, the information processing device 10a does not have to have all the configurations shown in FIG. For example, the information processing device 10 a may not have the sensor management unit 122 .

<2-2. Operation processing>
Next, notification processing according to the first embodiment will be described. FIG. 5 is a flowchart illustrating an example of the flow of notification processing according to the first embodiment.

As shown in FIG. 5, first, the line-of-sight sensor 20a starts sensing (step S103). Although the sensing trigger is not particularly limited, for example, the sensor management unit 122 of the information processing apparatus 10a can control the sensing to start a predetermined time (for example, two hours before) the estimated device operation time. The line-of-sight sensor 20a acquires line-of-sight information (line-of-sight position) of the user. The line-of-sight sensor 20 a is installed around the monitored device 40 and can detect the user's line of sight to the monitored device 40 . The line-of-sight sensor 20a acquires line-of-sight information for each unit time, and transmits the line-of-sight information to the information processing device 10a.

Next, based on the line-of-sight information transmitted from the line-of-sight sensor 20a, the operation intention determination unit 123 of the information processing device 10a performs a process of counting the number of times the user's line of sight is held in a specific range including the monitoring target device 40 (step S106). The line-of-sight retention determination unit 1231 determines whether the line-of-sight is retained. Details of the process of counting the number of gaze retention times will be described later with reference to FIGS. 6 and 7. FIG. Note that the counting process can be performed for a certain period of time (measurement time t). The certain period of time may be from a predetermined time before the estimated device operation time to the estimated device operation time.

Next, the working intention determination unit 123 determines whether or not the number of stays exceeds the threshold by the intention estimation unit 1232 (step S109). When the number of stays exceeds the threshold, it can be said that the user is conscious of the monitoring target device 40, and it is estimated that the user has an intention to operate (remembers to operate). On the other hand, if the number of stays is less than the threshold, it can be said that the user is not conscious of the monitored device 40, and it is estimated that the user has no intention of operating (forgetting to operate). In the present embodiment, it is possible to appropriately notify the failure to operate according to the user's actual situation, that is, the intention to operate estimated from the user's line-of-sight information.

Next, when the number of stays does not exceed the threshold (ie, when it is estimated that there is no intention to operate) (step S109/No), the notification control unit 125 determines that within the estimated device operation time (step S112/Yes), the user It is determined whether or not the user is indoors (step S115). Information on the estimated device operating time is obtained from the time management unit 121 . Also, whether or not the user is indoors can be determined based on the sensing data obtained from the human detection sensor 30 . “Indoor” may be the room in which the monitoring target device 40 is placed, or may be the inside of the home without being limited to the room. In the arrangement configuration in which the human detection sensor 30 is provided around the monitored device 40, it is determined whether or not the user is in the room where the monitored device 40 is placed. If the user is in the room within the estimated device operation time, there is a high possibility that the monitored device 40 will be used. That is, it is possible to avoid notifying the user who is out of the house that he forgot to operate.

Next, if the user is indoors (step S115/Yes), the notification control unit 125 determines whether or not the monitoring target device 40 is in operation (step S118). Whether or not the monitored device 40 is in operation is determined by information on the operating state of the monitored device 40 obtained from the operation monitoring unit 124 (for example, information on the amount of current flowing through the outlet to which the monitored device 40 is connected). determined based on

Then, if the monitoring target device 40 is not in operation (step S118/No), the notification control unit 125 notifies that the operation has been forgotten (step S121). Specifically, the notification control unit 125 performs control to transmit an execution instruction of the operation forgetting notification from the communication unit 110 to the user terminal 50 .

On the other hand, if the number of stays exceeds the threshold (step S109/Yes), if it is not within the device operation assumed time (step S112/No), if there is no user (step S115/No), or the monitoring target device is not in operation. If yes (step S118/Yes), the notification control unit 125 determines not to notify that the operation has been forgotten (step S124).

The operation processing of the notification system 1a according to the present embodiment has been specifically described above. As described above, in this embodiment, the user's line of sight is sensed to estimate whether or not the user intends to operate, and the user is notified of the forgetting to operate more appropriately. Note that the processes shown in steps S109 to S118 are not necessarily performed in the order shown in FIG. 5, and at least some of them may be performed in a different order, or may be processed in parallel.

(Counting processing of the number of gaze retention times)
Next, the process of counting the number of gaze dwell times shown in step S106 of FIG. 5 described above will be described with reference to FIG. FIG. 6 is a flow chart showing an example of the flow of count processing for the number of gaze dwell times according to the present embodiment.

As shown in FIG. 6, first, the line of sight of the user is detected by the line of sight sensor 20a (step S133). Specifically, the line-of-sight sensor 20a outputs the user's line-of-sight position (gazing point) as a two-dimensional coordinate position.

Next, when the line-of-sight coordinate position detected by the line-of-sight sensor 20a is within a predetermined specific range (step S136/Yes), the line-of-sight retention determining unit 1231 of the information processing device 10a determines that the line of sight is retained. (Step S139). On the other hand, when the coordinate position of the line of sight is not within the specific range (step S136/No), it is determined that the line of sight is not retained (step S142).

Here, with reference to FIG. 7, determination of gaze retention according to the present embodiment will be specifically described. FIG. 7 is a diagram specifically explaining determination of gaze retention according to the present embodiment. As shown in FIG. 7, the line-of-sight position of the user is output as a two-dimensional coordinate position Pn (xn, yn) on a two-dimensional image 210 captured so as to include the device to be monitored 40 within the angle of view. The line-of-sight sensor 20a is installed around the monitoring target device 40, detects the line-of-sight direction of the user and the distance to the user, and based on these, determines the viewpoint position ( gaze point) can be calculated as a two-dimensional coordinate position Pn(xn, yn). A specific range 212 surrounded by a point A (xa, ya), a point B (xb, yb), a point C (xc, yc), and a point D (xd, yd) is preset on the two-dimensional image 210. there is The specific range 212 is a two-dimensional plane that includes the area of the monitored device 40 on the two-dimensional image 210 . If the two-dimensional coordinate position Pn of the user's line of sight is within the specific range 212, the line of sight retention determination unit 1231 determines that the line of sight is retained. Note that the line-of-sight retention determination unit 1231 determines that the line-of-sight is retained when the two-dimensional coordinate position Pn (line-of-sight position) of the user's line of sight measured when Δt×n time has elapsed is within a specific range. good. That is, if the line of sight has only passed through the specific range while the line of sight is moving from Pn(xn, yn) to Pn+1(xn+1, yn+1), it is not determined that the line of sight is staying.

Then, the line-of-sight retention determination unit 1231 counts the number of times of line-of-sight retention when the line-of-sight retention is determined (step S145).

The processes shown in steps S133 to S145 are repeated for the measurement time t. The intention estimation unit 1232 estimates whether or not the user has an intention to operate the device based on the total number of gaze retentions obtained by the gaze retention determination unit 1231 . Note that the count of the number of stays in the line-of-sight dwell determination unit 1231 is reset when the assumed device operation time has passed.

The processing for counting the number of times the line of sight is held has been described above.

<<3. Second Embodiment>>
Next, a notification system 1b according to a second embodiment will be described with reference to FIGS. 8 to 10. FIG. The notification system 1b can realize a more appropriate notification by notifying the user of forgetting to operate the monitored device 40 when the user is performing a specific action that requires the operation of the device 40 to be monitored. For example, even if the user is indoors during the assumed operation time of the rice cooker, it is possible that the user is not operating the rice cooker for reasons such as eating out or taking out. The notification system 1b detects, for example, that the user is cooking (specifically, detects whether cooking utensils are being used in the kitchen), and even though the user is cooking, the rice is cooked. More appropriate notification can be realized by notifying that the rice cooker forgot to operate when the rice cooker is not operating during the assumed device operating time of the rice cooker. In addition, the timing of the notification is also optimized because the notification is performed in a situation where the user can operate the rice cooker, such as when the user is cooking.

FIG. 8 is a block diagram showing an example of the configuration of the notification system 1b according to the second embodiment. As shown in FIG. 8, the notification system 1b includes an information processing device 10b, a user terminal 50, a related object sensor 20b that is an example of the motion detection device 20, and a human detection sensor 30. FIG. The information processing device 10b communicates with the user terminal 50, the related object sensor 20b, and the human detection sensor 30 via a network. Since the user terminal 50 and the human detection sensor 30 are the same as those in the first embodiment, descriptions thereof are omitted here.

The related object sensor 20b is, for example, a camera, and detects (images) a tool (an example of a related object) used for "a specific action that requires operation of the monitoring target device 40". In this embodiment, to determine whether or not there is a specific action that requires operation of the monitoring target device 40, the action of the tool used for the specific action is determined based on sensing data (for example, a captured image) of the tool. The motion of the tool is not limited to the analysis of the captured image, and may be determined based on sensing data from various sensors (such as a motion sensor) provided on the tool, for example. Also, the determination method described here is an example, and the present embodiment is not limited to this. For example, the notification system 1b may recognize a user's action using a camera, a motion sensor, or the like, and determine whether or not a specific action is being performed.

<3-1. Configuration of Information Processing Device 10b>
The information processing device 10b has a communication unit 110, a control unit 120b, and a storage unit . The description of the same configuration as that of the first embodiment described with reference to FIG. 2 is omitted here.

The control unit 120 b can function as a time management unit 121 , a sensor management unit 122 , an operation monitoring unit 124 , a notification control unit 125 and an action determination unit 126 .

The action determination unit 126 uses the object detection unit 1261, the object position determination unit 1262, and the object motion determination unit 1263 to determine whether or not the user is performing a specific action that requires operation of the monitored device 40. Determining the motion of the relevant object used for the particular action. Specifically, the object detection unit 1261 detects object information based on sensing data from the related object sensor 20b. The related object sensor 20b is, for example, a camera, and the object detection unit 1261 outputs the two-dimensional coordinate position of the related object in the captured image (two-dimensional image) acquired from the camera as position information of the object. Next, the object position determination unit 1262 determines whether or not the position of the object is at the preset use position. A use position is a position used for a specific action, and a plane area included in a two-dimensional image obtained by the related object sensor 20b can be set. The use position may be set in advance by the user. Object motion determination section 1263 then determines whether the detected object is moving. The related object sensor 20b measures (images) the object every unit time (Δt) sufficient to detect the object. Measurements by the associated object sensor 20b may be performed for a measurement time t. The object detection unit 1261 outputs object position coordinates after Δt×n time has passed, based on the sensing data (captured image) of the related object sensor 20b. When the position of the object has moved more than a certain amount after Δt time at the use position (plane area), the object motion determination unit 1263 determines that the object is moving (the user performs a specific action using the object). is determined).

FIG. 9 is a diagram illustrating an example of object motion determination according to the present embodiment. The object detection unit 1261 detects the position (barycentric coordinate position) Q(x, y) of the object 45 from the two-dimensional image 220 obtained from, for example, the related object sensor 20b, as shown on the left side of FIG. Objects 45 (related objects) include, for example, cooking utensils used for cooking (an example of a specific action). The object detection unit 1261 detects the center-of-gravity coordinate position Q(x, y) of the object 45 for each unit time sufficient for motion determination. For example, the unit of time is Δt, and the two-dimensional coordinate position (coordinate position of the center of gravity) of the object 45 detected at each time is expressed as Qn (xn, yn) as the coordinate position after Δt×n time has elapsed. That is, when the object detection unit 1261 measures up to Δt×t time (measurement time t), the following array of two-dimensional coordinate positions is obtained as position information of the object.
Q1 (x1, y1), Q2 (x2, y2) ... Qt (xt, yt)

The object position determination unit 1262 determines whether or not the center-of-gravity coordinate position Q of the object 45 detected per unit time is at a preset use position on the two-dimensional image 220 . A region 222 surrounded by points E (xe, ye), points F (xf, yf), points G (xg, yg), and points H (xh, yh) is set in advance on the two-dimensional image 220 . Area 222 is assumed to be a place where cooking utensils are used for cooking, for example. Then, the object motion determining unit 1263 determines that the barycentric coordinate position Qn+1 (xn+1, yn+1) of the object 45 at tn+1 after Δt time has moved more than a certain amount with respect to the barycentric coordinate position Qn (xn, yn) of the object 45. If so, it is determined that the object 45 is moving (for example, cooking is being done). As shown in the two-dimensional image 224 on the right side of FIG. 9, if the object 45 is not moving even if it is in the use position (area 222), it can be determined that cooking (a specific action) is not being performed. This is because, for example, it is assumed that the washed cooking utensils are placed only to be dried. The action determination unit 126 outputs the determination result to the notification control unit 125 . The action determination unit 126 outputs the action determination result of the object used for the specific action as the determination result of whether or not the specific action is being performed.

As in the first embodiment, the notification control unit 125 determines whether or not to notify that the operation has been forgotten, and if so, transmits a notification execution instruction to the user terminal 50 via the communication unit 110. to control. Specifically, if the user's situation satisfies a predetermined condition, the monitoring target device 40 is not operating within a preset estimated device operating time, and the user is present, the notification control unit 125 notifies judge to do

In the second embodiment, the predetermined condition is that the user performs a specific action that requires the monitoring target device 40 to operate. Further, as the determination of the presence or absence of such a specific action, a motion determination of an object (related object) used for the specific action is performed. That is, in the related object sensor 20b, as an example of behavior information related to the monitoring target device 40 of the user, information related to a specific behavior that requires the operation of the monitoring target device 40 (specifically, for example, a captured image of the related object ) is obtained.

When the action determination unit 126 determines that a specific action is being performed (for example, when it is determined that an object used for a specific action is moving at the use position), the notification control unit 125 It is determined that the user's situation satisfies a predetermined condition. In this embodiment, the motion state of the object based on the captured image is used as the "user's situation (based on behavior information)".

An example of the configuration of the information processing apparatus 10b according to the second embodiment has been described above. Note that the configuration of the information processing device 10b is not limited to the example shown in FIG. At least part of the functions of the information processing device 10b may be provided in a device other than the information processing device 10b shown in FIG. Further, the information processing device 10b may be realized by a plurality of devices. Further, the information processing device 10b does not have to have all the configurations shown in FIG. For example, the information processing device 10b may not have the sensor management unit 122 .

<3-2. Operation processing>
Next, notification processing according to the second embodiment will be described. FIG. 10 is a flowchart showing an example of the flow of notification processing according to the second embodiment.

As shown in FIG. 10, first, the human detection sensor 30 detects whether or not a user has entered a specific area (for example, a kitchen) (step S203).

Next, sensing by the related object sensor 20b is started (step S206). In the present embodiment, the sensor management unit 122 can control the start of sensing by the related object sensor 20b using the entry of the user into a specific area (for example, the kitchen) as a trigger. The associated object sensor 20b may be a camera placed in a location whose detection range (imaging range) is the location where the object is used (for example, the kitchen). The related object sensor 20b acquires a captured image every unit time and transmits the captured image to the information processing device 10b.

Next, the object detection unit 1261 of the information processing device 10a detects an object from the captured image (step S209). The object detection unit 1261 can analyze the captured image to perform object recognition, and detect an object (for example, cooking utensils) that is used for a specific action that requires operation of the monitored device 40 .

Next, the object position determination unit 1262 determines whether or not the position of the detected object (for example, the barycentric coordinate position) is at a preset use position (planar area) on the captured image (two-dimensional image). (Step S212).

Next, when the object is in the use position (step S212/Yes), the object motion determination unit 1263 determines whether or not the object is moving (step S215). Specifically, it is determined based on whether the position of the object has moved after n hours. When an object used for a specific action is moving at the use position, it can be said that the user is performing a specific action that requires operation of the monitored device 40 . For example, if it is determined that the cookware is operating, it is determined that the user is cooking. The action determination unit 126 outputs the determination result to the notification control unit 125 .

Next, when it is determined that the object is moving (step S215/Yes), the notification control unit 125 inquires of the time management unit 121 and determines whether or not it is within the estimated device operation time (step S218).

Next, if it is within the estimated device operating time (step S218/Yes), the notification control unit 125 inquires of the operation monitoring unit 124 to determine whether the monitored device 40 is operating (step S221).

Then, if the monitoring target device 40 is not in operation (step S221/No), the notification control unit 125 notifies that the operation has been forgotten (step S224). Specifically, the notification control unit 125 performs control to transmit an execution instruction of the operation forgetting notification from the communication unit 110 to the user terminal 50 . As described above, in the present embodiment, by determining whether or not a specific action that requires the operation of the monitored device 40 is being performed, if there is a high possibility that the monitored device 40 will actually be used, , can be notified of forgetting to operate. In other words, notification suitable for the user's situation can be realized. In addition, when performing a specific action, there is a high possibility that the device can be operated. . For example, if you are relaxing in the living room watching TV and are notified that the rice cooker has forgotten to operate, it is troublesome to get up and move to the kitchen, so there is a high possibility that you will forget the notification without taking any action. , during cooking, it is easy to operate the rice cooker.

On the other hand, if the detected object is not at the use position (step S212/No), if the object is not moving (step S215/No), if it is not within the estimated device operation time (step S218/No), or if the monitoring If the target device is in operation (step S221/Yes), the notification control unit 125 determines not to notify that the operation has been forgotten (step S227). As a result, for example, when the user is cooking but it is not the expected device operation time, or when the device is already in operation, it is determined that notification is not necessary, and notification is not performed.

The operation processing of the notification system 1b according to this embodiment has been specifically described above. Note that the processes shown in steps S203 to S221 are not necessarily performed in the order shown in FIG. 10, and at least some of them may be performed in a different order, or may be processed in parallel.

<<4. Third Embodiment>>
Next, a notification system 1c according to the third embodiment will be described with reference to FIGS. 11 to 15. FIG. The notification system 1c learns the behavior of the user to further optimize the timing of notifying that the operation has been forgotten. For example, even if the user forgets to start the rice cooker immediately after returning home, the user has to wash his/her hands, change clothes, and put away his/her bag. You may forget about it. In this embodiment, the notification timing can be optimized by notifying the user of forgetting to operate the device in a situation where the user can (highly likely) operate the device. The situation in which the user can operate the device may be when a predetermined period of time has elapsed from a predetermined starting time based on the behavior of the user. The predetermined time is a desirable grace period until operation, and is referred to as an "optimal device operating grace period" in the present embodiment. The optimal device operation grace period can be set by machine learning, for example, based on the user's action history and the operation start time history of the monitoring target device 40 . In the notification system 1c, a notification of forgetting to operate is given at the timing when the optimum equipment operation grace period has passed from the time which is a predetermined starting point based on the behavior of the user. Note that, in the present embodiment, “time of returning home” is used as an example of a time that is a predetermined starting point based on user behavior. Instead of notifying the user immediately when he or she returns home during the estimated operating time of the device, the notification is sent when a predetermined grace period has elapsed since the time the user returned home, so that the notification is more suitable for the user's life rhythm after returning home. Timing notification can be realized.

FIG. 11 is a block diagram showing an example configuration of a notification system 1c according to the third embodiment. As shown in FIG. 11, the notification system 1c includes an information processing device 10c, a user terminal 50, a line-of-sight sensor 20a, and a human detection sensor 30. FIG. The information processing device 10c communicates with the user terminal 50, the line-of-sight sensor 20a, and the human detection sensor 30 via a network. The user terminal 50, the line-of-sight sensor 20a, and the human detection sensor 30 are the same as those in the first embodiment, so descriptions thereof are omitted here.

<4-1. Configuration of Information Processing Device 10c>
The information processing device 10c has a communication unit 110, a control unit 120c, and a storage unit 130c. Descriptions of the same configurations as those of the first and third embodiments described with reference to FIGS. 2 and 11 are omitted here.

The control unit 120 c can function as a time management unit 121 , a sensor management unit 122 , an operation intention determination unit 123 , an operation monitoring unit 124 , a notification control unit 125 and an action learning unit 127 . The storage unit 130 c also has a time storage unit 132 and an action learning DB (database) 133 .

The behavior learning unit 127 has a function of learning the behavior of the user in order to obtain the optimum device operation grace period. Specifically, the behavior learning unit 127 acquires the user's outing time (length of time spent out) and the time of returning home obtained from the human detection sensor 30, and the device operation start time obtained from the operation monitoring unit 124, It stores in action learning DB133. It should be noted that the out-of-home time and return-home time from the device operation start time to 24:00 on the day are discarded without being stored in the behavior learning DB 133 . Further, when the device is not operated, the behavior learning unit 127 deletes the user's going out time and returning home time acquired from the human detection sensor 30 from the behavior learning DB 133 at 24:00 on the day. FIG. 12 shows an example of data (action history data) stored in the action learning DB 133. As shown in FIG. As shown in FIG. 12, the date, outing time, return time, operation start time, and time difference between return home time and operation start time are stored. The out-of-home time may be the time during which the human detection sensor 30 could not detect the user (absence time). Also, a user's entry and exit may be detected by a human sensor provided at the entrance and a sensor for detecting door opening/closing and door locking/unlocking, and the outing time may be obtained. Alternatively, the out-of-home time may be acquired from a sensor provided in the user terminal 50 owned by the user or from the result of behavior analysis based on sensing data.

Also, the action learning unit 127 learns the time from when the user comes home until the device starts operating, based on the data stored in the action learning DB 133 . Specifically, for example, unsupervised machine learning clustering is used. The action learning unit 127 calculates an operation grace period after returning home (the time from the return time to the start of the appliance operation) from the return time and the device operation start time stored in the action learning DB 133, and calculates the calculated operation grace time and Clustering is performed from the outing time. In addition, there is a possibility that a cluster will be formed in the range where the calculated operation grace time exceeds 60 minutes, but in that case, it is considered that the user's going out will have little impact on the device operation, so it should be excluded from the clustering target in advance. can be Also, going out for less than 10 minutes may be considered to have little effect on device operation, and may be excluded from clustering targets. Since these values may be regarded as outliers during clustering, the exclusion process may be performed after clustering. FIG. 13 shows an example of clustering used for machine learning. As shown in FIG. 13, clusters C1 to C3, for example, are obtained with respect to out-of-home time and operating grace time.

Based on the result of clustering, the behavior learning unit 127 outputs the optimum timing for notifying that the device has forgotten to operate after the user returns home. Since it is difficult to imagine that the user's situation will change at minute time intervals after the user returns home, the timing for notifying that the device has forgotten to operate may be set at intervals of 5 minutes, 10 minutes, or the like. For example, when setting at intervals of 10 minutes, in the cluster C1 shown in FIG. 13, the data with the longest operating grace period after returning home is 18 minutes, so the optimal device operating grace period is set to 20 minutes. set. The behavior learning unit 127 transmits the set optimal device operation grace period to the time management unit 121 . The time management unit 121 stores the optimum device operating grace period in the time storage unit 132 . Also, the behavior learning unit 127 may set the optimum device operation grace time corresponding to the time when the person is out of the house, based on the clustering result.

Further, in the present embodiment, the optimal device operating grace period is set based on the outing time and the operating grace period, but the present embodiment is not limited to this. For example, the reason for going out may also be used for machine learning. Reasons for going out include, for example, shopping, going to work, going to work in the morning, taking a walk, checking mail, going out during the day on holidays, and the like. The reason for going out may be estimated based on the day of the week or time when going out (departure time, return time, length of time spent out, etc.), may be obtained from the user's schedule information, or may be obtained from the user's schedule information. may be estimated from sensing data (position data, etc.) of the user terminal 50 possessed by . Also, the user may arbitrarily input the reason for going out. By using the reason for going out, the action learning unit 127 can set the optimal device operation grace period according to the time for going out and the reason for going out. In addition, the behavior learning unit 127 can also learn using the return time and the operation grace period without using the outing time, and set the optimum device operation grace period according to the return time.

When the notification control unit 125 determines to notify that the operation has been forgotten, the notification control unit 125 inquires of the time management unit 121 about the optimum equipment operation grace period, and determines whether the optimum equipment operation grace period is within the estimated equipment operation time and from the user's return home time. is controlled so that the notification is made at the timing when the

An example of the configuration of the information processing apparatus 10c according to the third embodiment has been described above. Note that the configuration of the information processing device 10c is not limited to the example shown in FIG. At least part of the functions of the information processing device 10c may be provided in a device other than the information processing device 10c shown in FIG. Also, the information processing device 10c may be realized by a plurality of devices. Further, the information processing device 10c does not have to have all the configurations shown in FIG. For example, the information processing device 10c does not have to have the sensor management unit 122 .

<4-2. Operation processing>
Next, notification processing according to the third embodiment will be described. FIG. 15 is a flowchart illustrating an example of the flow of notification processing according to the third embodiment.

The processes shown in steps S323 to S338 and S344 shown in FIG. 15 are the same as the processes shown in steps S103 to S118 and S124 according to the first embodiment shown in FIG. In this embodiment, for example, whether or not to notify that the operation has been forgotten is determined based on the contents described in the first embodiment.

The number of times the line of sight stays does not exceed the threshold (that is, when it is estimated that there is no intention to operate) (step S329/No), within the estimated device operation time (step S332/Yes), and the user is present (step S335/ Yes), if the monitoring target device 40 is not in operation (step S338/No), the notification control unit 125 executes notification at notification timing considering the optimum device operation delay time (step S342). Specifically, the notification control unit 125 performs control to transmit an execution instruction of the operation forgetting notification from the communication unit 110 to the user terminal 50 .

The notification timing considering the optimal device operation delay time is, for example, the timing when the optimal device operation delay time has passed since the time when the user returns home. In this way, in the present embodiment, it is possible to avoid notification immediately after the user returns home, and perform notification at more optimal timing.

The operation processing of the notification system 1c according to the present embodiment has been specifically described above. Note that the processes shown in steps S323 to S344 are not necessarily performed in the order shown in FIG. 15, and at least some of them may be performed in a different order, or may be processed in parallel.

In addition, in the above-described embodiment, the return time is used as the starting point of the optimal equipment operation grace period, but the present embodiment is not limited to this. For example, it may be the time when the user performs some action, such as the time when the user wakes up, eats, takes a bath, or enters the room. Moreover, what time is set as the starting point may be set in advance for each monitoring target device.

In addition, in the present embodiment, after determining whether or not to notify that the device has forgotten to operate based on the contents described in the first embodiment, it has been explained that the notification timing is set in consideration of the optimum equipment operating grace period. The embodiment is not limited to this, and it may be determined whether or not to notify that the operation has been forgotten based on the contents described in the second embodiment. In this case, the configuration of the information processing device 10c shown in FIG.

Machine learning in the action learning unit 127 is not limited to unsupervised machine learning, and supervised machine learning may be used. For example, supervised machine learning may be used to set the optimal device operation grace period. Specifically, a regression model is generated in which the teaching data is "equipment operation grace time for each actual outing time (time from returning home to operation start time; for example, "time difference" shown in FIG. 12)". The behavior learning unit 127 inputs the "user's out-of-home time" into the regression model, and outputs the "optimal device operating grace period." For example, using a regression model generated based on action history data as shown in Table 1 below, when "outing time: 0h40m" is input, "optimal device operating grace time: 13m" can be output.

<<5. Fourth Embodiment>>
Next, a notification system 1d according to the fourth embodiment will be described with reference to FIGS. 16 to 19. FIG. The notification system 1d considers the information (inventory information) about the inventory of deliverables by the monitoring target device 40 when determining whether or not to notify the monitoring target device 40 of forgetting to operate, so that the notification system 1d can provide a more appropriate notification. come true. For example, in the case of a rice cooker, it is conceivable that the rice cooked in one operation of the device is frozen and stored, and the user does not operate the rice cooker because he intends to eat it. The notification system 1d can realize a more appropriate notification by avoiding notifying the operation failure when there is stock.

In addition, the notification system 1d, when the timing considering the optimal device operation delay time described in the third embodiment, has passed the estimated device operation time, the notification system 1d provides information on substitute products for deliverables obtained by the monitoring target device 40. may be presented to the user. For example, in the case of a rice cooker, if the estimated operating time of the appliance has passed, the rice will not be cooked at mealtime, and the staple food, rice, cannot be prepared. In this case, if there are, for example, frozen rice balls, frozen udon noodles, retort rice, etc., as substitutes for rice, the convenience of this system is further improved by presenting information on these (substitute product information) to the user. be able to.

Note that the notification system 1d may also present substitute product information when making a notification within the estimated device operation time.

FIG. 16 is a block diagram showing an example configuration of a notification system 1d according to the fourth embodiment. As shown in FIG. 16, the notification system 1d includes an information processing device 10d, a user terminal 50, a line-of-sight sensor 20a, a human detection sensor 30, and an inventory/substitute product sensor 60. FIG. The information processing device 10d communicates with the user terminal 50, the line-of-sight sensor 20a, the human detection sensor 30, and the inventory/substitute product sensor 60 via a network. The user terminal 50, the line-of-sight sensor 20a, and the human detection sensor 30 are the same as those in the first embodiment, so descriptions thereof are omitted here.

The inventory/substitute item sensor 60 has a function of detecting the inventory of deliverables obtained by the monitoring target device 40 and the substitute item. A plurality of inventory/substitute product sensors 60 may be provided. Also, one or more sensors for detecting inventory and one or more sensors for detecting substitute products may be provided. The inventory/replacement sensor 60 may be, for example, a camera that captures the contents of a freezer. In addition, the inventory/replacement sensor 60 may be a device that identifies and manages objects using RFID (Radio Frequency Identification) technology.

<5-1. Configuration of Information Processing Device 10d>
The information processing device 10d has a communication unit 110, a control unit 120d, and a storage unit 130d. Descriptions of the same configurations as those of the first and third embodiments described with reference to FIGS. 2 and 11 are omitted here.

The control unit 120 d can function as a time management unit 121 , a sensor management unit 122 , an operation intention determination unit 123 , an operation monitoring unit 124 , a notification control unit 125 , an action learning unit 127 , and an inventory/substitute product management unit 128 . The storage unit 130d also has a time storage unit 132, an action learning DB 133, and an inventory/substitute item DB 134.

The inventory/substitute product management unit 128 has a detection unit 1281 and a data management unit 1282 . The detection unit 1281 detects inventory information and substitute product information based on sensing data obtained from the inventory/substitute product sensor 60 . For example, it analyzes captured images inside the freezer acquired by the inventory/substitute product sensor 60, and detects the presence, number, position (two-dimensional coordinate position), etc. of inventory and substitute products by object recognition. The data management unit 1282 stores inventory information and substitute product information in the stock/substitute product DB 134 based on the detection result of the detection unit 1281 . FIG. 19 shows an example of data stored in the stock/substitute product DB 134. As shown in FIG. For example, as shown in FIG. 19, the inventory/alternative product DB 134 stores information such as the food name, date of storage, cooking time, remaining quantity, and the like. Information on food names and cooking times may be registered in advance. In addition, the inventory/substitute product sensor 60 photographs the inside of the refrigerator according to, for example, the opening/closing timing of the freezer, and the inventory/substitute product management unit 128 updates the number of products in stock and the number of substitute products. Not all stocks and substitutes are stored in the same place, and it is assumed that some of them are outside the freezer, for example. In addition, priority at the time of proposal is set for each substitute product. The priority may be arbitrarily set in advance, or may be appropriately set by the data management unit 1282 . The data management unit 1282 may change the default priority according to, for example, cooking time, storage date, remaining quantity, and the like.

The notification control unit 125 makes an inquiry to the inventory/substitute product management unit 128 and acquires inventory information when determining whether or not to notify that the operation has been forgotten. Specifically, the notification control unit 125 determines that notification should be made if there is no stock, and that notification should not be made if there is stock. The notification control unit 125 may also present substitute item information acquired from the inventory/substitute item management unit 128 when making the notification.

In addition, the notification control unit 125 may notify that the device has forgotten to operate at a timing that takes into consideration the optimum device operation grace period described in the third embodiment. At this time, as a result of considering the optimum equipment operation grace period, it is assumed that the estimated equipment operation time may be exceeded. In this case, the notification control unit 125 may notify the user of the information on the substitute product (substitute product information) for the deliverable obtained by the monitoring target device 40 . Specifically, the notification control unit 125 controls transmission of substitute item information to the user terminal 50 via the communication unit 110 .

An example of the configuration of the information processing device 10d according to the fourth embodiment has been described above. Note that the configuration of the information processing device 10d is not limited to the example shown in FIG. At least part of the functions of the information processing device 10d may be provided in a device other than the information processing device 10d shown in FIG. Also, the information processing device 10d may be realized by a plurality of devices. Further, the information processing device 10d may not have all the configurations shown in FIG. For example, the information processing device 10d may not have the sensor management unit 122 .

<5-2. Operation processing>
Next, notification processing according to the fourth embodiment will be described. FIG. 18 is a flowchart showing an example of the flow of notification processing according to the fourth embodiment.

As shown in FIG. 18, first, sensing by the line-of-sight sensor 20a and the inventory/substitution product sensor 60 is started (step S403). The line-of-sight sensor 20a may be controlled by the sensor management unit 122 to start from a predetermined time (for example, two hours) before the estimated device operation time. In addition, the inventory/substitute product sensor 60 may be controlled to start at the same timing as the line-of-sight sensor 20a by the sensor management unit 122, or may be controlled to start at the timing when the estimated device operation time has come.

Next, the inventory/substitute product management unit 128 of the information processing device 10d performs inventory/substitute product management processing based on the sensing data obtained from the inventory/substitute product sensor 60 (step S406). Specific contents will be described later with reference to FIG. 19 .

Next, in steps S409 to S418, processing similar to steps S106 to S115 according to the first embodiment shown in FIG. 5 is performed. In this embodiment, for example, whether or not to notify that the operation has been forgotten is determined based on the contents described in the first embodiment.

If the number of gaze dwell times does not exceed the threshold value (that is, if it is estimated that there is no intention to operate) (step S412/No), the device is within the assumed operation time (step S415/Yes), and the user is present (step S418) /Yes), the notification control unit 125 determines whether or not there is a product in stock for the monitoring target device 40 (step S421). Specifically, the notification control unit 125 inquires of the inventory/substitute product management unit 128 to acquire inventory information and make a determination.

Next, if there is inventory (step S421/Yes), the notification control unit 125 determines not to notify the operator of the failure to operate (step S430).

On the other hand, if there is no inventory (step S424/No), the notification control unit 125 inquires of the operation monitoring unit 124 to acquire operation information of the monitored device 40, and determines whether the monitored device 40 is operating. (step S424).

Then, if it is not in operation (step S424/No), the notification control unit 125 executes notification at notification timing considering the optimum equipment operation grace period (step S427). Specifically, the notification control unit 125 performs control to transmit an execution instruction of the operation forgetting notification from the communication unit 110 to the user terminal 50 . The notification control unit 125 may also present substitute product information when making such a notification.

In addition, the notification control unit 125 may present the substitute item information with the reset priority when the notification timing considering the optimum equipment operation delay time passes the estimated equipment operation time. The resetting of priority is performed, for example, according to the cooking time of each substitute item. As a result, even if there is no time to cook rice after the estimated operation time of the rice cooker has passed, the user can be presented with information on substitutes that take less time to cook, thereby making suggestions that can be made in time for meals.

The operation processing of the notification system 1c according to the present embodiment has been specifically described above. Note that the processes shown in steps S403 to S424 are not necessarily performed in the order shown in FIG. 18, and at least some of them may be performed in a different order, or may be processed in parallel.

(Inventory/substitute product management processing)
Next, the inventory/substitute product management process shown in step S406 of FIG. 18 will be described with reference to FIG. FIG. 19 is a flow chart showing an example of the flow of inventory/substitute product management processing according to the present embodiment.

As shown in FIG. 19, first, the information processing device 10d acquires sensing data from the inventory/alternative product sensor 60 (step S453).

Next, the inventory/substitute product management unit 128 of the information processing device 10d detects the inventory and the substitute product by the detection unit 1281 based on the sensing data (step S456). Sensing data is, for example, a captured image of the inside of the refrigerator, and inventory and replacement items are detected by object recognition.

Next, when the inventory can be detected (step S46/Yes), the data management unit 1282 stores the information of the inventory in the inventory/substitute product DB 134 (step S459). The inventory/alternative product DB 134 may store detailed information such as the date of receipt and the number of items in stock, or may store only information regarding the presence or absence of inventory.

On the other hand, if the inventory cannot be detected (step S456/No), the data management unit 1282 sets the priority of the first stage to each managed substitute item (step S462). Note that the data management unit 1282 updates the substitute product information stored in the inventory/substitute product DB 134 when a substitute product is detected from the sensing data. The priority in the first stage may be set, for example, in descending order of the number of remaining items or in descending order of the date of receipt. The notification control unit 125 can present the information of the first-stage high-priority substitute item to the user at the time of the operation failure notification within the estimated device operation time.

Next, when the notification timing determined by the notification control unit 125 passes the estimated device operation time (step S465/Yes), the data management unit 1282 sets the priority of the second stage to each substitute product (step S468). For example, when the data management unit 1282 receives an inquiry about a replacement product from the notification control unit 125 to the effect that the estimated device operation time has passed, the data management unit 1282 sets the priority of the second stage and notifies the replacement product information. Output to the control unit 125 . The priority of the second stage may be set, for example, in descending order of cooking time. For example, the notification control unit 125 can propose to the user the use of a substitute product with a short cooking time when notifying that the rice cooker forgot to operate after the estimated device operation time has passed.

The inventory/substitute product management process has been described above.

In addition, in the notification processing of the present embodiment described with reference to FIG. 17, the third embodiment described with reference to FIG. Embodiments are not limited to this, and considering the presence or absence of inventory and proposing alternatives can be combined with the notification processing of the first embodiment (see FIG. 5) and the notification processing of the second embodiment (see FIG. 10). good too. In this case, as the configuration of the information processing apparatus 10d shown in FIG. good too.

Also, if there is no information on a substitute product, it is not necessary to present the substitute product information. Further, for example, if the monitoring target device 40 is a washing machine, a sensor may detect the amount of laundry, and the information processing device 10d may notify that the operation is forgotten when there is laundry (alternative no alternatives are proposed). When combined with the first embodiment, the information processing device 10d notifies that the user has forgotten to operate when it is estimated that the user is in the room within the estimated operating time and has no intention of operating, and there is laundry. I do. As described above, in the fourth embodiment, the presence or absence of an object handled by the monitoring target device 40 is taken into consideration when determining whether or not to notify that the device has forgotten to operate. notification can be implemented.

<<6. Hardware configuration example >>
Next, the hardware configuration of the information processing apparatus 10 (10a to 10d) according to this embodiment will be described. The above operations are realized by cooperation between software and hardware described below.

FIG. 20 is a block diagram showing an example of the hardware configuration of the information processing device 900 according to this embodiment. The information processing device 900 is an example of a hardware configuration applied to the information processing device 10 according to this embodiment.

The information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, and an interface 907. , an input device 908 , an output device 909 , a storage device 910 , a drive 911 , and a communication device 913 .

The CPU 901 functions as an arithmetic processing device and a control device, and controls general operations within the information processing device 900 according to various programs. Alternatively, the CPU 901 may be a microprocessor. The ROM 902 stores programs, calculation parameters, and the like used by the CPU 901 . The RAM 903 temporarily stores programs used in the execution of the CPU 901, parameters that change as appropriate during the execution, and the like. These are interconnected by a host bus 904 comprising a CPU bus or the like. The cooperation of the CPU 901, the ROM 902 and the RAM 903 implements the control unit 120 (120a to 120d) of the information processing apparatus 10. FIG.

The host bus 904 is connected via a bridge 905 to an external bus 906 such as a PCI (Peripheral Component Interconnect/Interface) bus. Note that the host bus 904, the bridge 905 and the external bus 906 do not necessarily have to be configured separately, and these functions may be implemented in one bus.

The input device 908 includes input means for the operator to input information, such as a mouse, keyboard, touch panel, button, switch, or microphone, and input control for generating an input signal based on the operator's input and outputting it to the CPU 901. It is composed of circuits, etc. An operator who operates the information processing apparatus 900 can input various data to the information processing apparatus 900 and instruct processing operations by operating the input device 908 .

The output device 909 includes, for example, a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, a display device such as a lamp, and an audio output device such as a speaker.

The storage device 910 is a data storage device. The storage device 910 may include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, a deletion device that deletes data recorded on the storage medium, and the like. The storage device 910 implements the storage unit 130 (130c, 130d) of the information processing apparatus 10 .

A drive 911 is a storage medium reader/writer and is externally attached to the information processing apparatus 900 . The drive 911 reads information recorded on a removable storage medium 912 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 903 . Drive 911 can also write information to removable storage medium 912 .

A communication device 913 is a communication interface configured by a communication device or the like for communication. The communication device 913 implements the communication unit 110 of the information processing apparatus 10 .

Note that the hardware configuration of the information processing device 900 is not limited to the configuration shown in FIG. 20 . For example, the information processing apparatus 900 does not need to include the communication device 913 when performing communication via a connected external communication device. Further, the information processing apparatus 900 may not include the input device 908 or the output device 909, for example. Also, for example, part or all of the configuration shown in FIG. 20 may be realized by one or two or more ICs (Integrated Circuits).

<<7. Supplement >>
Although the preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention belongs can conceive of various modifications or modifications within the scope of the technical idea described in the claims. It is understood that these also naturally belong to the technical scope of the present invention.

For example, the monitored device 40 is not limited to household electrical equipment, and may be, for example, industrial equipment.

It is also possible to create one or more computer programs for causing hardware such as the CPU, ROM, and RAM built into the information processing device 10 to exhibit the functions of the information processing device 10 . Also provided is a computer-readable storage medium having the one or more computer programs stored thereon.

1 (1a to 1d) notification system 10 information processing device 110 communication unit 120 control unit 121 time management unit 122 sensor management unit 123 operation intention estimation unit 1231 gaze retention determination unit 1232 intention estimation unit 124 operation monitoring unit 125 notification control unit 126 action Determination unit 1261 Object detection unit 1262 Object position determination unit 1263 Object motion determination unit 127 Action learning unit 128 Inventory/substitute item management unit 1281 Detection unit 1282 Data management unit 130 Storage unit 131, 132 Time storage unit 133 Action learning DB
134 Inventory/substitute product DB
20 motion detector 20a line-of-sight sensor 20b related object sensor 30 human detection sensor 50 user terminal 60 inventory/replacement sensor

Claims (19)

Based on the user's behavior information related to the monitored device determined based on sensing data, the operation information of the monitored device, and the user's presence/absence information, it is possible to prevent the monitored device from forgetting to operate. a control unit that controls notification to the user;
An information processing device. When the user's situation based on the behavior information satisfies a predetermined condition, the user is present in the room, and the device to be monitored is not in operation within a preset estimated device operation time 2. The information processing apparatus according to claim 1, which performs control for notifying the user that the monitored device has forgotten to start. The control unit
determining the user's line-of-sight behavior toward the monitoring target device as the behavior information;
Furthermore, based on the determined line-of-sight behavior, it is inferred whether or not the user has an intention to operate the device to be monitored,
If the user has no intention to operate the device, it is determined that the predetermined condition is satisfied;
The information processing apparatus according to claim 2. The control unit
Determining gaze retention in a specific range including the monitoring target device as determination of the gaze behavior,
4. The information processing apparatus according to claim 3, wherein the presence or absence of the intention to operate the device is estimated based on the determination result of the line-of-sight retention. 5. The information processing apparatus according to claim 4, wherein said control unit counts the number of times of line-of-sight retention in said specific range, and estimates that there is no device operation intention when the number of times of line-of-sight retention is below a threshold. The control unit
determining whether the user is performing a specific action that requires operation of the monitored device;
If the user is performing the specific action, it is determined that the predetermined condition is satisfied;
The information processing apparatus according to claim 2. 7. The information processing apparatus according to claim 6, wherein said control unit determines an action of said tool based on sensing data of said tool used for said specific action as the determination of whether said specific action is being performed. . The information processing apparatus according to any one of claims 2 to 7, wherein said estimated device operation time is a time period during which it is assumed that said user performs an operation start operation of said monitoring target device. When notifying the user that the monitoring target device has forgotten to start operation, the control unit controls the time from a predetermined starting time based on the behavior of the user until the user performs an operation start operation of the monitoring target device. 9. The information processing apparatus according to any one of claims 1 to 8, wherein the processing is performed at the timing when the optimal device operation grace period, which is the optimum grace period, has passed. 10. The information processing apparatus according to claim 9, wherein said optimum device operation grace period is set by machine learning based on said user's action history and said monitoring target device's operation start time history. 11. The information processing apparatus according to claim 10, wherein the predetermined starting time based on the behavior of the user is the time of returning home. 12. The information processing apparatus according to claim 11, wherein said user's action history includes an outing time during which said user was out and a return time. 13. The information processing according to any one of claims 1 to 12, wherein said control unit notifies said monitoring target device of forgetting to operate according to inventory information of deliverables obtained from said monitoring target device. Device. 14. The information processing apparatus according to claim 13, wherein the control unit performs control to notify that the monitoring target device has forgotten to operate when the product obtained from the monitoring target device is out of stock. 15. The information processing apparatus according to claim 14, wherein said control unit also presents information on a substitute for said deliverable in said notification. The control unit presents to the user information on substitute products for deliverables obtained from the monitoring target device when the estimated device operation time has passed when the notification is given at the timing when the optimal device operation grace time has passed. 15. The information processing apparatus according to claim 13 or 14. the device to be monitored is a household electrical device;
The presence/absence of the user is the presence/absence of the user at home where the home electric device is placed.
The information processing apparatus according to any one of claims 1 to 16. the processor
Based on the user's behavior information related to the monitored device determined based on sensing data, the operation information of the monitored device, and the user's presence/absence information, it is possible to prevent the monitored device from forgetting to operate. A method of information processing, including controlling to notify a user. the computer,
Based on the user's behavior information related to the monitored device determined based on sensing data, the operation information of the monitored device, and the user's presence/absence information, it is possible to prevent the monitored device from forgetting to operate. A program that functions as a control unit that controls notification to the user.

Images