JP6347347B2 - Notification system, notification program, notification method, and notification device - Google Patents

Description

  The present invention relates to a notification system, a notification program, a notification method, and a notification device, and more particularly, to a notification system, a notification program, a notification method, and a notification device that notify information related to child behavior, for example.

  An example of background art is disclosed in Patent Document 1. The information processing apparatus disclosed in Patent Literature 1 predicts a user's emotion from the meta information of content viewed by the user, and recognizes the user's emotion from the prediction result and the user's facial expression, gesture, and the like. Information is recommended to the user based on the recognized emotion of the user.

JP 2007-41988 [G06T 7/20]

  However, in the information processing apparatus of Patent Document 1, in order to recognize the user's emotion, the user must continue to view the content. For example, even if an attempt is made to recognize a child's emotion using this information processing apparatus, the child's interest is likely to change, so that the child's emotion is difficult to be recognized properly.

  Therefore, a main object of the present invention is to provide a novel notification system, notification program, notification method, and notification device.

  Another object of the present invention is to provide a notification system, a notification program, a notification method, and a notification device that can reduce the burden on the childcare worker due to changes in children's emotions.

  The present invention employs the following configuration in order to solve the above problems. The reference numerals in parentheses, supplementary explanations, and the like indicate the corresponding relationship with the embodiments described in order to help understanding of the present invention, and do not limit the present invention.

  1st invention is the 1st acquisition means which acquires the sound information of the sound which generate | occur | produces in the space where a child exists, the prediction means which estimates whether a child will cry based on sound information, and a child crying behavior A notification system comprising first notification means for notifying a prediction result when predicted to be performed.

  In the first invention, in the notification system (100: reference numerals exemplifying corresponding parts in the embodiment, the same applies hereinafter), the first acquisition means (80, S5) is, for example, a space such as a house where a child is present. The sound information of the sound generated in the space is acquired using the microphone (14) provided in the inside. The predicting means (80, S43) predicts whether the child performs a crying behavior based on, for example, the maximum value of the sound volume among the sound information of the sound generated in the space. For example, when a crying behavior such as crying surprised by a loud sound is predicted, the first notification means (80, S47) notifies the prediction result.

  According to the first invention, since the childcare person can soothe the child before the child performs the crying behavior, the burden on the childcare person can be reduced.

  The second invention is dependent on the first invention and further comprises second acquisition means for acquiring position information of the child in the space, and the prediction means is a child's crying behavior based on the child's position information and sound information The notification system according to claim 1, wherein:

  In 2nd invention, a 2nd acquisition means (80, S41) acquires the positional information on the child in space using the some distance image sensor (12) provided in the ceiling of space. Then, based on the position information of the child and the sound information of the sound generated in the space, the predicting means predicts whether the child will cry.

  According to the second aspect, since the child's crying behavior is predicted using the child's position information, the accuracy of the prediction is improved.

  A third invention is dependent on the first invention or the second invention, and the sound information includes a maximum value of a sound volume generated in the space, and the predicting means is a child based on the maximum value of the sound volume. Predict what will cry.

  In the third invention, for example, if a loud sound suddenly occurs, the child is more likely to cry. Therefore, the predicting means predicts whether the child will cry based on the maximum value of the sound generated in the space.

  According to the third invention, the child's crying behavior can be predicted using the maximum value of the sound generated in the space.

  The fourth invention is dependent on any one of the first to third inventions, and the recognition means for recognizing the crying behavior performed by the child based on the sound information, and when the crying behavior of the child is recognized, Second notification means for notifying the recognition result is further provided.

  In the fourth invention, the recognition means (80, S35) recognizes the crying behavior based on, for example, the average and frequency of the sound volume among the sound information of the sound generated in the space. When the crying behavior is recognized, the second notification means (80, S39) notifies the recognition result of the crying behavior.

  According to the fourth aspect of the invention, the childcare worker can immediately go to the child even when the childcare worker cannot recognize the child who is crying.

The fifth invention is dependent on the fourth invention, the recognition means recognizes the cause of the crying behavior together with the child crying behavior, and the second notification means cry when the child crying behavior is recognized. Notify the cause of action.

  In the fifth invention, when the child performs crying behavior, the recognition means recognizes the cause of the crying behavior. When the child's crying behavior is recognized, the notification means also notifies the cause of the crying behavior.

  According to the fifth invention, the childcare person can know the cause of the child's crying behavior, and can soothe the child by snuggling up to the child's feelings.

  The sixth invention predicts whether the processor (80) of the notification system acquires sound information of sound generated in the space where the child is present (S5), and whether the child performs crying behavior based on the sound information And a notification program that functions as notification means (S47) for notifying the prediction result when the child is predicted to perform a crying action.

  In the sixth invention as well, as in the first invention, since the childcare person can soothe the child before the child performs crying behavior, the burden on the childcare person can be reduced.

  7th invention is an acquisition step (S5) which the processor (80) of a notification system acquires the sound information of the sound which generate | occur | produces in the space where a child exists, and predicts whether a child will cry based on sound information This is a notification method in which a prediction step (S43) to be performed and a notification step (S47) to notify a prediction result when the child is predicted to perform a crying behavior are performed.

  In the seventh invention as well, as in the first invention, the childcare person can soothe the child before the child performs crying behavior, and thus the burden on the childcare person can be reduced.

  The eighth invention relates to an acquisition means (80, S5) for acquiring sound information of a sound generated in a space where a child is present, and a prediction means (80, S43) for predicting whether the child performs a crying behavior based on the sound information. ), And a notification means (80, S47) for notifying a prediction result when a child is predicted to perform a crying action.

  In the eighth invention as well, as in the first invention, the childcare worker can soothe the child before the child performs crying behavior, so the burden on the childcare worker can be reduced.

  According to this invention, the burden on the childcare worker can be reduced.

  The above object, other objects, features, and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

FIG. 1 is an illustrative view showing an outline of a notification system according to an embodiment of the present invention. FIG. 2 is an illustrative view showing one example of a configuration of the notification system shown in FIG. FIG. 3 is a block diagram showing an example of the electrical configuration of the distance image sensor shown in FIG. FIG. 4 is a block diagram showing an example of the electrical configuration of the central controller shown in FIG. FIG. 5 is an illustrative view showing one example of a map of a space in which the notification system shown in FIG. 1 is used. 6 is an illustrative view showing one example of a configuration of an environmental data table stored in the memory of the central control unit shown in FIG. FIG. 7 is an illustrative view showing a list of causes of crying behavior recognized by the central control device shown in FIG. FIG. 8 is an illustrative view showing an example of a message notified from the central control unit shown in FIG. 4. FIG. 8A shows an example of a message notified when a crying behavior is recognized. B) shows an example of a message notified when crying behavior is predicted. FIG. 9 is an illustrative view showing one example of a memory map of a memory of the central control unit shown in FIG. FIG. 10 is a flowchart showing an example of acquisition processing of the processor of the central control unit shown in FIG. FIG. 11 is a flowchart showing an example of the notification process of the processor of the central controller shown in FIG.

  Referring to FIG. 1, a notification system 100 of this embodiment is used in a space (environment) such as a house where a child freely moves or a nursery school. Furniture such as a TV, sofa and table set is placed in the space, and children can move freely in the space. Parents and nursery teachers are also in the space, raising children and doing housework. Moreover, the childcare person possesses the portable terminal 10, and a plurality of distance image sensors 12 (12a, 12b,...) Are provided on the ceiling in the space, and a plurality of microphones 14 (14a, 14b,...) Are provided on the wall. ) Is provided.

  The mobile terminal 10 is a mobile phone also called a smartphone. The notification system 100 transmits (notifies) a message to the mobile terminal 10, and when the mobile terminal 10 receives the message, it notifies the childcare center of the reception of the message by voice or vibration. And the childcare person can confirm the message from the notification system 100 using the portable terminal 10. In another embodiment, a data terminal such as ipod touch (registered trademark) or PDA may be adopted as the mobile terminal 10.

  The plurality of distance image sensors 12 are provided on the ceiling of the space for the purpose of detecting child position information. The plurality of microphones 14 are provided on the wall of the space for the purpose of collecting sound (noise) generated in the space.

  The notification system 100 uses each distance image sensor 12 and each microphone 14 to predict and recognize the crying behavior performed by the child. Then, the result of prediction and recognition is notified to the mobile terminal 10.

  In addition, although the space of an Example is a house where a childcare person and a child exist, not only this but the notification system 100 can be used also in a kindergarten, a schoolchild nursery school, etc.

  For simplicity, only two microphones 14 are shown in FIG. 1, but more microphones 14 may be placed in the space, or one microphone 14 may be provided. And although only one childcare person (mobile terminal 10) is shown, there may be two or more childcare persons in the space.

  Further, when it is not necessary to distinguish the distance image sensors 12a, 12b,. When there is no need to distinguish between the microphones 14a, 14b,..., They are simply referred to as “microphone 14”.

  Referring to FIG. 2, central control device 16 of notification system 100 is also referred to as a notification device, to which distance image sensor 12 and microphone 14 are connected. Further, the central control device 16 performs wireless communication with the mobile terminal 10 via the network 1000. The central control device 16 acquires sensor information output by the distance image sensor 12 and the microphone 14 every first time (for example, 0.05 seconds). Then, the central control device 16 stores environmental data (see FIG. 6) including child position information and sound information of sounds generated in the space from the acquired information of each sensor in a table.

  FIG. 3 is a block diagram showing an electrical configuration of the distance image sensor 12. Referring to FIG. 3, distance image sensor 12 includes a control IC 60 and the like. An A / D converter 62, a camera 66, a depth sensor 68, a depth camera 70, an I / O 72, and the like are connected to the control IC 60.

  The control IC 60 includes a cache memory and the like, and controls the operation of the distance image sensor 12. For example, the control IC 60 operates according to a command from the central controller 16 and transmits the detected result to the central controller 16.

  A microphone 64 is connected to the A / D converter 62, and an audio signal from the microphone 64 is converted into a digital audio signal by the A / D converter 62 and input to the control IC 60. The sound collected by the microphone 64 is used to measure the volume of noise in the space together with the sound collected by the microphone 14.

  The camera 66 is a camera for photographing RGB information of the space in which the distance image sensor 12 is installed, that is, a color image. In addition, the camera 66 is provided in the distance image sensor 12 so as to be able to capture a space that is substantially the same as the space that is captured by the depth camera 70 described later.

  The depth sensor 68 is, for example, an infrared projector, and the depth camera 70 is, for example, an infrared camera. The depth sensor 68 irradiates the front of the distance image sensor 12 with, for example, infrared laser light. A special pattern is drawn in the space by the irradiated laser light, and the depth camera 70 captures the drawn pattern. The captured image is input to the control IC 60, and the control IC 60 analyzes the image to measure the depth information of the space irradiated with the laser light.

  The I / O 72 is a digital port capable of input / output control, and an audio signal, RGB information, and depth information are output from the output port and provided to the central controller 16. On the other hand, a control signal is output from the central control device 16 and applied to the input port.

  The distance image sensor 12 outputs RGB information and depth information and is sometimes called an RGB-D sensor.

  The distance image sensor 12 of the embodiment employs a product called Kinect (registered trademark) manufactured by Microsoft (registered trademark). However, in other embodiments, Xtion manufactured by ASUS (registered trademark), D-IMAGEr (registered trademark), which is a three-dimensional distance image sensor manufactured by Panasonic (registered trademark), and the like are employed as the distance image sensor 12. Also good.

  FIG. 4 is a block diagram showing an electrical configuration of the central controller 16. Referring to FIG. 4, the central controller 16 includes a distance image sensor 12 and a processor 80. The processor 80 is sometimes called a microcomputer or CPU. A plurality of distance image sensors 12, a memory 82, an A / D converter 84, an output device 86, an input device 88, a communication LAN board 90, and the like are connected to the processor 80.

  The distance image sensor 12 outputs depth information as described above. This depth information includes the shape of the child (person) in the space and the distance to the person. For example, when a person is sensed by the distance image sensor 12 provided on the ceiling, the shape of the head and both shoulders when the person is viewed from above and the distance to the head and both shoulders are obtained as depth information. .

  In addition, 35 distance image sensors 12 are installed at predetermined positions (known) in the space, and the processor 80 acquires depth information from each of the distance information sensors 12 so that the position information of the person in the space (world coordinate system) ( For example, the position coordinates (X, Y, Z) of feature points such as the center of gravity can be calculated.

  Further, since the distance image sensor 12 includes the microphone 64, the central control device 16 can also estimate the sound generation source from the position of the distance image sensor 12 to which sound information is input.

  In another embodiment, the position and posture of a person may be detected using a two-dimensional or three-dimensional LRF instead of the distance image sensor 12.

  The processor 80 includes an RTC that controls the operation of the central controller 16 and outputs date and time information. The memory 82 includes a ROM, an HDD, and a RAM. In the ROM and the HDD, a control program for controlling the operation of the central control device 16 is stored in advance. The RAM is used as a work memory or a buffer memory for the processor 80.

  A plurality of microphones 14 a and 14 b are connected to the A / D converter 84. The audio signal from each microphone 14 is converted into a digital audio signal by the A / D converter 84 and input to the processor 80. The sound collected by the microphone 14 is used for measuring the volume of noise in the space together with the sound collected by the microphone 64. In other embodiments, each microphone 14 may be independently connected to the processor 80. In this case, the processor 80 can specify the microphone 14 to which sound is input.

  The output device 86 is, for example, a display, and the input device 88 is, for example, a mouse or a keyboard. Then, a childcare worker or a space manager (for example, the head of the garden) can check and use the state of the central control device 16 using the output device 86 and the input device 88.

  The communication LAN board 90 is configured by a DSP, for example, and provides the transmission data provided from the processor 80 to the wireless communication device 92, and the wireless communication device 92 transmits the transmission data to the mobile terminal 10 via the network 1000. Further, the communication LAN board 90 receives data via the wireless communication device 92 and gives the received data to the processor 80.

  FIG. 5 is an illustrative view showing an example of a map of a space. This map corresponds to the space shown in FIG. 1 and shows the positions where the television (including the TV stand), the sofa and the table set are arranged. Moreover, although not shown on the map, height information (Z) is given to each furniture. For example, the distance from the floor to the surface of the TV stand is given as height information to the TV, the distance from the floor to the seat is given to the sofa as height information, and the seat from the floor to the chair is given to the table set. And the height from the floor to the table surface. These pieces of information are stored in the memory 82 of the central controller 16 as map data.

  FIG. 6 is an illustrative view showing one example of a configuration of an environment data table. The environmental data is data including the position information of the child at a certain date and time, the analysis result of the sound generated in the space, and the time related to the behavior of the child. The environmental data table stores environmental data including these pieces of information for each row. Referring to FIG. 6, the environment data table includes columns of date and time, children, voice, non-mealing time, and continuous waking time. The sound information columns include average, maximum, and frequency columns.

  The date and time column stores the date and time when the environmental data was acquired. As the format of the date and time stored, “yy” indicates “year”, “mm” indicates “month”, “dd” indicates “day”, and “H” indicates “hour”. , “M” indicates “minute”, and “S” indicates “second”.

  In the child column, “X, Y, Z” is stored as the position of the child. For example, by checking the position information of a child on a map, it is possible to determine whether the child collided with a person or an object or dropped. Further, it can be determined from the position information of the child that there is no movement because the child is asleep.

  In the average column of the sound information column, “AB” is stored as an average of sound volumes acquired from the distance image sensors 12 and the microphones 14. Also, “MB” is stored as the maximum volume of the sound acquired from each distance image sensor 12 and each microphone 14 in the maximum column of the sound information column. In the frequency column of the sound information column, “f” is stored as the frequency of the average volume.

  The average of sound “AB” indicates the average of the volume of noise in the space or the average of the volume of a child's cry. For example, it is used to determine whether or not the situation is always noisy because the child is scared and easily crying. In addition, when the child is crying, the volume of the crying voice changes depending on the cause of the crying behavior, so that the child is used to determine the cause of the crying behavior.

  The maximum “MB” of the sound indicates the maximum volume of noise in the space or the maximum volume of a child's cry. For example, a sudden loud sound makes it easier for a child to cry and is used to determine whether such a situation is present. When the child is crying, the volume of the cry changes depending on the cause of the crying behavior, so the maximum “MB” of the sound is used to determine the cause of the crying behavior.

  The sound frequency “f” indicates the frequency of the cry when the child is crying. For example, a child's cry may have a different frequency depending on the cause of the crying behavior. Therefore, the frequency “f” of the sound is used to determine the cause of the crying behavior.

  The unit of the average “AB” of the sound volume and the maximum “MB” of the sound is “dB”, and the unit of the sound frequency “f” is “Hz”.

  “FT” is stored as the time since the last meal of the child in the column of the time when the meal is not taken. Until the time “FT” is determined that the child has eaten, the corresponding variable continues to be counted. When the time “FT” is stored in the environmental data table, the time “FT” is calculated based on the corresponding variable. Remembered. Further, in the embodiment, whether the child has eaten is notified from the childcare person, and when this notification is given, the above-described variables are initialized. In another embodiment, the central control device 16 may determine whether the child has eaten from the position information and time of the child.

  “WT” is stored as the time since the child woke up in the column of the time of waking up. Until time “WT” is determined that the child has fallen asleep, the corresponding variable continues to be counted, and when time “WT” is stored in the environmental data table, time “WT” is stored based on the corresponding variable. The In the embodiment, whether or not the child has fallen is determined based on a change in the child's position information, and the variables described above are initialized when it is determined that the child has fallen. In other embodiments, it may be determined based on a notification from the childcare worker that the child has slept.

  Here, the position information of the child, the average and frequency of the sound in the sound information, the time of not eating and the time of waking up are information for recognizing the child's crying behavior. Sometimes called decision information. Moreover, since the maximum sound in the child position information and sound information is information for predicting the child's crying behavior, it may be referred to as second determination information.

  The central controller 16 stores such environmental data every first time in the environmental data table. The central control device 16 recognizes and predicts the child's crying behavior every second time.

  FIG. 7 is an illustrative view showing a specific cause of crying behavior. Referring to FIG. 7, in the embodiment, the cause of the crying behavior is recognized together with the crying behavior. The causes of child's crying behavior include “hungry”, “I want to sleep”, “surprised” and “painful”. If the cause of the crying behavior is recognized as “hungry,” the child is likely to be crying because they are hungry. If the cause of the crying behavior is recognized as “I want to sleep”, the child is likely to be crying because he / she became sleepy. If the cause of the crying behavior is recognized as “surprised”, the child is likely to be crying because of the surprise. If the cause of the crying action is “painful”, the child is likely to be crying because the child feels pain.

  In the embodiment, the cause is labeled as teacher data with respect to the first determination information when the child performs a crying action, and the teacher data is learned by a machine learning method such as SVM (Support Vector Machine). As a result, a discrimination model for recognizing crying behavior is created. If unknown first discrimination information for the second time is input to this discrimination model, at least one or more of the cause of the crying behavior and the probability indicating whether the crying behavior is being performed can be obtained. When the probability is greater than the first threshold, it is recognized that the crying behavior is being performed, and the cause of the crying behavior is output.

  For example, for the discrimination model for crying behavior, when the first judgment information when crying behavior is performed is input as unknown data, “hungry” with a probability of 90% and “pain” with a probability of 80% ”Is output, if the first threshold is 85%, it is recognized that the crying action is being performed due to“ hunger ”having a probability greater than the first threshold. If the first threshold value is 75%, it is recognized that the crying action is performed due to “hunger” having the highest probability. And if the 1st threshold is 95%, since each probability is smaller than the 1st threshold, the recognition result that crying action is not performed is outputted.

  With reference to FIG. 8A, for example, when a recognition result indicating that crying behavior is being performed due to “hungry” is output, the central control device 16 makes a response to the portable terminal 10 possessed by the childcare worker. , Send a message telling you that your child is crying because of "hungry". Thereby, for example, even if the childcare person is not near the child and does not recognize the child who is crying, the childcare person can immediately go to the child. In particular, the childcare person can know the cause of the child's crying behavior, and can soothe the child in close proximity to the child's feelings.

  In the embodiment, the teacher data is learned by a machine learning method such as SVM using the second determination information a predetermined time before the child is crying as teacher data. As a result, a discrimination model for predicting crying behavior that occurs within a predetermined time is created. When unknown second determination information for the second time is input to this determination model, the probability that a crying action will be performed within a predetermined time is output as a prediction result.

  Referring to FIG. 8B, for example, when the probability output as a prediction result is larger than the second threshold value, central controller 16 crys within a predetermined time against portable terminal 10 possessed by the childcare worker. Send a message telling you that will be done. As a result, the childcare worker can soothe the child before the child performs crying behavior, and thus the burden on the childcare worker can be reduced. In particular, in the embodiment, since the child's crying behavior is predicted using the position information of the child in addition to the sound information of the sound generated in the space, the prediction accuracy is improved.

  In addition, in an Example, recognition and prediction of a crying action are recognized for every 2nd time longer than 1st time required in order to acquire environmental data. That is, the crying behavior is recognized and predicted after the environmental data (first determination information and second determination information) necessary for recognizing and predicting the crying behavior is acquired. As a result, the child's crying behavior can be accurately recognized and predicted.

  The above has outlined features of the embodiment. Hereinafter, the embodiment will be described in detail using the memory map shown in FIG. 9 and the flowcharts shown in FIGS. 10 and 11.

  FIG. 9 is an illustrative view showing one example of a memory map of the memory 82 of the central controller 16 shown in FIG. As shown in FIG. 9, the memory 82 includes a program area 302 and a data storage area 304. The program storage area 302 acquires information from each sensor as a program for operating the central control device 16, recognizes and predicts the crying behavior and the acquisition program 310 for storing the environmental data in the environmental data table. A notification program 312 for notifying the result is stored. Although illustration is omitted, the program for operating the central control device 16 includes a program for creating a message to be notified.

  The data storage area 304 stores an environmental data table 330, map data 332, and the like, and is provided with a meal flag 334, a sleep flag 336, and the like. The environmental data table 330 is, for example, a table having the configuration shown in FIG. 6, and stores environmental data. The map data 332 is map data indicating positions where furniture is arranged as shown in FIG. 5, for example.

  The meal flag 334 is a flag used when measuring the time “FT” when no meal is taken. For example, the meal flag 334 is composed of a 1-bit register. When the meal flag 334 is turned on (established), a data value “1” is set in the register. On the other hand, when the meal flag 334 is turned off (not established), a data value “0” is set in the register. The meal flag 334 is turned on when, for example, the childcare person notifies that the child has eaten. When the meal flag 334 is turned on, a variable for counting the time “FT” not eating is initialized, and measurement of the time “FT” not eating is started.

  The sleep flag 336 is a flag used when measuring the time “WT” during which the user continues to wake up. The sleep flag 336 is turned on, for example, when it is determined that the child has gone to bed. When the sleep flag 336 is turned on, the count of the variable corresponding to the time “WT” during which the sleep is continued is stopped. Then, when it is determined that the child has woken up, it is turned off, the variable corresponding to the time “WT” that continues to wake up is initialized, and measurement of the time that has continued to wake up is started.

  Although not shown, the data storage area 304 is also provided with a buffer for temporarily storing the results of various calculations, and other counters and flags necessary for the operation of the central controller 16.

  The processor 80 of the central control device 16 processes a plurality of tasks including the acquisition process shown in FIG. 10 and the notification process shown in FIG. 11 under the control of the Linux (registered trademark) -based OS and other OSs. .

  FIG. 10 is a flowchart of the acquisition process. When the central controller 16 is turned on and an execution command for the acquisition process is issued, the acquisition process is executed. An execution instruction for the acquisition process is issued every first time.

  When the acquisition process is executed, the processor 80 acquires the current time in step S1. For example, the current time is acquired from the RTC that the processor 80 has. Subsequently, in step S3, the processor 80 acquires information of the distance image sensor 12. That is, the processor 80 acquires the depth information and the audio signal output from the distance image sensor 12.

  Subsequently, in step S5, the processor 80 acquires an audio signal from the microphone 14. That is, the processor 80 acquires a sound signal of the sound collected by the microphone 14 in order to calculate sound information. Subsequently, in step S7, the processor 80 calculates each time. In other words, each of the time “FT” not having a meal and the time “WT” continuing to wake up is calculated based on the corresponding variable. The processor 80 that executes the process of step S5 functions as a first acquisition unit.

  Subsequently, in step S9, the processor 80 creates environment data. That is, the child position information is calculated from the depth information, and the average, maximum, and frequency of the sound (noise) volume are calculated from the plurality of microphones 14 and the microphones 64 of the distance image sensor 12. These values and each time calculated in step S7 are used as one environment data.

  Subsequently, in step S11, the processor 80 stores the current time and environment data. That is, the environmental data is associated with the current time information acquired in step S <b> 1 and stored in the environmental data table 330. Then, when the process of step S11 ends, the processor 80 ends the acquisition process.

  FIG. 11 is a flowchart of the notification process. When the central controller 16 is turned on and a notification processing execution command is issued, the notification processing is executed. An instruction to execute the notification process is issued every second time.

  When the notification process is executed, the processor 80 determines whether or not there is a child in the space in step S31. That is, it is determined whether the position of the child is detected by the distance image sensor 12. If “NO” in the step S31, that is, if there is no child in the space, the processor 80 ends the notification process.

  If “YES” in the step S31, that is, if the position of the child is detected in the space, the processor 80 acquires the first determination information for the second time from the current time in a step S33. That is, the position information of the child, the average and frequency of the sound in the sound information, the time of not eating, and the time of waking up are read from the environment data table 330. Subsequently, in step S35, the processor 80 executes a crying action recognition process. That is, the first determination information for the second time is input to the determination model for recognizing crying behavior. The processor 80 that executes the process of step S35 functions as a recognition unit.

  Subsequently, in step S37, the processor 80 determines whether or not a crying action is being performed. That is, the processor 80 determines whether the cause of the crying behavior with a probability larger than the first threshold is output from the determination model. If “YES” in the step S37, that is, if it is recognized that the child is performing a crying action, the processor 80 notifies that the crying action is performed in a step S39. For example, as shown in FIG. 8A, a message notifying the cause of crying behavior is transmitted from the central control device 16 to the portable terminal 10. The processor 80 that executes the process of step S39 functions as a second notification unit.

  On the other hand, if “NO” in the step S37, for example, if a recognition result indicating that the crying action is not performed is output, the processor 80 in the step S41 obtains the second determination information for the second time from the current time. get. That is, the maximum sound in the child position information and sound information is read from the environment data table 330. Subsequently, in step S43, the processor 80 executes a crying behavior prediction process. For example, the second determination information for the second time is input to a determination model that predicts crying behavior. Note that the processor 80 that executes the process of step S41 functions as a second acquisition unit. Moreover, the processor 80 which performs the process of step S43 functions as a prediction means.

  Subsequently, in step S45, the processor 80 determines whether or not the probability of the prediction result is larger than the second threshold value. That is, it is determined whether the probability that the crying action is performed after a predetermined time is greater than the second threshold. If “YES” in the step S45, that is, if the probability that the crying action is performed after a predetermined time is larger than the second threshold value, the processor 80 notifies that the crying action is performed within the predetermined time in a step S47. For example, as shown in FIG. 8B, a message notifying that a crying action is performed within a predetermined time is transmitted from the central control device 16 to the mobile terminal 10. The processor 80 that executes the process of step S47 functions as a first notification unit.

  Then, if “NO” in step S45, that is, if the probability of the prediction result is smaller than the second threshold value, or if the process of step S47 ends, the processor 80 ends the notification process.

  In the embodiment, SVM is adopted as the machine learning method, but in other embodiments, an algorithm such as multiple regression analysis, neural network, or C4.5 may be adopted. For example, when multiple regression analysis is employed, the coefficient is calculated using each value of the first determination information or the second determination information as an explanatory variable. In this case, when the probability of the recognition result of the crying behavior is P and the sound volume when the child falls or collides is X, the mathematical formula 1 is obtained by multiple regression analysis.

[Equation 1]
P = aX
At this time, when the volume X is “80 dB”, the coefficient a is “a = 1/80”, and when the volume X is “80”, “P = 1”, that is, “100%”.

  In other embodiments, a simple threshold process may be used without using machine learning. When the average sound in the space is high, the child is likely to be afraid and cry. Therefore, when the average of the sound is higher than the first predetermined value, it may be predicted that the crying behavior will be performed within a predetermined time. In other words, the child's crying behavior can be predicted using the maximum value of the sound generated in the space.

  Furthermore, the frequency of the cry when the child feels pain is often greater than the second predetermined value. Therefore, when the frequency of the sound is higher than the second predetermined value, it may be recognized that a crying action is performed with a pain. In other words, the child's crying behavior can be recognized using the frequency of the sound generated in the space.

In yet another embodiment, in addition to the child position information, the child's posture (for example, head orientation (θ _H ) and shoulder orientations (θ _B )) may be calculated.

  In addition, the environmental data of other embodiments may include a result of facial recognition (expression) of the child, the intensity of the bass sound, the content of the content output from the television, the posture of the child, and the like. In addition, when it is possible to identify a collision or a drop that occurs in the space, sound information of the sound at that time may be included. When recognizing the crying behavior, the child's facial expression, the child's posture, the maximum sound may be used. Also, when predicting crying behavior, children's facial expressions, changes in children's location information (changes in the height direction due to falling), sound average, bass intensity, sound volume at the time of collision / dropping, content May also be used.

  Further, the first threshold value for recognizing the crying behavior and the second threshold value for predicting the crying behavior may be arbitrarily set by the childcare worker, or may be automatically set by the central control device 16. .

  The message transmitted to the mobile terminal 10 may be transmitted by e-mail or may be transmitted using dedicated software. Further, the destination of the message transmission is not limited to the portable terminal 10 and may be transmitted to a terminal such as a PC. Further, instead of the text message, video or audio that conveys the situation from the time when the crying behavior is recognized or predicted to the second time before may be transmitted to the mobile terminal 10.

  Further, the cause of the crying behavior may be different from that in the embodiment or the number may be different.

  In other embodiments, the crying behavior may be recognized or predicted using only the sound information of the space. In the notification process, the crying behavior may be predicted and notified, or the crying behavior may be recognized and notified. Further, the crying behavior may be recognized after the crying behavior is predicted.

  In the above-described embodiment, the word “greater than” is used for the threshold value, but “greater than the threshold value” includes the meaning of “greater than or equal to the threshold value”. Further, “smaller than a threshold” includes the meanings of “below the threshold” and “below the threshold”.

  In addition, the plurality of programs described in the present embodiment may be stored in the HDD of a data distribution server and distributed to a system having the same configuration as that of the present embodiment via a network. In addition, storage programs such as CDs, DVDs, and BDs (Blu-ray (registered trademark) Disc) are sold or distributed with these programs stored in storage media such as USB memory and memory card. May be. When the plurality of programs downloaded through the server and storage medium described above are applied to a system having the same configuration as that of this embodiment, the same effect as that of this embodiment can be obtained.

  The specific numerical values given in this specification are merely examples, and can be appropriately changed according to a change in product specifications.

DESCRIPTION OF SYMBOLS 10 ... Portable terminal 12a, 12b ... Distance image sensor 14a, 14b ... Microphone 16 ... Central control device 64 ... Microphone 80 ... Processor 82 ... Memory 100 ... Notification system 1000 ... Network

Claims (8)

First acquisition means for acquiring sound information of a sound generated in a space where a child is present;
A notification system comprising: prediction means for predicting whether the child performs crying behavior based on the sound information; and first notification means for notifying a prediction result when the child is predicted to perform crying behavior. Further comprising second acquisition means for acquiring position information of the child in the space;
The notification system according to claim 1, wherein the prediction unit predicts whether the child performs a crying behavior based on the position information of the child and the sound information. The sound information includes a maximum value of a sound volume generated in the space,
The notification system according to claim 1, wherein the prediction unit predicts whether the child performs crying behavior based on the maximum value of the volume. Recognizing means for recognizing the crying action performed by the child based on the sound information; and
The notification system according to claim 1, further comprising second notification means for notifying a recognition result when the child's crying behavior is recognized. The recognizing means recognizes the cause of the crying behavior together with the crying behavior of the child,
The notification system according to claim 4, wherein when the child's crying behavior is recognized, the second notification means notifies the cause of the crying behavior. Notification system processor,
An acquisition means for acquiring sound information of sounds generated in a space where children are present,
A predicting means for predicting whether the child performs a crying behavior based on the sound information, and a notification program that functions as a notification means for notifying a prediction result when the child is predicted to perform a crying behavior. The notification system processor
An acquisition step for acquiring sound information of a sound generated in a space where a child is present;
A notification method for executing a prediction step of predicting whether or not the child performs a crying behavior based on the sound information, and a notification step of notifying a prediction result when the child is predicted to perform a crying behavior. An acquisition means for acquiring sound information of sounds generated in a space where children are present,
A notification device comprising: prediction means for predicting whether the child performs crying behavior based on the sound information; and notification means for notifying a prediction result when the child is predicted to perform crying behavior .

Images