JP2019169011A - Symptom detection system and program - Google Patents

Abstract

To provide a symptom detection system which can precisely grasp various fluctuation of states related to a symptom and can easily collect data for learning.SOLUTION: A symptom detection system 10 is arranged by providing state data acquiring means 40 for continuously or repeatedly acquiring perception data which can be recognized with human sense as a state data of a symptom detection object on which any event may occur, a symptom feature extracting device 50 for, by using the acquired state data, carrying out pattern recognition for outputting whether or not the state of the symptom detection object falls within a specific state related to a symptom or its degree and for extracting symptom data obtained by this pattern recognition, a symptom detecting device 60 for carrying out time-series pattern recognition by using the extracted time-series feature data and for carrying out processing for outputting a score indicating degree of existence or absence of any symptom.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a sign detection system and program for detecting signs that occur before the occurrence of an event, such as livestock delivery, prisoner suicide in a cell, the onset of depression, the occurrence of an accident or dangerous behavior, the collapse of a tunnel, It can be used to detect signs of river floods, disasters such as volcanic eruptions, crimes such as theft and terrorist attacks.

  In general, signs of various characteristic events (events) that occur in the world are detected by various methods in order to reduce damage and risk at the time of occurrence. For example, in beef cattle breeding farmers, calving stillbirth at delivery is a major loss, so it is important to detect signs of calving and avoid stillbirth. For this reason, in recent years, a system for detecting signs of calving of cows has been developed, and its sales are also progressing.

  By the way, the bovine parturition sign detection system conventionally used is generally configured to detect the state of the cow by attaching a contact sensor to the cow. However, such a delivery sensory detection system using a contact sensor is expensive and highly invasive (possibly giving a stimulus that disturbs the homeostasis of the internal environment of the living body). Is also accompanied.

  Therefore, if the condition of the cow can be detected without contact, the invasive problem can be solved, the danger at the time of installation can be avoided, and furthermore, the condition of the cow can be detected using a camera. If it is possible, a relatively inexpensive sign detection system can be obtained.

  From the viewpoint of detecting a state using a camera, for example, a bathroom abnormality detection device that detects an abnormality in a bathroom is known (see Patent Document 1). In addition to cattle delivery, from the standpoint of detecting signs, there is a known sign detection device that can appropriately determine the data reference period in detecting signs of state change using measurement data from multiple sensors. (See Patent Document 2).

  Furthermore, as a vectorization technique used in the present invention, for example, a technique related to i-vector is known (see Non-Patent Document 1).

JP 2017-40989A Japanese Patent Laid-Open No. 2017-204017

Keisuke Fukuchi, Naohiro Tsuji, Tetsuji Ogawa, Tetsunori Kobayashi, "Non-negative matrix decomposition using utterance similarity based on i-vector and application to speaker clustering", Information Processing Society of Japan Research Report, July 20, 2012 Vol. 2012-SLP-92, no. 8

  As mentioned above, if a cow is used to detect the state of a cow without contact, a cow delivery sign detection system that is relatively inexpensive, non-invasive, and has little risk during installation can be realized. Can do.

  However, the state changes before calving in cows are various, and the normal state is observed even before the calving, and the state observed in the normal state is related to the sign. In other words, the cattle states that can be considered as signs are those that may appear at all times, including normal (regardless of whether they are normal or predictive). It is included.

  Therefore, even if the state of the cow is detected using the camera and the state monitoring is simply performed by the pattern recognition process using the detected image, at least effective prediction of the calving of the cow cannot be performed. If you capture the state that appears for the first time when the event occurs, the state will change sporadically or intermittently until the event occurs, or the state will gradually change toward the event occurrence In any case, simple pattern recognition processing using camera images can be used to monitor the state, but not only when it is a sign but also when there is a possibility that it will always appear, including sign detection This is because it is difficult to clearly determine whether or not the state appears as a sign by the conventional state monitoring by simple pattern recognition processing.

  In addition, if pattern recognition that takes into account temporal factors is considered, it is possible to detect signs even by capturing a state that may appear at all times. In order to take into account such elements, it is also conceivable to simultaneously input state data acquired at a plurality of times to the pattern recognizer. However, it is not practical to input a large amount of state data over a length of time that can detect the sign into the pattern recognizer because of the enormous amount of processing.

  In the specification of the present application, the term “normal” is used to mean a time other than the predictive time for the target event for which the predictor is to be detected. Thus, for example, when a specific major illness that leads to death is the target event, assuming that a “cough or runny nose” appears as a predictor of the major illness, you are only suffering from a mild cold. If you have a cough or runny nose, it means “a cough or runny nose” that appears in normal times. When we have a mild cold, we think that it is normal to think that it is normal, but mild cold is not the target event. Since it is considered as an event, cough and runny nose when a mild cold occurs, it is appearing at a time other than the predictive time of the target event (that is, normal in the present specification) "It turns out that.

  In order to perform pattern recognition processing, it is necessary to prepare a model by learning, but cow calving is not a frequent occurrence and should be forced to occur at a convenient time. However, there is also a problem that it is difficult to collect data for learning. This means that the model cannot be updated frequently, so it is difficult to improve the performance of the pattern recognizer at an early stage or to improve it step by step at high speed. It leads to.

  The above-described problems are not limited to the detection of cattle calving signs, but can occur in the same way when detecting signs that occur before the occurrence of various events.

  An object of the present invention is to provide a sign detection system and a program that can accurately grasp various state fluctuations related to a sign and can easily collect data for learning.

  <Basic configuration of the invention>

The present invention is a sign detection system for detecting a sign that occurs before the occurrence of an event,
State data acquisition means for continuously or repeatedly acquiring data including perceptual data that can be recognized by human perception as state data indicating a state of a sign detection target that may cause an event;
Using the state data acquired by the state data acquisition means, the state of the detection target is at least a predetermined one related to the predictor and related to the predictor including a specific state that is likely to appear at all times. An indication that a pattern recognition process that outputs data indicating whether or not one of the specific states corresponds or a degree of similarity to the specific state is executed, and a process of extracting feature data obtained by the pattern recognition process is repeatedly executed A feature extractor;
A predictive detector that performs time-series pattern recognition processing using the time-series feature data extracted by the predictive feature extractor and outputs a score indicating the presence / absence of presence / absence of a sign. To do.

  Here, an “event” is a characteristic event. For example, in the case of a system that detects a sign of cattle delivery, it is cow delivery, and “a sign detection target” is a cow. Yes, the “specific state” means, for example, a standing state of a cow, a raised state of a cow's tail (a continuous value such as the raising angle of a cow's tail or a range to which the continuous value belongs is a threshold value or a range. It is data showing the degree of similarity to the raised state of the cow's tail.), The exposed state of the cow's allanum / amniotic membrane, and the standing state of the cow (standing or sitting within a predetermined short time) State of rotation) around a cow's leg, a characteristic state of cow's cry, etc. In addition, the standing state of the cow and the rotating state with the one leg of the cow as the axis indicate the state that can be detected in a short time. The state shown is included. Details will be described later in [Description of Embodiments].

  In addition, “perception data” in the description “data including perceptual data recognizable by human perception” in “state data acquisition means” refers to image data (thermographic data or edge image) that can be recognized mainly by human vision. Processed image data, etc.), or sound data that can be recognized by human hearing, but may also be, for example, odor data. Furthermore, “data including perceptual data” does not use only perceptual data for pattern recognition processing by “predictive feature extractor”, but perceptual data such as output data from temperature sensors, acceleration sensors, etc. Data other than that may be used for pattern recognition processing.

  In addition, in the “predictive feature extractor”, “predictive at least one specific state related to the predictor including a specific state that is related to the predictor and may always appear” `` Related to '' means a specific state that has been found empirically or statistically to appear as a sign, and `` may appear at all times '' means whether it is a sign or not , Meaning a specific state that can appear at any time. “Predetermined” means a specific state selected in advance when constructing the system. Therefore, when learning for pattern recognition, the specific state is used for the state data used for learning. The tagging about is done.

  Furthermore, since it is “at least one specific state”, the number of “specific states” used for predictive detection may be one or plural, and in one case, that one specific state is “predictive”. It is related to "and is a specific state that may always appear". Even if a particular state is a “possibly appearing” state, there may be a change in the frequency of appearance of the state, or a characteristic pattern of the appearance interval (for example, several seconds When a temporal element is added as in the case where there is a characteristic appearance pattern that repeats appearing in two consecutive times), it is possible to detect a sign even if there is only one specific state.

  However, from the viewpoint of improving the certainty of predictive detection, in other words, predictive detection with even fewer detection omissions (that is, high recall) and fewer false detections (that is, high accuracy). From the viewpoint of realizing the above, it is desirable to use a plurality of “specific states”. That is, the predictive feature extractor may be configured to execute a pattern recognition process that outputs data indicating whether or not each of a plurality of specific states corresponds to each of the specific states. desirable. The plurality of “specific states” in the case of such a configuration include at least one “specific state that is related to the sign and may always appear”. . Therefore, “a specific state that is related to the sign and may appear at all times” is α, and “a specific state that is related to the sign and appears only at the time of the sign (does not appear in normal times)” is β Then, when α is 2 or more and there is no β (when all are α), there is only one α, but β is also 1 or more (α and β are mixed, α is There is a case where α is 2 or more and β is 1 or more (a case where α and β are mixed and α is plural).

  Further, the data indicating “whether or not” in “data indicating whether or not a specific state is applicable or the degree of similarity to the specific state” of the “predictive feature extractor” is In this way, it means data indicating the classification result in the case of being classified into two values (two patterns), and includes a likelihood indicating the certainty of corresponding to each of these patterns. In addition, “data indicating the degree of similarity” is, for example, when a specific state is determined by a threshold value for a continuous value, or in a range to which the continuous value belongs, the continuous value itself, or continuous This means data indicating which range the value belongs to (including the likelihood indicating the likelihood of belonging to each range), or if a continuous value is not involved, for example, “ Classified into three or more values (three patterns), such as “upper, middle, lower”, “large, medium, small”, “◎, ○, Δ, x”, “1, 2, 3, 4, 5”, etc. This means that the data indicates the classification result (including the likelihood indicating the certainty of corresponding to each pattern). Therefore, for example, when classified as “upper, middle, lower”, each of the subdivided “upper” state, “middle” state, and “lower” state is a “specific state” in the present invention. Rather, in the present invention, at least one of these three states is a “specific state” related to the predictor, for example, an “upper” state is a “specific state”. If so, the “middle” state and the “down” state are states other than the “specific state” that differ in the degree of similarity to the “specific state”. If the “middle” state is a “specific state”, the “down” state is a state other than the “specific state”.

  When there are a plurality of “specific states”, the number of “specific states” and the number of “pattern recognition processes” in the “predictive feature extractor” may or may not match. It does not have to be. Therefore, the “predictive feature extractor” is configured to perform determination output for each of a plurality of specific states by separate pattern recognition processing (that is, perform determination output for one specific state by one pattern recognition processing). , A configuration in which a plurality of them are prepared (see FIG. 3 to be described later), a configuration in which determination outputs for each of a plurality of specific states are collectively performed in one pattern recognition process (see FIG. 2 to be described later), and a configuration in which these are combined (For example, when there are specific states X, Y, and Z, each determination output for the specific states X and Y is performed in the pattern recognition process P, and the determination output for the specific state Z is the pattern recognition process Q. Any of the configurations performed in step 1) may be used. At this time, if the number of “pattern recognition processes” is two or more, different types of pattern recognition processes may be used. This is because there are a plurality of types of pattern recognition processes that can be adopted as described below. For example, when there are specific states X, Y, and Z, each determination output for the specific states X and Y is performed by a neural network, and the determination output for the specific state Z is a hidden Markov model (HMM). This is a configuration in which these are combined and input to the “predictor detector” at the subsequent stage.

  In addition, as the “pattern recognition process” in the “predictive feature extractor”, for example, a neural network can be adopted, and a convolutional neural network (CNN) is particularly suitable. Hidden Markov models (HMM), support vector machines (SVM), Bayesian networks, etc. can be employed.

  Furthermore, as the “time-series pattern recognition process” in the “predictor detector”, for example, a recurrent neural network (RNN) can be suitably used. In addition, for example, a hidden Markov model (HMM), etc. Can be adopted.

  The predictive detection system according to the present invention is configured to perform two-stage processing using the “predictive feature extractor” in the preceding stage and the “predictive detector” in the subsequent stage. Further, a configuration in which a plurality of stages of pattern recognition processing is performed may be employed, and a “predictor detector” in the subsequent stage may be configured to perform a plurality of stages of processing (time-series pattern recognition processing or processing in place thereof). Even when such a configuration is used, since it is roughly divided into two stages, the present specification will be described as including two stages including such a configuration.

  <Operations and effects obtained from the basic configuration of the invention>

  In such a sign detection system of the present invention, the sign detection is performed by a two-stage processor including a “prediction feature extractor” in the preceding stage and a “signature detector” in the subsequent stage. Are responsible for different roles.

  The “predictive feature extractor” in the previous stage captures a specific state related to a sign by pattern recognition processing, but does not recognize whether the state has appeared as a sign. Execute. Accordingly, processing for capturing a specific state is performed, but it is not determined whether or not the captured state is a sign. Then, such pattern recognition processing is repeated.

  On the other hand, the “predictor detector” in the subsequent stage detects the sign using time-series feature data obtained by repeating the pattern recognition processing by the “predictive feature extractor” in the previous stage. Therefore, whether or not the feature data obtained by the “predictive feature extractor” in the previous stage is information on a specific state appearing as a predictor by performing processing that takes into account temporal factors Determine.

  For this reason, even if the specific state related to the sign includes a specific state that may always appear, effective sign detection can be performed. That is, for a specific state that may appear at all times, even if the state can be captured by only one stage of pattern recognition processing, it is clear whether or not the captured state appears at the sign. However, it is difficult to detect signs, and it is also possible to simultaneously input the state data acquired at multiple times to the pattern recognizer in order to take the time factor into consideration. It is not practical to input a large amount of state data over a length of time enough to enable predictive detection into a pattern recognizer, resulting in a huge amount of processing. On the other hand, in the present invention, regardless of whether or not it appears as a sign by the “prediction feature extractor” in the preceding stage, it captures a specific state and appears as a sign by the “predictor detector” in the subsequent stage. Therefore, it is possible to detect the sign even if the specific state related to the sign includes a specific state that may always appear.

  In addition, because it is divided into a “predictive feature extractor” in the first stage and a “predictor detector” in the second stage, it is possible to increase the frequency of learning and speed up the performance improvement of the sign detection system. It becomes. This is because an event for which a sign is desired to be detected may not be a frequently occurring event or may be an event that cannot be forcibly generated at a convenient time. In such a case, it is difficult to collect data for learning. Assuming that the sign detection is performed by one-step pattern recognition processing, the pattern is rarely generated until several rare events occur. The model used in the recognition process cannot be updated, and the performance improvement of the sign detection system is considerably slow. On the other hand, in the present invention, the “predictive feature extractor” in the previous stage is captured regardless of whether or not it has appeared as a predictor, and therefore the learning data for that is the occurrence of an event. Can be collected at any time regardless. Therefore, since the model used in the pattern recognition processing in the “predictive feature extractor” in the previous stage can be updated frequently, the performance of the predictive detection system as a whole can be improved at its update speed. become.

  Furthermore, as described above, when the “predictive feature extractor” in the previous stage is learned and the model is updated, the state data used for learning includes “perceptual data that can be recognized by human perception”. Therefore, the tagging operation (annotation) for the perceptual data can be realized by crowdsourcing. This perceptual data is suitable for crowdsourcing in that it can be perceived by human perception (for example, the state can be easily identified when viewed or heard by a person). Next, since it is the state data input to the “predictive feature extractor” in the previous stage, it is possible to identify the state regardless of the event (for example, cattle delivery) or its predictor. This is because it is suitable for crowdsourcing in that anyone can identify the state without any expertise on the indication. For this reason, it is possible to collect a large amount of data for learning relatively inexpensively, quickly, and without much effort.

  Further, since perceptual data is used as the state data, for example, a non-contact type state data capturing device such as a camera or a microphone can be used. For this reason, non-invasiveness by avoiding attachment to a living body and avoidance of danger at the time of installation can be realized, and a system can be constructed using a relatively inexpensive device such as a camera or a microphone. And the above-mentioned purpose is achieved.

  <Configuration for extracting the intermediate layer output of a neural network>

In addition, in the predictive detection system described above,
The predictive feature extractor
It is desirable that the pattern recognition process is executed by a neural network, and the process of extracting the output data of the intermediate layer reaching the output node corresponding to at least one specific state is executed as the feature data.

  Here, as the “neural network”, for example, a convolutional neural network (CNN) or the like can be preferably used.

  The “output node” is a node constituting an output layer which is the final layer of the neural network. And, “an intermediate layer that leads to an output node corresponding to at least one specific state” means that if there is only one specific state, the specific state always corresponds to the specific state. This means extracting the middle layer output leading to the output node. On the other hand, when there are a plurality of specific states, it is not necessary to extract the intermediate layer output reaching all output nodes corresponding to the specific states for all of the plurality of specific states, and the final layer (output layer) This means that a specific state for extracting the output may be included. For example, when there are specific states X, Y, and Z, the determination output for each of the plurality of specific states X, Y, and Z is collectively performed by one pattern recognition process (see FIG. 2)), the network from the input layer to the extracted intermediate layer is shared, and therefore, the intermediate layer output is inevitably output for all of the specific states X, Y, and Z. Will be extracted. On the other hand, in the case where the determination output for each of the plurality of specific states X, Y, and Z is performed by separate pattern recognition processing (see FIG. 3 described later), for example, for the specific states X and Y Means that the output of each intermediate layer may be extracted, and for a specific state Z, the output of the final layer (output layer) may be extracted.

  Furthermore, the “intermediate layer” to be extracted is not limited to one layer, and may be a plurality of layers. That is, a plurality of intermediate layers may be extracted, and the intermediate layer outputs of the plurality of layers may be collected and input as feature data to the “predictor detector” at the subsequent stage. The term “multiple layers” here means a plurality of layers extracted from a network reaching the same output node. For example, in a network reaching an output node corresponding to one specific state X, This means that the second intermediate layer and the fourth intermediate layer are extracted from the output layer (final layer), and both are used as feature data. Thus, for example, for the intermediate layer leading to the output node corresponding to the specific state X, the previous intermediate layer is extracted, and for the intermediate layer reaching the output node corresponding to the specific state Y, the previous four layers are extracted. It does not mean that the intermediate layer is extracted, but such extraction means that one intermediate layer is extracted from each specific state.

  In this way, when the output of the intermediate layer of the neural network is extracted, compared to the case of extracting the output of the final layer (output layer), the subsequent stage of “predictive detection” that performs processing using time-series feature data It is possible to input information effective for subsequent processing as feature data to the “device”. In other words, the output data of the final layer (output layer) is data that only indicates whether or not it falls under a specific state, the amount of information is small, and information that can be used in subsequent processing is missing. Network from the state data input to the input layer of the neural network (a very large amount of information) to the output data of the final layer (the output layer) (a much narrowed amount of information) An intermediate amount of information existing above can be extracted and used in subsequent processing. In addition, since the output data of the final layer (output layer) has a small amount of information, if the determination result is incorrect, it has a large effect on the subsequent processing, but the intermediate layer output is used in the subsequent processing. Thus, the influence can be reduced.

  <Configuration to output scores corresponding to multiple predictive time intervals>

  In addition, the “predictor detector” in the latter stage divides the time zone before the occurrence of the event into one part (one time point) and divides it into two parts. And the other non-interesting section (the section that is far from the time of the event and is considered to be a sign that no sign appears) Although a score indicating the result of predictive detection can be output, in order to perform more detailed predictive detection, a plurality of predictive time intervals are set as follows (thus, at two or more (two time points) or more) It is desirable to divide into three or more and set a plurality of predictive time intervals and other non-target intervals).

In other words, in the aforementioned sign detection system,
The sign detector
It is desirable that a score corresponding to a plurality of predicting time sections set as sections in which a sign may appear by dividing a time zone before the occurrence of an event is output.

  Here, typical formats for setting the “predictive time interval” include the following two formats A and B non-target intervals. However, it is not limited to these two formats A and B. When dividing (N + 1) by dividing at N points (N time points), N predictive time intervals are set. In this case, the event occurrence time point is set to T0, and each time point of the break is set in order from the nearest (T -1), (T-2), (T-3), (T-4), ..., (TN).

  As shown in FIG. 4 to be described later, format A is a setting format in which a plurality of predictive time intervals do not overlap, and the portion before (TN) is a non-target interval, and (TN) to (T- ( N-1)), ..., (T-4) to (T-3), (T-3) to (T-2), (T-2) to (T-1), (T-1) to T0 is each predictive time interval.

  As shown in FIG. 5 to be described later, the format B is a setting format in which there are overlaps in a plurality of predictive time sections, and the section before (TN) becomes a non-target section, and (TN) to T0,. (T-4) to T0, (T-3) to T0, (T-2) to T0, and (T-1) to T0 are the predictive time intervals.

  As another setting format, for example, there is a case where a non-target section is set between a predictive time section and another predictive time section that does not overlap therewith. This is because, depending on the event, for example, there may be cases where signs appear in a concentrated manner in two periods (a plurality of distant periods), approximately one week before and approximately one month ago. Even in such a case, the predictive time interval may be set including an intermediate period (for example, a period between about one week before and about one month before).

  In this way, when it is configured to output scores corresponding to a plurality of predictive time intervals, it is possible to perform more detailed predictive detection, and it is also possible to adopt various methods for outputting the result of predictive detection It becomes possible to improve the usability of the system by the user.

  <Configuration to display composite distribution using prior information>

Furthermore, in the aforementioned sign detection system,
Prior information storage means for storing advance information that is predicted or scheduled in advance about the occurrence time of the event,
A composite distribution obtained by combining the probability distribution of event occurrence created using the prior information stored in the prior information storage means and the probability distribution of event occurrence created using the score output from the sign detector A composite that is generated using the score output from the predictor detector by displaying the prior information stored in the advance information storage means together with the feature data from the predictor feature extractor into the predictor detector It is desirable to have an output means for executing processing for displaying the distribution.

  Here, the “composite distribution” may be displayed on the time axis, for example, or may be displayed on the calendar using color shading.

  In this way, when the composition distribution is displayed using the prior information, it is possible to display the event distribution probability distribution created using the prior information gradually, and so on. Usability is improved. For example, when the display time (currently today) is far from the expected time or scheduled time of event occurrence, the probability distribution of event occurrence based on prior information is displayed almost as it is, and the expected time or schedule of event occurrence is displayed as it is. As the time approaches, the occurrence probability of the sign increases, so that it is possible to perform display reflecting the probability distribution of event occurrence based on the acquired state data.

  <Configuration to display event occurrence probability on calendar>

In the configuration for outputting scores corresponding to the plurality of predictive time intervals described above,
An output means for executing a process of displaying an event occurrence probability on a calendar using a score output from a predictive detector;
This output means is
If the time length does not match or the section is different between at least one of the multiple predictive time intervals and the calendar unit interval, the predictive time interval corresponds to the calendar unit interval And assigning to the calendar unit section by allocating the score corresponding to the predictor time section and allocating the score corresponding to the predictor time section, or each corresponding to a plurality of predictor time sections. It is desirable that at least a part of each score be assigned to the same unit section of the calendar.

  Here, the “calendar unit section” means the time length of the minimum unit divided for the description of the schedule (information input by the user, display of the input information, display of the presentation information from the supplier, etc.) It is a section and means a section having a time length suitable for indicating the probability of occurrence of an event in the section. For example, when the schedule is described in units of one day in a normal one-year calendar, the unit interval is one day (24 (24) divided from midnight to midnight the next day. More specifically, the probability of occurrence on April 1 and the probability of occurrence on April 2 are displayed. In addition, when the schedule is described by dividing the day into morning and afternoon in a normal one-year calendar, the unit interval is 12 hours divided by midnight and noon. Specifically, the occurrence probability of April 1 AM, the occurrence probability of April 1 PM, the occurrence probability of April 2 AM, the occurrence probability of April 2 PM, and the like are displayed. In addition, if the detailed description of the weekly calendar is divided into hour units such as 8 am to 9 am and 9 am to 10 am, the unit interval is 1 hour. Specifically, the probability of occurrence from 8 am to 9 am on April 1 and the probability of occurrence from 9 am to 10 am on April 1 are displayed.

  In addition, “displaying the event occurrence probability on the calendar” displays a numerical value (text number) indicating the probability of occurrence, and displays icons and colors such as face marks according to the numerical value. To do.

  When the event occurrence probability is displayed on the calendar as described above, and the score is divided or summed according to the difference in the time length of the section or the shift of the section (see FIG. 9 described later). In addition to realizing easy-to-view display, it is also possible to display information related to sign detection together with general schedule information on the calendar, it is possible to realize a user-friendly system. In addition, since the scores are distributed and summed, it is possible to cope with switching display of calendar unit sections by the user. In addition, since the score is distributed and summed, the current position (score output time), which is the starting point for displaying the sign detection result, moves on the calendar as time passes. It becomes possible to respond. This also facilitates the display of the composite distribution using the above-described prior information.

  <Detailed configuration regarding distribution display>

  Then, in the configuration for outputting the scores corresponding to the plurality of predictive time intervals described above, in order to smoothly connect or consistently connect the probability distributions of the plurality of predictive time intervals or the cumulative distribution thereof, the following Such a configuration may be adopted.

That is, in the configuration that outputs the scores corresponding to the plurality of predictive time intervals described above,
An output means for executing a process of displaying a probability distribution of event occurrence or a cumulative distribution thereof using a score output from the predictive detector;
This output means is
(1) When a plurality of predictive time intervals are set without overlapping, the probability distribution of each of the multiple predictive time intervals is simulated using any of a plurality of types of distributions prepared in advance. When creating a distribution in which the local gradient change of the probability distribution curve in a state where the probability distributions of the plurality of predictive time intervals are combined or connected is minimized, or less than or less than a predetermined threshold The process to choose,
(2) In a case where a plurality of predictive time sections are set to overlap, for a section in which a plurality of predictive time sections having different time lengths overlap, the score corresponding to the shortest predictive time section is used to simulate A process that adopts a probability distribution of event occurrence or its cumulative distribution to be created, and does not adopt a probability distribution of event occurrence or its cumulative distribution that is created in a pseudo manner using scores corresponding to other predictive time intervals,
(3) When a plurality of predictive time sections are set in an overlapping manner, for a section in which a plurality of predictive time sections having different time lengths overlap, a score corresponding to each predictive time section is used in a pseudo manner. A process of synthesizing the probability distribution of the event occurrence to be created or its cumulative distribution at a predetermined ratio;
(4) When a plurality of predictive time intervals are set in an overlapping manner, the probability distribution of each of the multiple predictive time intervals or the cumulative distribution thereof is used using any of a plurality of types of distributions prepared in advance. The process of selecting a distribution to be used to create one's own probability distribution or its cumulative distribution using a score corresponding to a predictive time interval with a shorter time length than oneself when creating it in a pseudo manner It is good also as a structure which performs at least 1 process.

  <Invention of program>

  The program of the present invention is for causing a computer to function as the predictive detection system described above.

  The above-mentioned program or a part thereof is, for example, a magneto-optical disk (MO), a compact disk (CD), a digital versatile disk (DVD), a flexible disk (FD), a magnetic tape, or a read-only memory (ROM). Recorded on recording media such as electrically erasable and rewritable read only memory (EEPROM), flash memory, random access memory (RAM), hard disk drive (HDD), solid state drive (SSD), flash disk, etc. For example, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), the Internet, an intranet, an extra Tsu bets like a wired network or a wireless communication network, and further is capable of transmitting using a transmission medium such as a combination thereof, also can be delivered by placing the carrier. Furthermore, the above program may be a part of another program, or may be recorded on a recording medium together with a separate program.

  As described above, according to the present invention, the two-stage processing is performed by the preceding sign feature extractor and the latter sign detector, so that various state fluctuations related to the sign can be accurately detected. In addition, it is possible to detect signs effectively, facilitate data collection for learning, and speed up the system performance improvement associated with it. Therefore, crowd sourcing can be used for data collection for learning, and there is an effect that data collection can be performed inexpensively, quickly and without trouble.

1 is an overall configuration diagram of a sign detection system according to an embodiment of the present invention. Explanatory drawing which shows the detail of the internal structure of the sign feature extractor and the sign detector of the said embodiment. Another explanatory view showing the details of the internal configuration of the sign feature extractor and sign detector of the embodiment. Explanatory drawing which shows the condition at the time of the learning of the precursor detector of the said embodiment, and operation | use. Another explanatory view showing the situation at the time of learning and operation of the sign detector of the embodiment. The figure which shows an example of the crowdsourcing work screen of the said embodiment. The figure of the flowchart which shows the whole flow of the precursor detection process of the said embodiment. Explanatory drawing of the output format which expresses the generation probability of the said embodiment on a time-axis. The detailed explanatory view of the display processing of the probability of occurrence on the calendar of the embodiment. Explanatory drawing of the output format which expresses the accumulation probability of the said embodiment on a time-axis. Explanatory drawing of the output format which expresses the generation probability of the said embodiment by distribution. Explanatory drawing of the display process of the synthetic distribution using the prior information of the embodiment. Explanatory drawing of the output format which expresses the generation | occurrence | production probability of the said embodiment by cumulative distribution. Description of the case where static pattern recognition processing is performed by the predictive detector according to the first modification of the present invention (when a set of feature data of cut sections is input to the static pattern recognizer one by one) Figure. Explanatory drawing of the case where static pattern recognition processing is performed by the sign detector which is the second modification of the present invention (when the similarity between distributions is obtained by capturing a set of feature data in a cut section as a distribution) . Explanatory drawing when performing a static pattern recognition process with the predictor detector which is the 3rd modification of this invention (when the set of the feature data of a cutting area is vectorized, and the similarity between vectors is calculated | required). Explanatory drawing in the case of performing a static pattern recognition process with the sign detector which is the 4th modification of this invention (when forming a segment in the sign time section).

  An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 shows the overall configuration of the sign detection system 10 of the present embodiment. 2 and 3 show details of the internal configuration of the predictive feature extractor 50 and the predictive detector 60. FIG. FIGS. 4 and 5 show the status of learning and operation of the sign detector 60, and FIG. 6 shows an example of the crowdsourcing work screen 200. FIG. 7 is a flowchart showing the overall flow of the sign detection process, and FIGS. 8 to 13 show various output formats of the sign detection result.

  <Overall configuration of the sign detection system 10>

  In FIG. 1, the sign detection system 10 includes a state data capturing device 20 that captures state data indicating a state of a predictive detection target, and a connection that transmits the state data captured by the state data capturing device 20 to the network 1. The apparatus 21 includes a sign detection processing apparatus 30, a data collection apparatus 80, a learning apparatus 90, and a crowd sourcing system 100 that are apparatuses and systems connected to the network 1.

  The network 1 is connected to a user terminal 120 operated by a user (a user of the sign detection result) and an administrator terminal 130 operated by an administrator of the present system. Furthermore, in the crowdsourcing system 100, a large number of worker terminals 110 operated by a large number of workers (crowds) that process requested work (so-called crowdsourcing microtasks) are connected to a network (network 1 or other network). It is supposed to be connected via.

  Here, the network 1 is mainly an external network such as the Internet, but may be a combination of this and an internal network such as an intranet or a LAN. The network 1 may be wired, wireless, or mixed wired / wireless. It doesn't matter if it is. The network 1 may be, for example, an internal network such as an intranet or a LAN limited within a company, a factory, a business office, a group company, a campus, a predetermined area, or the like.

  In crowdsourcing, usually, a requested job is transmitted to the worker terminals 110 of a large number of workers (crowds) connected to a wide network such as the Internet. On the premise that it can be secured, the number of workers may be limited (for example, only to students from several universities) by using an internal network such as an intranet or a LAN. Note that the limitation of the worker is not limited by the network, but may be performed by an access restriction or a work qualification restriction.

  Further, in the present embodiment, as a system configuration for performing various processes related to the sign detection using the state data captured by the state data capturing device 20, the connection device 21 and the sign detection processing apparatus 30 each composed of separate computers. In addition, a configuration in which the data collection device 80, the learning device 90, and the crowd sourcing system 100 are connected by the network 1 is employed, but the system configuration by such network connection is merely an example. Therefore, the sign detection system 10 does not necessarily need to be configured by a plurality of computers as described above, and the functions realized by the devices 21, 30, 80, 90 and the system 100 are integrated into one computer. A configuration may be used, or each function may be distributed to an arbitrary number of computers. For example, each function realized by the sign detection processing device 30 (functions of the status data acquisition means 40, the sign feature extractor 50, the sign detector 60, and the output means 70) may be distributed to a plurality of computers. Alternatively, a part of each function realized by the sign detection processing device 30 (for example, the function of the state data acquisition unit 40 or the function of all or part of the sign feature extractor 50) is connected to the connection device 21. It may be realized with.

  <"Events" that detect signs, "detection targets", and "specific conditions" related to signs>

  An “event” that detects a sign is a characteristic event that has been empirically or statistically known to cause a sign before the event occurs (characteristic is a phenomenon that occurs regularly) The sign can be regarded as a “specific state” for the “sign detection target”.

  The “predictive detection target” may be a living organism (for example, an animal such as a cow, a human being, a plant, a fish, a bird, a microorganism, etc.) or an object (solid, liquid, gas, or a mixture of them, Or a group of objects or organisms that exist within a certain area formed by dividing the space (for example, mountains and parts thereof, ponds, swamps, rivers, or the like) Part of the sea, part of the sea, island and part of it, part of the sky and night sky, part of the earth, etc.). In addition, the thing which can recognize the state by human perception only after performing a visualization process etc. like microorganisms, gas, etc. is contained, for example.

  A “specific state” is a state that is empirically or statistically known to appear as a precursor before the occurrence of the target event. And “a specific state related to a sign and appearing only at the time of the sign (not appearing in normal times)”.

  Specifically, for example, when the sign detection system 10 is a system for detecting a sign of cattle delivery, the “event” is cow delivery, and the “sign detection target” is a cow. The “specific state” includes, for example, a standing state of a cow, a raised state of a cow's tail (a continuous value such as a raising angle of a cow's tail, or a range to which the continuous value belongs is determined by a threshold or a range It is data showing the degree of similarity to the raised state of the cow's tail.), The cow's allantoic / amniotic membrane exposed state, and the cow's standing / sitting state (the state of standing or sitting within a predetermined short time ), A rotating state around one leg of the cow, and a characteristic state of the cow's cry.

  Among the above, the standing state of the cow and the rotating state around the one leg of the cow are the states that can be detected in a short time, and the “specific state” includes such a short time. A state indicating the operation within is included. The short time here is a time that can be captured within the range of the state data (a plurality of pieces of image data and a plurality of frames constituting a moving image) at a plurality of times that are simultaneously input to the predictive feature extractor 50. . Therefore, the time is shorter than the predictive time interval (see FIGS. 4 and 5) set by the predictive detector 60. For example, if the predictive time interval is several hours or several days, The short-time operation is, for example, an operation that can be captured in seconds or minutes. In addition, a characteristic state of the cow cry is a state that can be captured by sound data. In this case, one state data is sound data having a certain length of time. The time length of the sound data is also shorter than the predictive time interval (see FIGS. 4 and 5). Furthermore, a characteristic state of the cow's cry may be captured by inputting a plurality of temporally continuous sound data (a plurality of state data) to the predictive feature extractor 50 at the same time.

  Further, when the sign detection system 10 is a system for detecting a suicide sign of a prisoner in a cell, the “event” is a suicide of a prisoner in the cell, and the “predictor detection target” is set in the cell. The “specific state” is a prisoner, for example, a solitary state of a solitary cell, a state of hitting a head against a wall, or the like.

  Further, when the sign detection system 10 is a system that detects a sign of the onset of depression, the “event” is the onset of depression, the “prediction detection target” is a person or a human face, The “specific state” is, for example, a characteristic state regarding the facial expression of a person such as a state of smile loss.

  When the sign detection system 10 is a system for detecting a sign of an accident or dangerous action, the “event” is an accident or dangerous action, and the “sign detection target” is, for example, an elderly person living alone, needing care For example, an elderly person, a child, a sick person, etc., and the “specific state” is, for example, a state of bending down on the floor, an unnatural sitting position, a state of scolding the floor or stairs, a state of grabbing on an object, or leaning against a wall State, approaching to a dangerous place, etc.

  In addition, when the sign detection system 10 is a system for detecting a sign of a disaster such as a collapse of a tunnel, a flood of a river, an eruption of a volcano, the “event” indicates a collapse of a tunnel, a flood of a river, a volcano Occurrence of disasters such as eruptions, etc., “predictive detection targets” are tunnels, rivers, volcanoes, etc., and “specific states” are, for example, the state where walls and ceilings are peeled off, the state where walls and ceilings are cracked, etc. A state where abnormal noise is generated, a state where the bank is collapsed, a state where gas is blown out, a state where smoke is raised, and the like.

  Furthermore, when the sign detection system 10 is a system for detecting a sign of crime such as theft or the occurrence of terrorism, the “event” is a crime such as theft or the occurrence of terrorism. For example, a person walking around a building at night or a person approaching a trash can, etc., and the `` specific state '' is, for example, a repeated state of the same action, a moving state / intrusion state to a place other than a normal passage place This is the state of putting a suspicious object in the trash bin.

  <Configuration for status data capture, output, and network transmission>

  The state data capturing device 20 is a device that captures and outputs state data indicating the state of the sign detection target (such as a cow), and specifically includes, for example, a camera and a microphone. The state data capturing device 20 is preferably a non-contact type device for a sign detection target (such as a cow) such as a camera or a microphone, but may be a contact type device.

  In addition, the state data capturing device 20 includes a device that outputs perceptual data that can be recognized by human perception as state data. That is, when there is only one type of status data capturing device 20, the status data capturing device 20 is a device (eg, a camera or a microphone) that always outputs perceptual data that can be recognized by human perception. When there are a plurality of types of status data capturing devices 20, at least one of them is a device that outputs perceptual data that can be recognized by human perception. This perceptual data is mainly image data (a still image or a moving image) that can be recognized by human vision or sound data that can be recognized by human hearing, but may also be, for example, odor data. .

  The thermographic data output from the thermographic camera is included in image data that can be recognized by human vision. This is because it is possible to visually recognize which part of the face or body is hot, and therefore, the data is also subject to crowdsourcing. The same applies to processed image data such as an edge image, which is included in image data that can be recognized by human vision and is also subject to crowdsourcing.

  Furthermore, the state data capturing device 20 may include a device that outputs data other than the perceptual data, such as a temperature sensor or an acceleration sensor. Therefore, the predictive feature extractor 50 includes, as state data, data other than the perceptual data (eg, output data from the temperature sensor, acceleration sensor, etc.) together with the perceptual data (eg, output data from the camera, microphone, etc.). You may input simultaneously.

  The connection device 21 is a device having a function as an edge device that transmits data to the network 1 (for example, a function such as conversion of a communication protocol), is configured by a computer, and is output from the status data capturing device 20. The status data is received by wire or wireless, and the received status data is sent to the network 1 together with time information (date / time information of year / month / day / hour / minute / second, or information indicating the data generation order to replace it) and device identification information. The process which transmits to the precursor detection processing apparatus 30 (status data acquisition means 40) via this is performed. The time information may be given by the connection device 21 or may be output from the state data capturing device 20. In addition, when the status data acquisition unit 40 provided in the sign detection processing device 30 is provided in the connection device 21, the status data acquisition unit 40 transmits the status data output from the status data capturing device 20 to the wired or In this case, time information and device identification information are added to the status data.

  There may be one connection device 21 or a plurality of connection devices 21. In the case of a plurality of connection devices 21, the connection devices 21 may be located at remote locations. This is because if the sign detection target is a large target such as a tunnel, a river, a volcano, or the like, state data may be captured at a plurality of locations.

  In addition, one connection device 21 may receive state data captured by one state data capturing device 20, or may receive a plurality of types of state data captured by a plurality of state data capturing devices 20. Also good.

  <Overall configuration of the sign detection processing device 30>

  The sign detection processing device 30 is constituted by one or a plurality of computers (servers), and includes a state data acquisition means 40, a sign feature extractor 50, a sign detector 60, an output means 70, and a prior information storage means. 71. The predictive feature extractor 50 also includes a predictive feature extraction pattern recognition processing means 51 for executing a predictive feature extraction pattern recognition algorithm, and a predictive feature extraction model storage means for storing a predictive feature extraction model used for the pattern recognition processing. 52. Further, the sign detector 60 includes a sign detection processing unit 61 that executes a sign detection process such as a time-series pattern recognition process, and a sign detection model storage unit 62 that stores a sign detection model used for the sign detection process. Yes.

  Here, the state data acquisition means 40, the sign feature extraction pattern recognition processing means 51, the sign detection processing means 61, and the output means 70 are central arithmetic processing provided inside the computer main body constituting the sign detection processing device 30. It is implemented by a device (CPU) and one or a plurality of programs that define the operation procedure of this CPU. Further, as the sign feature extraction model storage unit 52, the sign detection model storage unit 62, and the prior information storage unit 71, for example, a hard disk drive (HDD), a solid state drive (SSD), or the like can be employed.

  The status data acquisition means 40 executes a process of receiving status data (including the added time information and device identification information) transmitted from the connection device 21 via the network 1. The transmission of the state data from the connection device 21 may be performed at a timing managed on the connection device 21 side, or may be performed based on an acquisition request from the state data acquisition unit 40.

  In addition, the state data acquisition unit 40 performs various preprocessing such as background removal, normalization, and edge image creation before inputting the acquired state data to the predictive feature extractor 50, if necessary. Execute.

  In the present embodiment, the state data acquisition unit 40 is configured to receive the state data from the connection device 21, but as a configuration to directly acquire the state data output from the state data capturing device 20. Also good. In some cases, the connection device 21 is provided with the status data acquisition means 40. In addition, without the network configuration, there is a sign near the installation location of the status data capturing device 20 (around the location where the sign detection target such as a cow is detected). This is because the detection process may be configured.

  <Configuration of predictive feature extractor 50>

  The sign feature extractor 50 uses the state data acquired by the state data acquisition means 40, and whether the state of the sign detection target (such as a cow) corresponds to a specific state related to the sign (such as a cow delivery sign) The pattern recognition process which outputs the data which shows the degree of similarity with respect to the said specific state is performed, and the process which extracts the feature data obtained by this pattern recognition process is repeatedly performed. When there are a plurality of specific states, a pattern recognition process is executed to output data indicating whether each of the specific states corresponds to each of the plurality of specific states or the degree of similarity to each of the plurality of specific states. The extracted time-series feature data is temporarily stored in feature data storage means (not shown).

  At this time, the number of pattern recognition processes may be one or plural, and in the case of plural patterns, different types of pattern recognition processes may be used. Also, the number of pattern recognition processes need not match the number of specific states.

  The “specific state” related to the sign is a predetermined state and is a state set from the learning stage of the sign feature extractor 50. When detecting a calving parturition of a cow, as described above, the standing state of the cow, the raising state of the tail of the cow, the exposed state of the allantoic membrane / amniotic membrane of the cow, and the like.

  Specifically, in this embodiment, the predictive feature extractor 50 is configured to execute pattern recognition processing using a convolution / neural network (CNN) as an example. However, a Hidden Markov Model (HMM) or the like may be adopted, or a plurality of types of pattern recognition processes may be used in combination, such as a configuration in which the CNN output and the HMM output are combined and input to the precursor detector 60 at the subsequent stage. May be.

  As shown in FIGS. 2 and 3, the CNN includes an input layer, a plurality of intermediate layers, and an output layer (final layer). The predictive feature extractor 50 includes a CNN intermediate layer (for example, two layers before the final layer). The processing is performed to extract the feature amount output from the intermediate layer or the like) as the feature data to be input to the sign detector 60 in the subsequent stage. Although the number of specific states is arbitrary, it is assumed here that there are three specific states X, Y, and Z for convenience of explanation.

  In FIG. 2, the predictive feature extractor 50 outputs the discrimination results of each of a plurality of specific states X, Y, and Z by one pattern recognition process (CNN). That is, the output layer (final layer) of the CNN has two output nodes (2 units) corresponding to a specific state X (for example, the standing state of the cow), and the likelihood that the state X is likely to be satisfied. Degree and likelihood indicating the likelihood of not being in state X is output. However, since the intermediate layer output is extracted as the feature data, these likelihoods are not used in the sign detector 60 in the subsequent stage. Note that these likelihoods are used in the determination process for determining whether or not to use the data for subsequent learning.

  Similarly, in the output layer (final layer) of CNN, there are two output nodes (2 units) corresponding to a specific state Y (for example, the cow's tail raised state). Likelihood indicating likelihood and likelihood indicating likelihood of not being in state Y are output. In addition, the output layer (final layer) of the CNN has two output nodes (2 units) corresponding to a specific state Z (for example, an exposed state of bovine allantoic membrane / amniotic membrane). A likelihood indicating the likelihood and a likelihood indicating the probability that the state is not the state Z are output.

  Therefore, in FIG. 2, the network part from the input layer to the intermediate layer from which feature data is extracted is shared by three specific states X, Y, and Z, and the parameters of that part are shared. become.

  On the other hand, in FIG. 3, the predictive feature extractor 50 executes different pattern recognition processing (CNN) for each of the plurality of specific states X, Y, and Z, and performs a plurality of specific states X, Y, and Z. Each discrimination result is output. At this time, the state data input to each pattern recognizer (CNN) is common, and a plurality of feature amounts output from the intermediate layer of each pattern recognizer (CNN) are collected and used as a sign of the subsequent stage as feature data. Input to the detector 60.

  Moreover, in FIG. 3, although it is set as the structure which extracts an intermediate | middle layer output about all of the several specific states X, Y, and Z, it is not necessary to extract an intermediate | middle layer output about all, About a part, The output layer (final layer) output may be extracted. Further, since each pattern recognizer corresponding to a specific state X, Y, Z is a separate pattern recognizer, the parameters of the network part leading to the intermediate layer from which the feature data is extracted are different from each other. The number of intermediate layers constituting the pattern recognizer may be different. Furthermore, the position of the intermediate layer from which the feature data is extracted (which number of layers is to be extracted) may be different for each pattern recognizer.

  Thus, the predictive feature extractor 50 may have the configuration shown in FIG. 2 or the configuration shown in FIG. 3, but the configuration shown in FIG. 2 is preferable from the viewpoint of simplifying the system configuration and reducing the processing amount.

  The state data simultaneously input to the predictive feature extractor 50 may be only state data at one time (for example, image data of a still image at a certain moment), or state data at a plurality of times (for example, a plurality of frames constituting a moving image). Image data). In particular, when capturing a state (such as a cow's standing or rotating state around a cow's leg) that shows a motion that can be detected in a short time as a “specific state” related to a sign. Enter time status data. Further, when inputting state data at a plurality of times, assuming that the state data is D1, D2, D3, D4, D5,... In the order of occurrence, first, a set of D1, D2, and D3 is input. Next, it is preferable to input the set of D2, D3, and D4, and then input the set of D3, D4, and D5 while shifting the state data one by one. If D1, D2, and D3 are input, then D4, D5, and D6 are input, then D7, D8, and D9 are input without overlapping, Well, or as a compromise between them, (D1, D2, D3, D4), (D3, D4, D5, D6), (D5, D6, D7, D8), etc. Yes, but it ’s also an input that you do n’t want to shift .

  The sign feature extraction model (CNN parameters, etc.) to be stored in the sign feature extraction model storage means 52 is prepared by prior learning. This prior learning method is the same as the learning method when updating the model by the sign feature extractor learning means 91 described later, and may be a general learning method. That is, it is performed using a certain number of state data and tag information corresponding to each state data. Specifically, with respect to one state data (for example, image data), whether or not it corresponds to a specific state X (for example, a cow standing state), a specific state Y (for example, raising the tail of a cow) Data (1 or 0) indicating whether or not it corresponds to a specific state Z (for example, an exposed state of bovine allantoic membrane or amniotic membrane) as tag information, and learning I do.

  <Configuration of the sign detector 60>

  In this embodiment, the sign detector 60 executes time-series pattern recognition processing using the time-series feature data extracted by the sign feature extractor 50, and outputs a score indicating the presence or absence of the sign or the presence or absence of the sign. The process is executed.

  At this time, for example, a recurrent neural network (RNN) can be adopted as the time-series pattern recognition process, and a long / short term memory (LSTM) is particularly suitable. However, the time-series pattern recognition process is not limited to RNN, and for example, a hidden Markov model (HMM) may be employed.

  As shown in FIGS. 4 and 5, the sign detector 60 divides a time zone before the occurrence of an event to set a plurality of sign time sections as sections where a sign may appear, The score corresponding to each of the predictive time intervals is output. The event occurrence time point is T0, and the previous (past) time points are (T-1), (T-2), (T-3), (T-4),. Assuming that (TN), there are two types of formats A (in the case of FIG. 4) and B (in the case of FIG. 5) as the formats for setting a plurality of predictive time intervals and outputting scores. The intervals between the time zones may be equal time intervals with the same pitch, or may be non-uniform time intervals by changing the pitch. For example, the time lengths of (T-4) to (T-3), The time length of (T-3) to (T-2) may be the same or different.

  The format A in FIG. 4 is a setting format in which a plurality of predictive time intervals A1, A2, A3, A4,. The period before (T−N) is the non-target section, which is the predictive time section AN = (TN) to (T− (N−1)), and the predictive time section A4 = (T−4) to (T−). 3), the predictor interval A3 = (T-3) to (T-2), the predictor interval A2 = (T-2) to (T-1), and the predictor interval A1 = (T -1) to T0.

  Note that (T-1), (T-2), (T-3), (T-4),..., (TN) are points in time (past) before the event occurrence point T0. , By inverting these on the time axis (rearranging each time point arranged in the past direction toward the future direction while keeping the time interval between the respective time points), the current (score output is performed) Time points) T1, T2, T3, T4,..., TN, which are time points from Tnow toward the future, are defined (see FIG. 8). For example, if (T-1) is 12 hours before the event occurrence time T0, T1 is 12 hours after the current Tnow, and (T-2) is 24 hours before T0. If it is a time point, T2 will be a time point 24 hours after Tnow.

  In the format A, a score indicating the probability that the current Tnow is at the time point of the predictive time interval A1 = (T−1) to T0 is output (this is the time when the score is output and when viewed in units of days). . In other words, a score indicating the probability of occurrence of an event is output within the current Tnow to T1 period (the length of the period is the same as the predictive time section A1).

  Similarly, in the format A, a score indicating the probability that the current Tnow belongs to the predictive time section A2 = (T-2) to (T-1) is output. In other words, a score indicating the probability that an event will occur is output within the period from T1 to T2 (the length of the period is the same as the predictive time section A2) as viewed from the current Tnow.

  In the format A, a score indicating the probability that the current Tnow belongs to the predictive time section A3 = (T-3) to (T-2) is output. In other words, as viewed from the current Tnow, a score indicating the probability of occurrence of an event is output within the period from T2 to T3 (the length of the period is the same as the predictive time section A3).

  Further, in the format A, a score indicating the probability that the current Tnow is a time point belonging to the predictive time section A4 = (T-4) to (T-3) is output. In other words, a score indicating the probability that an event will occur is output within the period from T3 to T4 (the length of the period is the same as the predictive time interval A4) as viewed from the current Tnow.

  In the format A, each score indicating the probability that the current Tnow belongs to each of the predictive time sections A1 to AN and the score indicating the probability that the current Tnow belongs to the non-target section (normal time) are summed. Then it becomes 1.

  At the time of learning in the format A, the predictor detector learning means 92 is input with “predictive time interval A1 = (T−1) to T0” = 1 for feature data belonging to the predictive time interval A1, and otherwise. 0 is entered otherwise. For the feature data belonging to the predictive time interval A2, “predictive time interval A2 = (T−2) to (T−1)” = 1 is input, and it does not belong to other intervals. Enter 0. For the feature data belonging to the predictive time section A3, enter “predictive time section A3 = (T-3) to (T-2)” = 1, and it does not belong to any other section. Enter 0. For the feature data belonging to the predictive time section A4, “predictive time section A4 = (T-4) to (T-3)” = 1 is input and it does not belong to any other section. Enter 0. For feature data belonging to the predictive time section AN, “predictive time section AN = (TN) to (T− (N−1))” = 1 is input, and it does not belong to other sections. Otherwise, enter 0. For feature data belonging to a non-target section, “Non-target section = (before T−N)” = 1 is input, and since it does not belong to any other section, 0 is input otherwise.

  On the other hand, the format B in FIG. 5 is a setting format in which a plurality of predictive time intervals B1, B2, B3, B4,. The period before (T−N) is the non-target section, which is the predictor section BN = (TN) to T0, the predictor section B4 = (T−4) to T0, and the predictor section B3 = (T −3) to T0, the signing time interval B2 = (T−2) to T0, and the signing time interval B1 = (T−1) to T0.

  In the format B, a score indicating the probability that the current Tnow is a time point belonging to the predictive time interval B1 = (T−1) to T0 is output. In other words, a score indicating the probability that an event will occur is output within the current Tnow-T1 period (the length of the period is the same as the predictive time section B1). A score indicating the probability that the current Tnow does not belong to the predictive time interval B1 = (T−1) to T0 is also output, but the illustration is omitted.

  Similarly, in the format B, a score indicating the probability that the current Tnow is a time point belonging to the predictive time interval B2 = (T−2) to T0 is output. In other words, a score indicating the probability that an event will occur is output within the current Tnow to T2 period (the length of the period is the same as the predictive time section B2). A score indicating the probability that the current Tnow does not belong to the sign time interval B2 = (T−2) to T0 is also output, but is not shown.

  Also, in the format B, a score indicating the probability that the current Tnow is a time point belonging to the predictive time interval B3 = (T−3) to T0 is output. In other words, a score indicating the probability of occurrence of an event is output within the current Tnow to T3 period (the length of the period is the same as that of the predictive time section B3). A score indicating the probability that the current Tnow does not belong to the predictive time interval B3 = (T−3) to T0 is also output, but is not shown.

  Further, in the format B, a score indicating the probability that the current Tnow is a time point belonging to the predictive time interval B4 = (T−4) to T0 is output. In other words, a score indicating the probability of occurrence of an event is output within the current period Tnow to T4 (the length of the period is the same as that of the predictive time section B4). A score indicating the probability that the current Tnow does not belong to the predictive time interval B4 = (T−4) to T0 is also output, but is not shown.

  At the time of learning of the format B, since the characteristic data belonging to “(T−1) to T0” belongs to the sign time interval B1, the sign detector learning means 92 is assigned to “the sign time interval B1 = (T−1)”. ˜T0 ”= 1, and also belongs to the predictive time interval B2, so that“ predictive time interval B2 = (T−2) to T0 ”= 1 is input and the predictive time interval B3. Therefore, “predictive time interval B3 = (T−3) to T0” = 1 is input, and since it also belongs to the predictive time interval B4, “predictive time interval B4 = (T−4) − Enter T0 ”= 1.

  Further, during the learning of the format B, since the feature data belonging to “(T-2) to (T-1)” does not belong to the predictive time section B1, the sign detector learning means 92 is notified as “predictive time”. Section B1 = (T-1) to T0 ”is input as“ = 0 ”and belongs to the predictive time section B2, so“ predictive time section B2 = (T−2) to T0 ”is input as“ 1 ”. Since it also belongs to the sign time interval B3, it is input that “the sign time interval B3 = (T-3) to T0” = 1 and also belongs to the sign time interval B4. “B4 = (T−4) to T0” ”= 1.

  Further, during the learning of the format B, since the feature data belonging to “(T-3) to (T-2)” does not belong to the predictive time section B1, the sign detector learning means 92 is notified as “predictive time”. Section B1 = (T-1) to T0 "= 0 is input, and since it does not belong to the predictive time section B2," predictive time section B2 = (T-2) to T0 "= 0. Since it is input and belongs to the sign time section B3, “predict time section B3 = (T−3) to T0” = 1 is input and also belongs to the sign time section B4. Section B4 = (T−4) to T0 ”= 1.

  Furthermore, during the learning of the format B, the characteristic data belonging to “(T-4) to (T-3)” does not belong to the predictive time section B1 to the predictive detector learning unit 92. Section B1 = (T-1) to T0 "= 0 is input, and since it does not belong to the predictive time section B2," predictive time section B2 = (T-2) to T0 "= 0. Since it is input and does not belong to the predictive time interval B3, “predictive time interval B3 = (T−3) to T0” = 0 is input and belongs to the predictive time interval B4. “Time interval B4 = (T−4) to T0” ”= 1.

  When learning in the format B, since the feature data belonging to the non-target section is not included in the predictive time section B1, the sign detector learning means 92 does not include the “predictive time section B1 = (T−1) to T0. "= 0" and does not belong to the predictive time interval B2, so enter "predictive time interval B2 = (T-2) to T0" = 0 and enter the predictive time interval B3. Therefore, “predictive time interval B3 = (T−3) to T0” = 0 is input, and since it does not belong to the predictive time interval B4, “predictive time interval B4 = (T−4)”. ~ T0 "= 0.

  Since the recurrent neural network (RNN) has a structure in which the output of a certain layer is input to the next layer and reflected in the next processing, the output reflecting the previous processing result can be performed. Since the previous processing result further reflects the previous processing result, a time-series pattern recognition process that reflects the previous processing result is realized.

  The sign detector 60 can be configured to reflect the processing results before the previous time by such an internal structure of the RNN. However, the predictor detector 60 may be configured as follows and consider temporal factors. That is, a plurality of time-series feature data with different extraction times extracted repeatedly by the preceding sign feature extractor 50 (multiple time feature data obtained by a plurality of successive pattern recognition processes in the sign feature extractor 50). These may be input to the precursor detector 60 at the same time. When inputting feature data at a plurality of times, assuming that the feature data is F1, F2, F3, F4, F5,... In the order of occurrence, first, a set of F1, F2, F3 is input, and then , F2, F3, and F4 are input, followed by F3, F4, and F5. For example, the feature data is preferably input while being shifted one by one. F1, F2, and F3 may be input, then F4, F5, and F6 may be input, and then F7, F8, and F9 may be input. Or, as a compromise between them, there is an overlap such as (F1, F2, F3, F4), (F3, F4, F5, F6), (F5, F6, F7, F8). The input may not be shifted one by one.

  The sign detector 60 may be input with the advance information stored in the advance information storage unit 71 or information automatically generated from the advance information together with the feature data. The prior information is, for example, user possessed information input from the user terminal 120 by the user or from the administrator terminal 130 by the administrator of the system, or an administrator input from the administrator terminal 130 by the administrator For example, in the case of detecting a calving sign of a cow, it is a scheduled date of delivery of a cow held by a user (farmer or the like). The information automatically generated from the prior information is the number of days until the expected date of calf birth calculated as the difference between today (the date of predictive detection processing) and the expected date of calf delivery.

  <Configuration of output means 70>

  The output means 70 executes processing for outputting a sign detection result using the score output from the sign detector 60. Specifically, the output unit 70 transmits the data for outputting the sign detection result to the user terminal 120 (or a cooperative system managed by the user) via the network 1, or as necessary. A process of transmitting to the administrator terminal 130 is executed. In addition, the output means 70 displays the sign detection result as the sign detection processing apparatus 30 when the user or the administrator is in the vicinity of the place where the sign detection processing apparatus 30 (the apparatus provided with the output means 70) is installed. Screen display on a display device (not shown) connected to the printer, printing with a printing device (not shown), outputting sound with a speaker (not shown), or a notification sound or notification light according to the sign detection result May be executed by a notification device (not shown) such as a buzzer or a lamp.

  When performing processing for transmitting data for outputting the sign detection result to the user terminal 120 or the like, the data for outputting the sign detection result to be transmitted is mainly data for screen display, but data for voice output is added. Alternatively, only data for audio output may be used. The transmission timing may be the timing on the output means 70 side, or may be transmitted in response to a request from the user terminal 120 or the like.

  Specifically, for example, the output data of the sign detection result may be automatically transmitted by e-mail or the like, and access to the sign detection processing device 30 is urged by automatic transmission of e-mail or automatic call (for example, The output data of the sign detection result may be transmitted after receiving an output request (mainly a screen display request) from the user terminal 120 or the like. In the case of transmission after receiving an output request from the user terminal 120 or the like, output data (output data stored in output data storage means (not shown)) that has already been generated is output as a sign detection result. ) Or after receiving the output request, the output data of the sign detection result is generated using the output score of the sign detector 60 stored in the score storage means (not shown), You may send it.

  The output means 70 outputs the state data (including the added time information and device identification information) input to the preceding sign feature extractor 50, and the final layer (output layer) of the sign feature extractor 50. A process of transmitting the data (likelihood), the feature data input to the sign detector 60 at the subsequent stage, and the score output from the sign detector 60 to the data collection device 80 via the network 1 is also executed.

  <Various output formats by the output means 70>

  The output means 70 uses the score output from the sign detector 60 to create output data for various sign detection results. All output formats (mainly display formats) described below can be freely switched by a user or an administrator. At this time, any output format may be set as a default setting on the system side, or a user or an administrator may set a standard output format by himself / herself. The output means 70 may be configured to output only one or some of the various output formats described below.

  <Output format by language>

  First, there is an output format by language. The sign detection result is expressed in text and notified by screen display, or it is read out by voice and notified, or both of them are executed.

  The notification content includes the detection target (biology, object, etc.) for which a predictor has been detected, the identification information of the detection target when there are multiple predictive detection targets, the details of the predicted event, and the occurrence of the predicted event The timing, the occurrence probability of a predicted event, and the like are included, and among these contents, the contents required by the user may be output (screen display and / or voice output).

  For example, “Probability of cow 1032 to deliver within 12 hours 80%”, “Cow A or cow B is likely to deliver 2-4 days later”, “Prisoner X may become depressed if left as it is. Etc. In addition, when there are multiple predictive detection targets, such as expressing that “the cow is likely to deliver within 24 hours” when there are multiple cows, it may be expressed without adding the predictive detection target identification information. Often, in this case, which cow is specified is included, a case where the cow is not notified and a case where the cow is not specified are included.

  In the case where the score output of the predictor detector 60 is format A (see FIG. 4), as an example, the predictor time intervals A1 to A4 are set, and (T-1) = 12 hours before (T1 = 12 hours after), ( T-2) = 24 hours before (T2 = 24 hours later), (T-3) = 2 days before = 48 hours before (T3 = 2 days after = 48 hours), (T-4) = 4 days before = 96 hours It is assumed that it is before (T4 = 4 days later = 96 hours later). At this time, using the scores of the predictive time intervals A1 to A4 as they are, “the probability that a cow will deliver within 12 hours from now is 10%, the probability between 12 hours and 24 hours later is 20%, 24 Probability between time and after 2 days is 50%, and probability between 2 days and 4 days is 10% ", etc. Good.

  For example, add the scores of the predictive interval A1 and A2 (10% of A1 + 20% of A2 = 30%), and output such as "the probability that a cow will deliver within 30 hours is 30% ..." May be. In this case, for example, as a system (that is, common to all users), the predictive time intervals A1 to AN are set at relatively short time intervals, and each user is notified at the notification stage to each user. Can customize the desired time interval and output the sign detection result. This customization setting may be already set in the system at the time of delivery of the system, or may be set by the user after the system starts operating. In the latter case, the user can change the setting at any time interval at any time, and the score output of the sign detector 60 is common to all users, and at the stage of output processing by the output means 70, the user Processing (addition processing) according to the setting is performed.

  If anything, the score output of the sign detector 60 is in the format B (see FIG. 5), which is more suitable for the output of the cumulative probability, but in the case of the format A (see FIG. 4). However, the cumulative probability may be output. For example, the scores of the predictive time intervals A1 and A2 are added (10% of A1 + 20% of A2 = 30%), and the scores of the predictive time intervals A1 to A3 are added (10% of A1 + 20% of A2 + A3 And the scores of the predictive interval A1 to A4 are added (10% of A1 + 20% of A2 + 50% of A3 + 10% of A4 = 90%), “within 12 hours from now The probability that a cow will deliver is 10%, the probability within 24 hours is 30%, the probability within 2 days is 80%, and the probability within 4 days is 90%.

  When the score output of the sign detector 60 is in the format B (see FIG. 5), if the scores of the sign time sections B1 to B4 are output as they are, a cumulative probability is output. For example, “The probability that a cow will deliver within 12 hours from now is 10%, the probability within 24 hours is 30%, the probability within 2 days is 80%, and the probability within 4 days is 90%.” Can do.

  Here, as described above, if the cumulative probability is output using the score in the case of the format A (see FIG. 4), the probabilities of the predictive time intervals A1, A2,. This is not an inconsistent output expression. This is because the cumulative value does not decrease, and since each score is output so that the total value becomes 1 from the beginning, it does not exceed 1 because it is added. . On the other hand, in the case of the format B (see FIG. 5), if the score is output as it is, for example, “... the probability within 2 days is 80%, the probability within 2 days is 78%”, etc. Since such contradictory scores may be output, for example, when such contradictory scores are output, the output means 70 takes an average of the contradictory scores, for example, “... the probability within 2 days is 79%, Processing for removing contradiction may be executed such that the probability within 4 days is 79%.

  <Output format for visualizing the probability of occurrence>

  Next, there is an output format for visualizing the probability of occurrence. In addition to the contents of <Output format by language> described above, it is a format that adds an expression that visualizes the probability of occurrence. In addition to the language expression (screen display and / or voice transmission), numerical information about the probability of occurrence The screen image will also be transmitted. Note that the amount of information that can be transmitted is small, but only the screen image may be used without expressing the language.

  For example, there is a display with an icon such as a face mark. Specifically, for example, “confident face” is displayed for a high occurrence probability, “normal face” is displayed for an intermediate occurrence probability, and “suspicious face” is displayed for a low occurrence probability. . Moreover, if it shows with the magnitude | size of an animal, it will be an elephant, a lion, a cat, a grasshopper, an ant etc. in order with high occurrence probability, for example. In terms of magnitude of value, gold medals, silver medals, bronze medals, etc., in descending order of occurrence probability.

  The occurrence probability may be displayed with a gauge. Specifically, for example, if the probability of the predictive time interval A3 is 50%, “… 50%: (0%) □□□□□ _____ (100%)” or the like is output, or “… 50% : (0%) __________ (100%) "or the like. The latter is similar to displaying the probability of occurrence at the position of the needle on the rotation meter.

  Furthermore, the occurrence probability may be displayed by a change in color (hue / lightness / saturation). Specifically, for example, using the color of the traffic light, if the low probability of occurrence is “green” and the high probability of occurrence is “red” and they are arranged in the order from 0% to 100%, “green”, “yellow” It can be displayed as “green”, “yellow”, “orange”, “red”, and the like. In this way, it is not always necessary to continuously change the hue, lightness, and saturation in a strict sense corresponding to the magnitude of the occurrence probability, but in a state close to the display by the icon such as the face mark described above, Colors that match human images may be arranged step by step. In a strict sense, the hue, brightness, and saturation may be derived from the numerical value of the occurrence probability using a predetermined function.

  Such <output format for visualizing occurrence probability> may be applied to the cumulative probability. That is, for example, numerical values to be visualized by icons such as face marks, gauges, rotation meters, colors, etc. are not limited to occurrence probabilities (see FIG. 4) in each section set so as not to overlap. It may be a cumulative probability such as the occurrence probability (see FIG. 5) in each section set in the above. At this time, as described in the above <Output format by language>, the numerical value of the cumulative probability is the predictor time interval A1, A2,... When the score output of the predictor detector 60 is the format A (see FIG. 4). Or an output score in the case of the format B (see FIG. 5) may be used as it is, or a numerical value obtained by executing a contradiction elimination processing.

  <Output format that expresses the probability of occurrence on the time axis>

  On the time axis, the numerical value indicating the probability of occurrence (a number as a text) is displayed, or on the time axis, the above-mentioned <output format for visualizing the probability of occurrence> is displayed, or time There is an output format for displaying the magnitude of occurrence probability on the axis by the length of the line segment in the direction intersecting the time axis (mainly the height of the line segment in the direction orthogonal to the time axis). The time axis may be a calendar. The occurrence probability expressed on the time axis may be a cumulative probability (see FIG. 10).

  For example, as shown in FIG. 8, the time until the event occurrence starting from the current time (the time of score output, or the current time, or the current time when capturing by date) is one-dimensional (on the time axis). The length of the line segment) is displayed, and on the time axis, the numerical value of the probability of occurrence (number as text) itself is displayed as in (1), or a face corresponding to the size of the numerical value An icon such as a mark may be displayed, a color corresponding to the size of a numerical value may be displayed as in (2), or (1) and (2) may be combined as in (3). Also, it is displayed with the height of the bar graph orthogonal to the time axis as in (4), or is expressed as the height in the direction orthogonal to the time axis over the entire area as in (5). May be. The width in the case of (5) indicates the section length for each section, but the area has no meaning, and therefore assumes a uniform distribution as in (1) in FIG. 11 described later. is not. Furthermore, as shown in (6), the time axis is fitted to the calendar, and the occurrence probability value (number as text) is displayed on the calendar itself, or a face mark or the like corresponding to the size of the value is displayed. An icon may be displayed, or a color corresponding to the size of the numerical value may be displayed.

  In the calendar display of (6) above, since the time lengths of the predictive time sections A3 and A4 are different from those of the calendar unit section, the probabilities of the predictive time sections A3 and A4 are regarded as a uniform distribution, and the calendar at the same time Allocation is performed according to the length of the unit section (in this example, it is equally divided into two and four), and the calendar is assigned to the unit section. Details of this allocation processing will be described later with reference to FIG.

  In the calendar display of (6) above, the sign detection result may be displayed together with general schedule information (such as a meeting) as shown in the figure. At this time, when displaying a color according to the numerical value of the probability of occurrence on the calendar, the general schedule information is superimposed on the general schedule information (overlapping the colored section and the general schedule information is displayed). Display). In addition, when displaying the sign detection result on the calendar, the present (score output time) that is the starting point for displaying the sign detection result will move toward the future on the calendar. Accordingly, the display portion of the sign detection result also moves in the future direction. Therefore, although the past time is displayed on the calendar, the display portion of the sign detection result is not already displayed in the section (the section before the present). Thus, only general schedule information remains.

  As shown in FIG. 9, when the event occurrence probability is displayed on the calendar, as described above, the current starting point for displaying the sign detection result (score output time) is the future direction on the calendar. Since it moves toward, usually, the calendar unit section (the section of the minimum unit time length set for schedule description) and the predictive time section are shifted. When setting the predictive time interval, even if all the time lengths of the multiple predictive time intervals are made to match the time length of the calendar unit interval, the current (score output time) moves, so both intervals Are in a state of matching (a state in which the separation positions at both ends of both sections match). If the two sections are always matched, the score output time must be matched with the break position of the calendar unit section. If the length of the calendar unit section is one day, for example, one day The sign detection (score output) is performed only once.

  As shown in FIG. 9, the output means 70 executes a process of displaying the event occurrence probability on the calendar using each score output from the sign detector 60. At this time, each score (each score) is displayed. ) Is assigned to the calendar unit interval. This allocation process includes at least one predictive time interval (considering that the length of each predictive time interval may be uneven) among a plurality of predictive time intervals, and a calendar unit interval. This is executed when the time lengths do not coincide with each other or both sections are shifted in time. In the example of FIG. 9, the time length differs between the predictive time section and the calendar unit section, and the separation positions at both ends of the section are shifted in time. The time lag is inevitably generated because each predictive time section moves with respect to the unit section of the calendar that is not moving. The score (probability 14%) corresponding to the predictive time interval A6 is regarded as a uniform distribution within A6, and is prorated according to the time length of (T5 to C1) and (C1 to T6), and (T5 to C1) 11% is assigned to the unit section on April 10 of the calendar, and 3% of (C1 to T6) is assigned to the unit section on April 11 of the calendar. In addition, since the predictive time interval A7 is entirely within the calendar April 11 unit interval, the entire A7 score (probability 20%) is assigned to the calendar April 11 unit interval. Assigned. Further, the score (probability 15%) corresponding to the predictive time interval A8 is regarded as a uniform distribution within A8, and is distributed according to the time lengths of (T7 to C2) and (C2 to T8), and (T7 to C2 5% is assigned to the unit section of the calendar on April 11, and 10% of (C2 to T8) is assigned to the unit section of the calendar on April 12.

  As shown in FIG. 10, the occurrence probability expressed on the time axis may be the cumulative probability. That is, in (4) in FIG. 8 described above, the occurrence probability is indicated by the height of the bar graph orthogonal to the time axis. However, as shown in FIG. 10, the cumulative probability is indicated by the height of the bar graph, for example. Also good. At this time, the cumulative probability to be displayed is obtained by adding the score corresponding to each of the predictive time intervals A1, A2,... When the score output is format A (see FIG. 4). If the score output is in the format B (see FIG. 5), the scores corresponding to the predictive time intervals B1, B2,... May be used as they are, or the scores obtained by performing the contradiction elimination processing May be used. When such cumulative probability is displayed, a section having a large slope of the line connecting the vertices of each bar graph is a section having a high occurrence probability. In the example of FIG. Probability (50%) is the highest. For convenience of explanation, numerical values are made to correspond in the format A (see FIG. 4) and the format B (see FIG. 5), but even if the sign detection is performed using the same state data, it is like this. The numerical values rarely correspond.

  <Output format for expressing the probability of occurrence as a distribution>

  In the above-described <output format for expressing the occurrence probability on the time axis>, the occurrence probability is expressed as data (discrete distribution data) distributed at each point on the time axis. On the other hand, there is an output format in which the occurrence probability is expressed by a distribution using a probability density function. Note that (5) in FIG. 8 described above looks like a distribution by a probability density function, but it has already been explained that this is not the case. The probability density function may be a cumulative probability density function (see FIG. 13).

  In the distribution expression by this probability density function, since the occurrence probability is captured by area, when comparing the occurrence probability of each predictive section, the relative size is recognized by considering the area. Therefore, this point is when the relative size is recognized in the above-mentioned <output format that expresses the probability of occurrence on the time axis>, for example, by considering the height (line length) of the bar graph. Is different. However, at which position on the time axis the occurrence probability of the event is high (for example, when cows are likely to be delivered) in the case of <output format that expresses the occurrence probability on the time axis> described above. Similarly, it can be recognized by the height of the mountain indicated by the distribution by the probability density function.

  In FIG. 11, the distribution of (1) assumes that each of the predictive time sections A1, A2,... Is a uniform distribution, and scores corresponding to the predictive time sections A1, A2,. Is indicated by the area. In other words, if the area is calculated by integrating the distribution based on the probability density function thus displayed, the probability is obtained. Therefore, although it looks similar to (5) in FIG. 8, it is an output format of a different expression.

  Further, the distribution (2) in FIG. 11 is assumed to be a non-uniform distribution (for example, a normal distribution). When assuming a normal distribution, for example, for the section from T1 (after 12 hours) to T2 (after 24 hours) (predictive time section A2), the central time point between T1 and T2 (after 18 hours) is the apex. And T1 (after 12 hours) to T2 (after 24 hours) are assumed to be a normal distribution with a σ interval (σ is a standard deviation), and the other intervals are assumed to have a similar normal distribution. Are added together to synthesize the pseudo distribution of the overall distribution based on the probability density function. At this time, the normal distribution of each predictor time interval A1, A2,... Is such that the ratio of the areas thereof is the ratio of the scores (probabilities) of each predictor time interval A1, A2. In addition, although the distribution outside the σ interval is also added and combined here, the distribution outside the σ interval is not adopted, and distributions only in the predictive time intervals A1, A2,. Good.

  In addition, the output unit 70 prepares a plurality of types of distributions in advance in a case where the overall distribution based on the probability density function is created in a pseudo manner assuming the distribution of each section in this way. A process of selecting a distribution in which the local gradient change of the probability distribution curve in a state where the probability distributions of the time intervals are combined or connected is minimized or less than or less than a predetermined threshold is executed. May be.

  Here, the “local gradient change” is a change in gradient within a predetermined short time (may be unit time), and an extreme uneven portion (pointed portion) occurs in the probability distribution curve. Since the gradient change becomes large, the output means 70 selects a distribution that has no or very few extreme irregularities. The distribution to be selected is, for example, a normal distribution, a Pareto distribution, or the like, and is a distribution prepared in advance. Here, since the distribution in which the 1σ section of the normal distribution is applied to each predictor time section and the distribution in which the 2σ section of the normal distribution is applied to each predictor time section are regarded as different distributions, for example, 0.8σ, Distributions such as 1σ, 1.5σ, 2σ,... Are prepared.

  Furthermore, the output unit 70 also executes an output process of the sign detection result reflecting the prior information (for example, the expected date of cattle birth) predicted or scheduled in advance for the event occurrence time. This advance information is stored in advance information storage means 71 (see FIG. 1).

  In other words, the output unit 70 generates event occurrences created using the probability distribution of event occurrences created using the prior information stored in the prior information storage unit 71 and the scores output from the sign detector 60. A process of displaying a combined distribution obtained by combining the probability distribution is executed.

  Further, the output means 70 uses the advance information stored in the advance information storage means 71 (including information automatically generated from the advance information) together with the feature data from the predictive feature extractor 50 as the sign detector 60. May be executed to display a composite distribution created using each score output from the sign detector 60. Here, the information automatically generated from the advance information is, for example, the number of days until the expected delivery date of the cow when the expected delivery date of the cow is advance information. Since the score output is described in detail in <Configuration of the sign detector 60>, description thereof is omitted here.

  In the former case, the probability distribution based on the prior information and the probability distribution based on the output score from the sign detector 60 are created separately and synthesized by the output means 70, whereas in the latter case, the output means 70 is the same as the output process when the prior information is not reflected from the viewpoint of performing the output process in various output formats using the output score from the sign detector 60. Therefore, the former case will be described below.

  As shown in FIG. 12, the probability distribution based on the prior information does not move, whereas the probability distribution based on the output score from the predictive detector 60 moves in the future direction with the passage of time. At the time Tnow (1) far from the event occurrence time (for example, the expected date of birth of the cow), the output score (occurrence probability) is zero or substantially zero. However, as the event occurrence time approaches, the numerical value of each score increases, and a probability distribution based on each score is created. When the current position (at the time of score output) moves from Tnow (2) to Tnow (3), the state data changes, so the shape of the probability distribution based on each score changes. Accordingly, the shape of the combined distribution of the probability distribution based on the prior information and the probability distribution based on each score also changes.

  The probability distribution based on prior information is assumed to be a normal distribution, for example. An appropriate distribution is selected from information obtained from experience or statistics, and prepared in advance. The probability distribution based on this prior information can be easily combined with the probability distribution based on each score if the data is previously discretized according to the calendar unit interval, and the combined distribution can be displayed on the calendar. Become. For example, in the example of FIG. 8 (6) described above, the probability distribution based on the prior information is discretized so as to fit a calendar having a unit interval of 12 hours (morning and afternoon of each day). Discretized data such as the probability of the morning, the probability of the afternoon of the month, and the like is calculated in advance and stored in the prior information storage means 71.

  When creating a composite distribution of a probability distribution based on prior information and a probability distribution based on each score, for example, a weighted arithmetic mean or a geometric mean can be used. The probability distribution based on prior information (data at each time point on the time axis) is Pd, the probability distribution based on each score (data at each time point on the time axis) is Pe, α is the weight of the geometric mean, and β is the phase When the weight of the arithmetic mean and γ is the weight between the geometric mean and the arithmetic mean, the composite distribution (data at each time point on the time axis) Pm can be calculated by the following equation.

Pm = [gamma] * {Pd [ ^alpha] * Pe ^{(1- [alpha])} } + (1- [gamma]) {[beta] * Pd + (1- [beta]) * Pe}.

  Here, when α = 1, only a probability distribution based on prior information is considered, and when α = 0, only a probability distribution based on each score is considered.

  When performing the process of displaying the composite distribution on the calendar, the composite probability data is calculated for each calendar unit section according to the above equation. At this time, if the calendar unit section and each predictive time section do not match (when there is a difference in time length or a time difference), the above-described processing of FIG. 9 is executed.

  As shown in FIG. 13, when the score output is in the format B (see FIG. 5), the event occurrence probability may be an output format that expresses the cumulative distribution. In the case of the format B, there is an overlap in each of the predictive time sections B1 to BN. Therefore, when creating a cumulative distribution, it is necessary to handle the overlapping sections. For example, for the predictive time section B1, there is no predictive time section shorter than the self, but for the predictive time section B2, there is a predictive time section B1 shorter than the self, and similarly, for the predictive time section B3, There are also short predictive time sections B1 and B2, and for the predictive time section B4, there are predictive time sections B1, B2 and B3 shorter than oneself. Processing is required.

  When the score output is in the format A (see FIG. 4), there is no overlap between the sections, so that no contradiction occurs even if all the predictive time sections A1 to AN are regarded as a uniform distribution. However, when the score output is in the format B (see FIG. 5), there is a score corresponding to the predictive time interval shorter than that of the self, so if one's own distribution is assumed without considering it, a contradiction (cumulative distribution) Despite this, there is a possibility that a contradiction that the value decreases in the middle). Each predictive time interval shorter than yourself may be consistent if it is close to a uniform distribution. For example, there is a clear peak in the probability of occurrence in one of the predictive time intervals shorter than you. In such a case, it is highly possible that a contradiction will occur if one assumes a uniform distribution without considering that fact. Therefore, in order to avoid such a contradiction and display a smooth cumulative distribution, the output unit 70 executes the following processing.

  First, when the score output is in the format B (see FIG. 5), the output means 70 selects the probability distribution of each of the plurality of predictive time intervals or the cumulative distribution thereof from any of a plurality of types of distributions prepared in advance. In this case, the distribution used to create one's probability distribution or its cumulative distribution using the score corresponding to the predictive time interval with a shorter time length than one's own is used. Execute the process to select.

  For example, a plurality of types of distributions such as a normal distribution or a cumulative distribution thereof, a Pareto distribution or a cumulative distribution thereof, and the like are prepared in advance. For example, when determining the probability distribution used for the predictor time interval B4 or its cumulative distribution, the score corresponding to the predictor time interval B3 having a shorter time length than the predictor time interval B4 (self), or the predictor time interval A distribution to be adopted is determined from a plurality of types of distributions prepared in advance using all the scores corresponding to B1, B2, and B3. Similarly, for the predictive time interval B3, both the score corresponding to the predictive time interval B2 having a shorter time length than the predictive time interval B3 (self) or the scores corresponding to the predictive time intervals B1 and B2 are used. For the time interval B2, the distribution to be adopted is determined using the score corresponding to the predictive time interval B1 having a shorter time length than the predictive time interval B2 (self). In addition, since there is no predictive time interval shorter than the predictive time interval B1, the distribution to be adopted may be determined in advance, such as a uniform distribution, a normal distribution, or a cumulative distribution thereof.

  Since the distribution is selected so that no contradiction occurs, for example, when the distribution of the predictor time interval B4 is selected, the value of the cumulative distribution at time T3 is the score value corresponding to the predictor time interval B3 (example in FIG. 13). Then, select not to be less than 80%). Similarly, when selecting the distribution of the sign time interval B3, select so that the cumulative distribution value at the time T2 does not fall below the score value corresponding to the sign time interval B2 (30% in the example of FIG. 13). To do. When selecting the distribution of the predictor time interval B2, the value of the cumulative distribution at the time point T1 is selected so as not to fall below the score value (10% in the example of FIG. 13) corresponding to the predictor time interval B1.

  In addition, when an overall cumulative distribution (cumulative distribution over all sections from Tnow to T4) is created by synthesizing the cumulative distributions of the predictive time sections B1 to B4, overlapping sections (in the example of FIG. 13). , (Tnow to T1), (T1 to T2), and (T2 to T3)), a process of combining the respective distributions at a predetermined ratio is executed. Since the predetermined ratio includes zero, it is roughly divided into the following two processes.

  The first process adopts a probability distribution of event occurrence or its cumulative distribution that is created in a pseudo manner using a score corresponding to the shortest predictive time interval for overlapping intervals, and other predictive time intervals This is a process that does not adopt a probability distribution of event occurrence or its cumulative distribution that is created in a pseudo manner using a score corresponding to.

  In the example of FIG. 13, the first process adopts the distribution of the shortest predictive time section B1 among the four predictive time sections B1 to B4 for the section (Tnow to T1), and (T1 to T2). ), The distribution of the shortest predictive time section B2 among the three predictive time sections B2 to B4 is adopted, and for the section (T2 to T3), of the two predictive time sections B3 and B4, The distribution of the shorter sign time interval B3 is adopted. Accordingly, the distribution of (Tnow to T3) is discarded (not adopted) for the predictive time section B4, and only the distribution of (T3 to T4) is adopted, and the distribution of (Tnow to T2) is used for the predictive time section B3. Is discarded (not adopted), only the distribution of (T2 to T3) is adopted, and for the predictive time interval B2, the distribution of (Tnow to T1) is discarded (not adopted), and only the distribution of (T1 to T2) is adopted. And the entire distribution is adopted for the predictive time interval B1.

  In the second process, for overlapping sections, a probability distribution of event occurrences or a cumulative distribution thereof that are artificially created using the scores corresponding to the respective predictive time sections are synthesized at a predetermined ratio. Execute the process.

  In the second process, in the example of FIG. 13, the distribution data H11, H12, H13, and H14 of the distribution of the four predictive time periods B1 to B4 are combined into the synthesis rates M11, M12, Adopted at a ratio of M13 and M14 (the sum of M11, M12, M13 and M14 is 100%, ie, 1), and they are summed. For the section (T1 to T2), the distribution data H22, H23, and H24 of the three predictive time sections B2 to B4 are adopted at the ratios of the synthesis rates M22, M23, and M24 (M22, M23, and M24). The sum is 100%, i.e. 1). Further, for the section (T2 to T3), the distribution data H33 and H34 of the two predictive time sections B3 and B4 are adopted at the ratio of the synthesis rates M33 and M34 (the total of M33 and M34 is 100%, I.e., 1), summing them. As a result of the summation, if there is a contradiction (a contradiction that the cumulative probability decreases when moving from the relevant section where the summation is performed to the right-hand section), the distribution in the section where the summation is performed May be compressed (multiplied by the same coefficient over the entire section), or the synthesis rate may be changed.

  Further, when the score output is in the format B (see FIG. 5), when the occurrence probability is displayed on the calendar, there is no contradiction (a contradiction that the cumulative probability is reduced halfway) by the above synthesis. Using the created cumulative distribution, it is possible to calculate the probability assigned to the calendar unit interval. For example, for the unit interval on April 22 of the calendar, a difference value (HC2−HC1)% obtained by subtracting the value (probability) of HC1 from the value (probability) of HC2 in the cumulative distribution is assigned. A difference value (HC3−HC2)% obtained by subtracting the value (probability) of HC2 from the value (probability) of HC3 in the cumulative distribution can be assigned to the unit interval.

  When the score output is in the format B (see FIG. 5), when the occurrence probability is displayed on the calendar, the process of creating a consistent cumulative distribution by the above synthesis is not performed, that is, the contradiction. Without considering the presence or absence of occurrence, it is considered that all the predictive time sections B1 to B4 have a uniform distribution, and the probability assigned to the calendar unit section may be calculated. For example, if the score (probability) of the predictive time section B4 corresponding to (Tnow to T4) = 90%, the 90% is allocated according to the length of the section, and the probability of (Tnow to T1) = 11.1. 25%, probability of (T1 to T2) = 11.25%, probability of (T2 to T3) = 22.5%, probability of (T3 to T4) = 45%, of which there is no overlap of sections ( The probability of T3 to T4) = 45% may be adopted and assigned to the calendar unit interval by the process shown in FIG. However, the numerical value of (T3 to T4) probability = 45% is contradictory when there is a clear peak in the probability of occurrence in another section before that (for example, section (T2 to T3)). Is likely to occur.

  Also, if the score (probability) of the predictive time interval B3 corresponding to (Tnow to T3) = 80%, the probability of (Tnow to T1) = 20% is allocated according to the length of the interval. The probability of (T1 to T2) = 20%, the probability of (T2 to T3) = 40%, and the probability of (T2 to T3) = 40% is adopted, or (T2 to T3) Probability = 40% multiplied by the composite rate M33, and the probability (T2 to T3) = 22.5% obtained from the score (probability) = 90% of the predictive time interval B4 described above was multiplied by the composite rate M34. You may employ | adopt the value which totaled the value. However, in addition to the possibility of inconsistency in such processing, the total value of the probability of each section may exceed 100%. Adjustment processing (for example, processing for multiplying the probability of each adopted section by a certain coefficient, etc.) is required. Therefore, it is preferable to create a cumulative distribution with no contradiction as shown in FIG. 13 and calculate the probability to be assigned to the calendar unit section, as shown in FIG.

  In addition, the further adjustment process for making the total value within 100% is, for example, the probability (T3 to T4) of the score (probability) = 90% of the predictive time interval B4 = 45%, Score (probability) of predictor time interval B3 = probability of (T2 to T3) = 80% out of 80% and score (probability) of predictor time interval B2 = probability of (T1 to T2) out of 30% = 15% and the score (probability) of the predictive time interval B1 = 10%, and when it becomes 110%, each probability is multiplied by 90% / 110% to 36.82%, 32. 73%, 12.27%, 8.18%, and the total is 90%. The total is adjusted to 90% in accordance with the score of the longest predictive time interval B4.

  <Configuration of Data Collection Device 80>

  The data collection device 80 includes one or a plurality of computers (servers), and includes a data collection unit 81 having a database management system (DBMS) function and a database 82. The database 82 includes a predictive feature extractor input / output data storage means 83 for storing input / output data of the predictive feature extractor 50, and a predictor detector input / output data storage means 84 for storing input / output data of the predictive detector 60. Is included.

  Here, the data collection means 81 is realized by a central processing unit (CPU) provided in the computer main body constituting the data collection device 80 and one or a plurality of programs that define the operation procedure of this CPU. The The database 82 is realized by, for example, a hard disk drive (HDD), a solid state drive (SSD), or the like.

  In the predictive feature extractor input / output data storage means 83, data transmitted via the network 1 by the output means 70 of the predictive detection processing device 30, that is, state data input to the preceding predictive feature extractor 50 ( Added time information and device identification information), and output data (likelihood) of the final layer (output layer) of the predictive feature extractor 50 are stored. Among these state data, state data in which the determination result is not particularly clear (state data in which the difference between the likelihood indicating the certainty of the specific state and the likelihood indicating the certainty not being the specific state is small ) Or tagging work by crowdsourcing (judgment work by a person as to whether or not it is in a specific state) is performed on all the state data, and the predictive feature extractor input / output data storage means 83 includes: Tag information that is a work result of the crowdsourcing is also stored in association with the state data. Then, the state data subjected to crowdsourcing and the tag information thereof are used for learning processing by the sign feature extractor learning unit 91 of the learning device 90.

  In the sign detector input / output data storage means 84, data transmitted via the network 1 by the output means 70 of the sign detection processing apparatus 30, that is, feature data input to the sign detector 60 in the subsequent stage, and sign The score output from the detector 60 is stored. When an event actually occurs, information about the time of occurrence of the event is fed back and transmitted from the user terminal 120 by the user, or transmitted from the administrator terminal 130 by the system administrator who has received the notification from the user. Therefore, feature data is automatically tagged based on information about the actual time of occurrence (automatic determination of whether the feature data was extracted at a time point belonging to a non-target interval or multiple predictive time intervals) In the sign detector input / output data storage means 84, tag information based on the feedback is also stored in association with the feature data. The feature data to which the feedback is performed and the tag information thereof are used for the learning process by the sign detector learning unit 92 of the learning device 90.

  <Configuration of learning device 90>

  The learning device 90 includes one or a plurality of computers (servers), and includes a sign feature extractor learning unit 91 and a sign detector learn unit 92. Here, each of the learning means 91 and 92 is realized by a central processing unit (CPU) provided inside the computer main body constituting the learning device 90 and one or a plurality of programs that define the operation procedure of this CPU. Is done.

  The sign feature extractor learning unit 91 executes the learning process for the sign feature extractor 50 using the state data and tag information stored in the sign feature extractor input / output data storage unit 83 of the data collection device 80. Thus, a predictive feature extraction model (CNN parameter or the like) is newly created, and a process of updating the predictive feature extraction model stored in the predictive feature extraction model storage unit 52 with the created model is executed. The learning process by the predictive feature extractor learning unit 91 is executed when a certain amount of learning data is prepared by crowdsourcing, so the execution timing of crowdsourcing (for example, every other week, every other month) Etc.).

  The sign detector learning unit 92 executes the learning process for the sign detector 60 by using the feature data stored in the sign detector input / output data storage unit 84 of the data collection device 80 and the tag information thereof. A new sign detection model (such as an RNN parameter) is created, and a process of updating the sign detection model stored in the sign detection model storage means 62 with the created model is executed. The learning process by the sign detector learning unit 92 is executed when an event has actually occurred once or several times, and a certain amount of learning data has been prepared. Just run it.

  <Configuration of crowdsourcing system 100>

  The crowdsourcing system 100 is configured by one or a plurality of computers (servers) and performs crowdsourcing for creating learning data for the predictive feature extractor 50. The crowdsourcing system 100 may be configured to cooperate with an external system configured by one or a plurality of computers (servers) operated externally.

  Specifically, the crowd sourcing system 100 is the state data stored in the predictive feature extractor input / output data storage unit 83 from the data collection device 80, particularly the state data whose determination result is not clear (a specific state). State data in which the difference between the likelihood indicating the certainty of the thing and the likelihood indicating the certainty that it is not a specific state is small), or all the state data is acquired via the network 1 and the acquired state data And using the comparison data corresponding to the device identification information added to the state data, for example, a microtasking process for creating a crowdsourcing work screen 200 as shown in FIG. Work with display data (HTML data, etc.) on the screen 200 via a network (network 1 or other network) By transmitting to the terminal 110, the work of performing simple judgment (so-called crowdsourcing micro task) to perform a large number of workers operating the operator terminal 110 (crowd).

  In the case of a configuration that cooperates with an external system, the crowdsourcing system 100 executes a microtasking process for creating a crowdsourcing work screen 200 as shown in FIG. Data (HTML data or the like) may be transmitted to an external system, or status data and comparison data corresponding to the data may be transmitted to the external system, and the crowd sourcing work screen 200 is created by the external system. Tasking processing (the screen display form is handed over to an external system in advance) may be executed. Then, display data (HTML data or the like) for the crowdsourcing work screen 200 is transmitted from the external system to the worker terminal 110 via the network.

  Here, the comparison data is data (for example, image data, sound data, etc.) prepared in advance and stored in the crowdsourcing system 100, and the status data capturing device 20 (camera, etc.) corresponding to the device identification information. ) Is a data indicating a specific state to be captured using the state data captured. For example, using the image data from the camera, a specific state X (for example, a cow standing state), a specific state Y (for example, a cow's tail raised state), and a specific state Z (for example, cow urine) In the case of capturing the exposure state of the membrane / amniotic membrane), the device identification information is information for identifying the camera, and the comparison data is an image that actually shows the specific states X, Y, and Z. It is data.

  In FIG. 6, the crowdsourcing work screen 200 is acquired from each display unit 201, 202, 203 of the image actually showing the specific state X, Y, Z displayed by the comparison data, and the data collection device 80. A display unit 210 for an image (an image that the operator wants to judge the state) displayed based on the state data is provided. A worker (crowd) who operates the worker terminal 110 reads the question text described on the screen 200, and compares the images on the display units 201, 202, and 203 with the images on the display unit 210, thereby answering. When the check input to the unit 220 is performed and the “Send” button 230 is clicked, the answer data input to the answer unit 220 is transmitted to the crowdsourcing system 100 via the network. Data is received, tag information associated with the status data is created using the received answer data, and the created tag information is stored in the predictive feature extractor input / output data storage means 83 of the data collection device 80 A process of storing the data in association with the data is executed.

  In the case of a configuration that cooperates with an external system, the response data may be transmitted to the external system. In this case, the crowdsourcing system 100 acquires the response data via the external system. .

  A crowd (for example, a cloud worker such as a housewife or a student) determines what kind of data the image data (status data and comparison data) displayed on the crowdsourcing work screen 200 is (for example, the data of the data). Judgment work can be performed without knowledge or skills without being particularly aware of the meaning, purpose, nature, situation, value, etc. In the case of a configuration that cooperates with an external system, the work (microtask) requested by the sign detection system 10 is a work of another system (microtask) generated by a system different from the sign detection system 10. Presented side by side to the crowd. The cloud worker freely selects a job from various jobs, but when the job worker happens to select a job requested by the sign detection system 10, the cloud worker operates the worker terminal 110 illustrated in FIG. Become a worker (crowd).

  Moreover, the download of the crowdsourcing work screen 200 to the worker terminal 110 from the crowdsourcing system 100 or an external system linked thereto is performed by, for example, the following method. As a first method, a job (so-called crowdsourcing) requested to a worker (crowd) who operates the worker terminal 110 using a general purpose application such as LINE or a communication means such as e-mail. Communicate that there is a microtask). The worker who has received this communication accesses a predetermined website or a website specified in a communication sentence (text or voice) by a communication tool or e-mail. This website is configured by the crowdsourcing system 100 or an external system linked thereto. When a worker (crowd) who operates the worker terminal 110 browses the electronic bulletin board provided on the website with a general-purpose browser and selects his / her job on the electronic bulletin board (freely arbitrary) Or a job designated by the system may be selected.) The crowdsourcing system 100 or an external system linked with the crowdsourcing system 100 displays the crowdsourcing work screen 200 (see FIG. 6) is transmitted to the worker terminal 110 via the network.

  As a second method, a dedicated application installed in the worker terminal 110 displays an electronic bulletin board on the screen of the worker terminal 110, and an operator operating the worker terminal 110 You may make it select the work which he is in charge of, and others are the same as that of the case of said 1st method.

  <Configuration of Worker Terminal 110, User Terminal 120, and Administrator Terminal 130>

  The worker terminal 110, the user terminal 120, and the administrator terminal 130 are configured by a computer, and include input means such as a mouse and a keyboard and display means such as a liquid crystal display. These terminals 110, 120, and 130 may be mobile devices such as smartphones and tablet terminals, for example.

  <Overall flow of predictive detection processing>

  In the present embodiment, processing for detecting a sign that occurs before the occurrence of an event (for example, calving) is performed by the sign detection system 10 as follows.

  In FIG. 7, learning is performed in advance before starting system operation, and a predictive feature extraction model (such as a CNN parameter) and a predictive feature detection model (such as an RNN parameter) are created. The extracted model storage means 52 and the sign detection model storage means 62 are stored. The learning method for creating these models is the same as the processing by the sign feature extractor learning means 91 and the sign detector learn means 92.

  Subsequently, after the operation of the system is started, first, state data (mainly perceptual data such as image data and sound data) indicating the state of the sign detection target (eg, cow) is detected by the state data capturing device 20 (camera, microphone, etc.). ) And status data output from the status data fetching device 20 is received by the connection device 21 by wire or wirelessly. Then, the status data is detected from the connection device 21 through the network 1 together with the time information (date / time information of year / month / day / hour / minute / second, or information indicating the data generation order to replace it) and the device identification information. The data is transmitted to the processing device 30, and the status data acquisition means 40 acquires the status data in the sign detection processing device 30 (step S 1).

  Next, the state data acquired by the state data acquisition unit 40 is input to the predictive feature extractor 50, and the predictive feature extraction model storage unit 52 stores the predictive feature extraction model storage unit 52 using the predictive feature extraction pattern recognition processing unit 51. Is used to perform pattern recognition processing (CNN or the like) (step S2), and the obtained output data of the intermediate layer or the like is extracted as feature data (step S3).

  Further, the time series feature data extracted by the sign feature extractor 50 is input to the sign detector 60, and the sign detection processing means 61 uses the sign detection model stored in the sign detection model storage means 62 in time series. Pattern recognition processing (such as RNN) is executed, and a score as a sign detection result is output (step S4).

  Thereafter, the output means 70 outputs the sign detection result using the score output from the sign detector 60 (step S5). The output of the sign detection result is a screen display of the sign detection result on the user terminal 120 or the like. In addition, you may transmit the data for output of an indication detection result via the network 1 to the cooperation system which a user manages. The output means 70 transmits the input / output data of the sign feature extractor 50 and the sign detector 60 to the data collection device 80 via the network 1.

  In addition, the learning data generation process, the learning process, and the model update process described below are processes that are executed in parallel with the sign detection and the output process (steps S1 to S5) described above. However, when the model is updated by the following processing, the processing of steps S1 to S5 is performed using the updated model. Therefore, for the sake of simplification of explanation, the description is made as a series of processing. Assumed to be performed.

  That is, first, when it is the generation timing of the learning data for the predictive feature extractor 50, the state data stored in the predictive feature extractor input / output data storage unit 83 of the data collection device 80 by the crowdsourcing system 100. Then, a number of workers (crowds) who operate the worker terminal 110 are caused to perform tagging work (annotation) (step S6). The generation timing of this learning data is a relatively short interval such as every other week or every other month.

  Subsequently, when it is the learning timing for the sign feature extractor 50 (usually, it may be immediately after step S6), the sign feature extractor input / output of the data collection device 80 is input by the sign feature extractor learning means 91. A new predictive feature extraction model (CNN parameters, etc.) is created by executing a learning process for the predictive feature extractor 50 using the state data stored in the data storage means 83 and its tag information. The predictive feature extraction model stored in the predictive feature extraction model storage means 52 is updated with the model (step S7).

  Next, when it is the generation timing of the learning data for the sign detector 60 (usually, it may be immediately after the event has actually occurred), the event is generated by the user or the system administrator who is the substitute for the event. With respect to the information regarding the timing, the feedback processing to the system from the user terminal 120 or the administrator terminal 130 is performed, and tag information is added to the feature data stored in the predictive detector input / output data storage means 84 of the data collection device 80. Is given (step S8). The generation timing of the learning data is a relatively long interval because it is necessary to actually generate an event that does not occur frequently.

  Subsequently, when it is the learning timing for the sign detector 60 (normally, it may be immediately after step S8), the sign detector input / output data storage means of the data collection device 80 is obtained by the sign detector learning means 92. By using the feature data stored in 84 and the tag information thereof, a learning process for the sign detector 60 is executed to newly create a sign detection model (such as an RNN parameter). The sign detection model stored in the detection model storage means 62 is updated (step S9).

  If the sign detection is continued, the process returns to step S1, and if not continued, the sign detection process by the sign detection system 10 is ended (step S10).

  <Effect of this embodiment>

  According to this embodiment, there are the following effects. That is, the sign detection system 10 is configured to detect the sign by using a two-stage processor including the sign characteristic extractor 50 in the previous stage and the sign detector 60 in the subsequent stage. Can be assigned a role.

  Specifically, the sign feature extractor 50 is in charge of processing for capturing a specific state related to the sign by pattern recognition, and determines whether or not the specific state appears as a sign. It is possible to be in charge of pattern recognition processing that is not aimed at.

  On the other hand, the sign detector 60 performs sign detection using the time-series feature data obtained by repeating the pattern recognition processing by the sign feature extractor 50. Therefore, by performing processing that considers temporal factors, It can be determined whether or not the feature data obtained by the predictive feature extractor 50 is information regarding a specific state appearing as a predictor.

  For this reason, the sign detection system 10 can perform effective sign detection even if the specific state related to the sign includes a specific state that may always appear. That is, for a specific state that may appear at all times, even if the state can be captured by only one stage of pattern recognition processing, it is clear whether or not the captured state appears at the sign. However, it is difficult to detect signs, and it is also possible to simultaneously input the state data acquired at multiple times to the pattern recognizer in order to take the time factor into consideration. It is not practical to input a large amount of state data over a length of time enough to enable predictive detection into a pattern recognizer, resulting in a huge amount of processing. On the other hand, in the sign detection system 10, the sign feature extractor 50 captures a specific state regardless of whether or not it appears as a sign, and whether or not it appears as a sign by the sign detector 60. Alternatively, since the degree is determined, the sign detection can be performed even if the specific state related to the sign includes a specific state that may always appear.

  In addition, since the sign detection system 10 divides the sign detection function into the sign feature extractor 50 and the sign detector 60, it is possible to increase the frequency of learning and speed up the performance improvement of the system. Can be planned. This is because an event for which a sign is desired to be detected may not be a frequently occurring event or may be an event that cannot be forcibly generated at a convenient time. In such a case, it is difficult to collect data for learning. Assuming that the sign detection is performed by one-step pattern recognition processing, the pattern is rarely generated until several rare events occur. The model used in the recognition process cannot be updated, and the performance improvement of the system is considerably slow. On the other hand, in the sign detection system 10, the sign feature extractor 50 captures a specific state regardless of whether or not it has appeared as a sign. Therefore, the learning data for that is the occurrence of an event. Regardless of whether it is collected at any time. Accordingly, since the predictive feature extraction model stored in the predictive feature extraction model storage unit 52 can be updated frequently, the performance of the predictive detection system 10 as a whole can be improved at the update speed.

  Further, in the sign detection system 10, when the sign feature extractor 50 performs learning and updates the sign feature extraction model, the state data used for the learning includes perceptual data that can be recognized by human perception. Therefore, the tagging operation (annotation) for the perceptual data can be realized by crowdsourcing. This perceptual data is suitable for crowdsourcing in that it can be perceived by human perception (for example, the state can be easily identified when viewed or heard by a person). Next, since the state data is input to the predictor feature extractor 50, the state can be identified regardless of the event (for example, cattle calving) and its predictor. This is because it is suitable for crowdsourcing in that anyone can identify the state without specialized knowledge. For this reason, a large amount of data for learning can be collected relatively inexpensively, quickly, and without much effort.

  Further, since the predictive feature extractor 50 uses perceptual data as state data, for example, a non-contact type state data capturing device 20 such as a camera or a microphone can be used. For this reason, non-invasiveness by avoiding attachment to a living body and avoidance of danger at the time of installation can be realized, and a system can be constructed using a relatively inexpensive device such as a camera or a microphone. it can.

  Furthermore, since the predictive feature extractor 50 extracts the intermediate layer output of the neural network, information useful for the processing of the predictive detector 60 in the subsequent stage as compared with the case of extracting the final layer (output layer) output. Can be input as feature data. That is, the output data of the final layer (output layer) is data that only indicates whether or not it falls under a specific state, and there is a small amount of information, and there is information that can be used in the processing of the predictive detector 60. Since it may be missing, on the network from the state data (very large amount of information) input to the input layer to the output data (substantial amount of information) of the final layer (output layer) Can be extracted and used in the processing of the sign detector 60. Further, since the output data of the final layer (output layer) has a small amount of information, if the determination result is incorrect, the influence on the processing of the predictive detector 60 is great, but the output of the intermediate layer is detected by the predictive detector. If it is used in the processing of 60, the influence can be reduced.

  In addition, since the sign detector 60 is configured to set a plurality of sign time sections as sections in which a sign may appear and output a score corresponding to them, more detailed sign detection is performed. In addition, various methods can be adopted for outputting the result of predictive detection, and the user can improve the usability of the system.

  Furthermore, since the sign detection system 10 includes a prior information storage unit 71 and the output unit 70 is configured to display a composite distribution using the prior information, the event occurrence probability distribution based on the prior information is gradually reduced. The display to be corrected can be performed, and the user-friendliness of the system can be improved. For example, when the display time (currently today) is far from the expected time or scheduled time of event occurrence, the probability distribution based on prior information is displayed almost as it is, and approaches the expected time or scheduled time of event occurrence. As it comes, it is possible to display such that the probability distribution based on the acquired state data is gradually reflected.

  Further, since the output means 70 displays the event occurrence probability on the calendar using the output score from the sign detector 60, it is possible to realize an easy-to-see display and the sign together with general schedule information on the calendar. Since information related to detection can also be displayed, the user-friendliness of the system can be improved also in this respect.

  Then, when displaying the event occurrence probability on the calendar, the output means 70 performs the distribution and summation of the scores according to the difference in the time length of the sections and the shift of the sections (see FIG. 9). It is possible to correspond to the switching display of unit intervals. In addition, since the score is distributed and summed, the current position (score output time), which is the starting point for displaying the sign detection result, moves on the calendar as time passes. Can respond. Thereby, the composite distribution using the prior information can be easily displayed.

  Moreover, since the output means 70 can output the sign detection result in various output formats (see FIGS. 8 to 13), the output means 70 can perform output in a format according to the user's request, and a plurality of predictive times. A display in which the probability distributions of the sections or the cumulative distribution thereof are connected smoothly or without contradiction can be realized.

  [Deformation form]

  Note that the present invention is not limited to the above-described embodiment, and modifications and the like within a scope where the object of the present invention can be achieved are included in the present invention.

  For example, in the above-described embodiment, the sign detector 60 has been described as having a configuration for performing time-series pattern recognition processing using a recurrent neural network (RNN), but the sign detector used in the sign detection system of the present invention. However, the present invention is not limited to this. The sign detector described below has an internal configuration different from the internal configuration of the above-described embodiment (see FIGS. 2 and 3), but the overall configuration of the system is the same as the state of FIG. 1 of the above-described embodiment. Since these are the same, in order to simplify the explanation and illustration, FIG. 1 also shows the contents relating to the sign detector described below. Accordingly, although the internal configuration is different from that of the RNN, the sign detector described below is also described and illustrated with the reference numeral 60 as in the above embodiment (see FIGS. 14 to 17). .

  First, the “time-series pattern recognition process” of the sign detector according to the present invention stores an internal memory (the previous or previous internal process result) like the RNN or hidden Markov model (HMM) of the above embodiment. It is not limited to processing by a pattern recognizer that realizes series modeling in the internal structure by having an internal memory), but also includes processing by a static pattern recognizer that does not perform series modeling by internal memory It is. In other words, since the information in the internal memory is reflected in the pattern recognizer having the internal memory, the same input is not always the same for the same input, but in the static pattern recognizer, the same input is used. In contrast, the same output is always obtained. Therefore, “time-series pattern recognition processing” in the present invention includes processing by a static pattern recognizer that always obtains the same output for the same input. Specifically, for example, a mixed Gaussian model ( GMM), simple neural network (NN without internal memory), support vector machine (SVM), etc. Including these, “time-series pattern recognition processing” refers to the time-series data in a situation where data at a new time is continuously generated even if it is a static pattern recognizer. This is because if a plurality of pieces of data within a period (a fixed period of movement) are cut out and input repeatedly, processing that takes into account temporal factors can be realized. In the case of RNN, since it has an internal memory, even if only one-time feature data is input, processing that takes into account temporal elements can be realized in the internal structure. In such a case, Further, the fact that a plurality of time characteristic data may be input simultaneously is described in detail in the description of the above embodiment.

  Next, a specific example will be described. In the examples of FIGS. 14 to 17 below, a plurality of feature data within a certain period (a certain period of movement) from among the time-series feature data in a situation where feature data at a new time is continuously generated. Is the same in that it is cut out and input to a static pattern recognizer. However, the processing described below does not mean that the processing cannot be applied to a pattern recognizer that realizes sequence modeling using an internal memory, and may be applied.

  <When inputting a set of feature data of cut section C one by one to a static pattern recognizer>

  In FIG. 14, the predictive time interval is the same as that in FIG. 4 of the above-described embodiment. Here, four predictive time intervals A1, A2, A3, A4 and a non-target interval (normal time) AX are set. It shall be. In this example, the sign detector 60 cuts out and inputs a plurality of feature data included in the cut-out section C, which is the latest fixed period, from the time-series feature data during operation. The number of feature data in the cut section C is K. The time length of the cut section C is shorter than the predictive time sections A1, A2, A3, A4 and the non-target section (normal time) AX. In addition, if the time-series feature data is F1, F2, F3, F4,... (K = 100, for example), the cutting processing by the cutting section C is (F1, F2,..., F100). , (F2, F3,..., F101) may be cut out one by one, and cut out without overlapping, such as (F1, F2,..., F100), (F101, F102,..., F200). Alternatively, as shown in (F1, F2,..., F100), (F11, F12,..., F110), there is an overlap, but the shift amount is not one but is cut out to be 2 or more. Also good. The setting of the predictive time sections A1 to A4 and the non-target section AX and the setting of the cut-out section C are the same in the examples of FIGS. 15 to 17 below.

  In the example of FIG. 14, the K feature data in the cut section C are input one by one rather than all at once. At this time, the input order is arbitrary, and there is no need to consider the time series. This is because K feature data are regarded as a set. Since K feature data are input to the pattern recognizer one by one, K (K sets) sub-scores are obtained from the pattern recognizer. In this example, each likelihood for A1, A2, A3, A4, and AX is included in one (one set) subscore. Therefore, the predictor detector 60 calculates and outputs a final score using 5 × K likelihoods. For example, since there are K likelihoods corresponding to A1, the scores corresponding to A1 are calculated by adding the K likelihoods and dividing by K. The same applies to A2, A3, A4 and AX. This includes processing (algorithm) by the sign detection processing means 61 including processing for calculating a final score using such a subscore, that is, processing by a pattern recognizer that does not have an internal memory. Of course, during the processing by the sign detection processing means 61, K feature data is stored and K sub-scores are stored (K sets), so that there is a working memory, but series modeling is realized. This means that there is no internal memory (internal memory for storing the previous or previous internal processing result).

  In the example of FIG. 14, the learning process is the same as in the case of the embodiment (see FIG. 4). In addition, the format B (see FIG. 5) described in the above embodiment can be selected as the format for setting the predicting time interval and outputting the score.

  <When obtaining a set of feature data in the cut-out section C as a distribution and calculating similarity between distributions>

  In the example of FIG. 15, a set of feature data is regarded as a distribution both during learning and during operation. However, the number of samples in the distribution differs between learning and operation. At the time of learning, since it is feature data belonging to each of the predictive time sections A1 to A4 and the non-target section AX, there are a large number of feature data, and in particular, the non-target section AX has a very large number of feature data. Depending on the nature of the event that detects the sign, for example, there are thousands, tens of thousands, and hundreds of thousands of samples. On the other hand, at the time of operation, since it is K feature data in the moving cut section C, the number of samples is relatively small. The cut-out section C is a section for determining under what circumstances the current time is related to the detection of an event sign (12 hours before, 24 hours before, etc.). Depending on the time interval of feature data extraction by the predictive feature extractor 50, the number of digits of the number of samples is relatively small compared to the time of learning.

  In the example of FIG. 15, a set of feature data that is an M-dimensional vector is regarded as a distribution during learning and operation, and a representative value (representative vector) of each distribution is obtained. That is, during learning, since there are distributions for A1, A2, A3, A4, and AX, representative values for these distributions are obtained separately. At the time of operation, since there is a distribution based on K feature data (M-dimensional vectors) in the cut section C, a representative value for the distribution is obtained. The number of dimensions of the representative value (representative vector) may or may not match the number of dimensions (M dimension) of the feature data.

  Various representative values (representative vectors) of the distribution can be selected, and may be a combination of a plurality of representative values instead of a single representative value. For example, an average value, a mode value, a median value ( Median), standard deviation, variance vector, frequency vector, or the like can be selected. FIG. 15 shows, as an example, a normal distribution as a distribution, and shows a combination of an average vector μ and a standard deviation vector σ, both of which have the same number of dimensions as feature data and are M-dimensional vectors. (Μ1, σ1) is obtained for the predictive time interval A1, and (μ2, σ2), (μ3, σ3), (μ4, σ4), (μX, σX) are similarly obtained for the other intervals. . The representative values (representative vectors) of the distributions for the sections A1 to A4 and AX obtained during learning are stored in the sign detection model storage unit 62. Similarly, during operation, (μ, σ) regarding the distribution during operation is obtained.

  Here, the representative value of the distribution is a representative vector, but it may be a scalar value, or may be a matrix or a tensor amount.

  When the representative value (representative vector) of the distribution is a frequency vector, for example, it can be obtained as follows. When the values of the respective elements constituting the feature data that are M-dimensional vectors (respective dimension values) are all posterior probabilities, the frequency at which the value of each element is equal to or greater than the threshold value P for each dimension. , Normalization based on the number of feature data (K in operation) that is the number of samples of distribution, and values of each dimension obtained by this normalization may be used as constituent elements of representative values (representative vectors) it can.

  Specifically, the number of feature data is K = 5, the number of dimensions of feature data is M-dimension = 4 dimensions, and the threshold is P = 0.75 (75%). It is assumed that the contents of the fifth feature data are as follows.

First feature data = (0.12, 0.31, 0.81, 0.97)
Second feature data = (0.11, 0.33, 0.45, 0.66)
Third feature data = (0.30, 0.38, 0.96, 0.82)
4th feature data = (0.13, 0.62, 0.68, 0.51)
5th feature data = (0.88, 0.62, 0.68, 0.78)

  At this time, if only the values of the elements of the first dimension are viewed, they are 0.12, 0.11, 0.30, 0.13, 0.88, and therefore the threshold P in this dimension is equal to or greater than 0.75. The number of occurrences of the element is one element, and when normalized, it is one element out of five elements, so 1/5 can be adopted as the value of this dimension. The same goes for the other dimensions.

  Frequency vector = (1/5, 0/5, 2/5, 3/5)

  Although described with K feature data at the time of operation, only the number of samples is increased at the time of learning, which is the same processing. A frequency vector is created for each of the sections A1, A2, A3, A4, and AX. The number of dimensions of the frequency vector matches the number of dimensions of feature data (M dimension = 4 dimensions).

  Further, the frequency vector may be obtained by processing (algorithm) using a static pattern recognizer having no internal memory as follows. For example, all methods used for general identification problems, such as GMM, simple NN, and SVM, can be employed. At this time, the five sections A1, A2, A3, A4, and AX are divided into several sections as sections for calculating the frequency vector, and likelihoods corresponding to these sections are output. The division form of this section is arbitrary. For example, in the case of two divisions, it is divided into “section A1 + A2 + A3 + A4 = section A1234” and section AX, or divided into section A1 and “section A2 + A3 + A4 + AX = section A234X”. Or, it can be divided into “section A1 + A2 = section A12” and “section A3 + A4 + AX = section A34X”. In the case of three divisions, for example, it is divided into a section A1, “section A2 + A3 + A4 = section A234”, and section AX, “section A1 + A2 = section A12”, “section A3 + A4 = section A34”, section It can be divided into AX. Similarly, it is possible to divide into four parts and five parts. When the number of parts is divided into five parts, it is divided into the original five sections A1, A2, A3, A4 and AX. Furthermore, as the frequency vector calculation section, two or more sections formed by dividing at a position completely different from the section positions of the five sections A1, A2, A3, A4, and AX may be prepared.

  For example, when five sections A1, A2, A3, A4, and AX are prepared as the frequency vector calculation sections in the same way as the original five sections A1, A2, A3, A4, and AX, The likelihoods corresponding to these five sections are collectively output (a numerical value indicating the likelihood that the feature data belongs to A1, a numerical value indicating the likelihood that the feature data belongs to A2,... A pattern recognizer that outputs numerical values indicating the likelihood may be constructed, and the likelihood corresponding to each of the sections A1, A2, A3, and A4 is separately output (if the predictive time section A1, the feature data is 4 pattern recognizers that output a numerical value indicating the certainty of belonging to A1 and a numerical value indicating the certainty of not belonging to A1) may be constructed. When two sections (for example, section A 1234 and section AX) are formed as the frequency vector calculation section, only one pattern recognizer is required.

  Specifically, for example, it is assumed that two sections A1234 and AX are provided as sections for calculating the frequency vector. In the case of a mixed Gaussian model (GMM), a section that outputs the likelihood corresponding to the section A1234 by maximum likelihood learning using all feature data of the two sections A1234 and AX obtained as learning data. Assume that a GMM for A1234 is constructed. Therefore, the GMM for section A1234 outputs the likelihood corresponding to A1234 (a numerical value indicating the likelihood that the feature data belongs to A1234 and a numerical value indicating the likelihood that the feature data does not belong to A1234) as a probability. Shall. Then, the numerical value (probability) indicating the probability of belonging to A1234 is divided into five sections (0 or more, less than 0.2, 0.2 or more, less than 0.4, 0.4 or more, less than 0.6, 0.6 The frequency vector of 0.8 or more and 0.8 or more is obtained. As the contents of the feature data, the numerical example described above is used.

  The first feature data = (0.12, 0.31, 0.81, 0.97) is input to the A1234 GMM, and the output (likelihood belonging to A1234 = 0.80, not belonging to A1234) If likelihood = 0.20), a likelihood belonging partition display vector of (0, 0, 0, 0, 1) is created using likelihood = 0.80 belonging to A1234.

  The second feature data = (0.11, 0.33, 0.45, 0.66) is input to the A1234 GMM, and the output (likelihood belonging to A1234 = 0.91, not belonging to A1234) If likelihood = 0.09), a likelihood belonging partition display vector of (0, 0, 0, 0, 1) is created using likelihood = 0.91 belonging to A1234.

  The third feature data = (0.30, 0.38, 0.96, 0.82) is input to the A1234 GMM, and the output (likelihood belonging to A1234 = 0.63, does not belong to A1234 If likelihood = 0.37), a likelihood belonging partition display vector of (0, 0, 0, 1, 0) is created using likelihood = 0.63 belonging to A1234.

  The fourth feature data = (0.13, 0.62, 0.68, 0.51) is input to the A1234 GMM, and the output (likelihood belonging to A1234 = 0.55, does not belong to A1234 If the likelihood is 0.45), a likelihood belonging section display vector of (0, 0, 1, 0, 0) is created using the likelihood = 0.55 belonging to A1234.

  The fifth feature data = (0.88, 0.62, 0.68, 0.78) is input to the A1234 GMM, and the output (likelihood belonging to A1234 = 0.38, does not belong to A1234 If likelihood = 0.62), a likelihood belonging partition display vector of (0, 1, 0, 0, 0) is created using likelihood = 0.38 belonging to A1234.

  When normalization is performed using the five likelihood attribution section display vectors obtained for the first to fifth feature data, the following is obtained.

  Frequency vector = (0/5, 1/5, 1/5, 1/5, 2/5)

  If three sections (for example, A1, A234, AX) are provided as frequency vector calculation sections, the five-dimensional frequency vector as described above is obtained for the likelihood for the section A1, and the likelihood for the section A234 is obtained. As for the degree, a five-dimensional frequency vector as described above is obtained. Combining these two vectors (an operation to obtain a vector composed of each component of a matrix obtained by calculating the direct product (tensor product) of two vectors. For example, u = (u1, u2, ..., um) is m. If a dimensional vector, v = (v1, v2,…, vn) is an n-dimensional vector, its direct product is an m × n matrix whose components are ((u1 × v1, u1 × v2, …, U1 × vn), (u2 × v1, u2 × v2,…, u2 × vn),…, (um × v1, um × v2,…, um × vn)). To obtain an m × n-dimensional vector), a 25-dimensional frequency vector (u = (u1, u2, u3, u4, u5) becomes a 5-dimensional frequency vector, v = (v1, v2, v3, When (v4, v5) is a five-dimensional frequency vector, the combined frequency vector is (u1 × v1, u1 × v2,…, u1 × v5, u2 × v1, u2 × v2,…, u2 × v5,…, u5 × v1, u5 × v2,..., u5 × v5))).

  In addition, the number of sections for dividing a numerical value (probability) indicating the certainty of belonging to each section (excluding the section farthest from the event occurrence time) may be different for each section. For example, three intervals (for example, A1, A234, AX) are provided as frequency vector calculation intervals, and the number of sections in the interval A1 is (0 or more, less than 0.3, 0.3 or more, less than 0.7, 0 .7 or more) and the number of sections in section A234 are divided into two sections (0 to less than 0.7, 0.7 or more), one 3D frequency vector and one 2D frequency vector. Since one is obtained, they are combined to obtain a 2 × 3 = 6 dimensional frequency vector.

  Similarly, five sections A1, A2, A3, A4, and AX are provided as frequency vector calculation sections, and the number of sections in all sections (four sections excluding the section AX farthest from the event occurrence point) is set. Assuming that each has two sections, the two-dimensional frequency vectors as described above are obtained for the likelihoods for each of the sections A1 to A4, resulting in a total of four two-dimensional frequency vectors. A frequency vector is obtained. If four sections are provided as frequency vector calculation sections, and the number of sections in each section (three sections excluding the section farthest from the event occurrence point) is divided into five sections, three sections, and two sections, respectively. Since one 5-dimensional frequency vector, one three-dimensional frequency vector, and one two-dimensional frequency vector are obtained, they are combined to obtain a 5 × 3 × 2 = 30-dimensional frequency vector. .

  Although described with K feature data at the time of operation, only the number of samples is increased at the time of learning, which is the same processing. At the time of learning, for example, when a GMM for one section A1234 (one GMM) is prepared, a frequency vector is created for each of the original sections A1, A2, A3, A4, and AX for the GMM for A1234 To do. A pattern recognizer called GMM for A1234 is a recognizer for the section when the section for calculating the frequency vector is formed, and therefore has a different concept from the original sections A1, A2, A3, A4, and AX. It is a recognizer for the section which is divided. More specifically, when all feature data belonging to the original section A1 is input to the A1234 GMM, a large number of likelihood belonging section display vectors are created, and normalization is performed using all of them. The frequency vector as described above is created, and the frequency vector is the frequency vector for the original section A1. For the original sections A2, A3, A4, and AX, if the same processing is performed using the GMM for A1234, the respective frequency vectors for the original sections A2, A3, A4, and AX are created. Then, the generated frequency vector for the original sections A1, A2, A3, A4, and AX is stored in the predictive detection model storage unit 62 as a representative value (representative vector) of the distribution and handled as a parameter. The GMM parameters for A1234 are also used during operation, and are therefore stored in the sign detection model storage means 62.

  Further, as described above, when forming a section for calculating a frequency vector, it can be made independent of the original sections A1, A2, A3, A4, and AX, and can be an arbitrary section. Various recognizers such as GMM for A34, GMM for A34, GMM for A123, and GMM for A234 are prepared. Therefore, when feature data belonging to each of the original sections A1, A2, A3, A4, and AX is input to each of the recognizers, the original sections A1, A2, A3, and the number of the recognizers are the same. A frequency vector for each of A4 and AX will be created. At this time, depending on the number of prepared recognizers, that is, depending on the number of divisions when forming a section for calculating a frequency vector, A process occurs in which the frequency vectors thus synthesized are combined into one frequency vector. Regardless of how the frequency vector calculation section is set, there is only one frequency vector for the original section A1, and similarly there is one frequency vector for the original sections A2, A3, A4, and AX. It is to make it one by one. More specifically, when a section divided into two (for example, two sections A1234 and AX) is formed as a section for calculating the frequency vector, since there is one recognizer, synthesis is not necessary. Form divided sections, prepare four recognizers (or one recognizer that outputs the outputs of the four recognizers together), and all the sections (except the sections farthest from the event occurrence time) ) Is divided into two sections, as described above, a total of four two-dimensional frequency vectors are synthesized to form a 16-dimensional frequency vector. Since each of the four recognizers creates a two-dimensional frequency vector for the original section A1, by synthesizing the four two-dimensional frequency vectors for the original section A1, 16 for A1 is synthesized. This means that a dimensional frequency vector is created, and similarly, for the original sections A2, A3, A4, and AX, a 16-dimensional frequency vector is created for each of A2, A3, A4, and AX. Thereby, a frequency vector is created for each of the sections A1, A2, A3, A4, and AX. In addition, since the frequency vector is two-dimensional (in the case of synthesis, four dimensions, eight dimensions, and sixteen dimensions), it does not match the number of dimensions of feature data (M dimensions = 4 dimensions). Therefore, the number of dimensions of the representative value (representative vector) of the distribution formed by the plurality of feature data may not match the number of dimensions of the feature data.

  As described above, during operation, a representative value (representative vector) of the distribution of K feature data in the cut-out section C is obtained, and a representative value of the distribution of each section A1, A2, A3, A4, and AX. Under a situation where (representative vector) is obtained in advance and stored in the sign detection model storage means 63, the similarity between representative values (representative vectors) can be calculated. That is, the similarity L1, L2, L3, L4, LX between the representative value (representative vector) of the distribution at the time of operation and the representative value (representative vector) of the distribution of each section A1, A2, A3, A4, AX. Can be calculated. At this time, as the similarity, for example, when a normal distribution is assumed and an average vector μ and a standard deviation vector σ are obtained as representative values of the distribution, KL divergence (KL distance) can be used. Further, for example, when a frequency vector is obtained, Euclidean distance, cosine similarity, and the like can be used.

  Since the calculated similarities L1, L2, L3, L4, and LX are subscores, a final score is calculated and output using these subscores. For example, in the case of the KL distance indicating the distance between the distributions, the value is 0 when the distributions completely match, and the numerical value increases when they are separated from each other. A conversion process or the like is performed to increase the numerical value so that the probability can be imaged. Such conversion processing may be performed by a function or a table. Further, the function of such conversion processing may be provided in the output means 70, and in that case, the similarity that is the sub-score becomes the final output score of the sign detector 60.

  <When obtaining a similarity between vectors by vectorizing a set of feature data in the cut section C>

  In the example of FIG. 16, a mixed Gaussian model (GMM) for each section A1, A2, A3, A4, and AX and the entire GMM are constructed by maximum likelihood learning during learning. The entire GMM is a model obtained by learning using all feature data belonging to each section A1 to A4, AX (A1 feature data set + A2 feature data set +...), Universal background Model (UBM). The GMM for each section A1 to A4 and AX is a partial information (predictive time) obtained from a set of feature data belonging to each section A1 to A4 and AX with respect to the UBM indicating the overall and average contents. If it is section A1, it is a model to which information (only A1 information) is adapted.

  Subsequently, a super vector (SV) is created for each of the GMMs in the sections A1 to A4 and AX and the entire GMM. This SV is a high-dimensional vector formed by concatenating all the Gaussian distribution average vectors μ constituting the GMM. This creates an A1 GMM SV, an A2 GMM SV, an A3 GMM SV, an A4 GMM SV, an AX GMM SV, and an overall GMM SV.

  Then, using these SVs, a projective matrix and i-vectors for the sections A1 to A4 and AX are calculated by performing probabilistic factor analysis based on the following formula. The i vector is a low-dimensional vector having a dimension number smaller than that of the SV. The system administrator who constructs the system selects and determines the dimension number of the i vector. In addition, the technique itself regarding i vector is a well-known technique (refer nonpatent literature 1).

  SV of GMM of each section A1 to A4, AX = SV of entire GMM + projection matrix × i vector of each section A1 to A4, AX

  Thereby, the i vector of A1, the i vector of A2, the i vector of A3, the i vector of A4, the i vector of AX, and one projection matrix are obtained. After that, the i-vector, projection matrix, and overall GMM SV of each section A1 to A4 and AX are stored in the sign detection model storage means 62. The reason why the projection matrix and the SV of the entire GMM are stored is that they are used as fixed values when the i-vector at the time of operation is calculated by the above formula.

  Note that the i-vector may not be calculated, and the process at the time of learning may be the process up to the calculation of the SV, and the SV may be used for the similarity determination. When using SV for similarity determination, it is necessary to store the SV of each of the sections A1 to A4 and AX in the sign detection model storage means 62. However, since the i vector is not calculated during operation, It is not necessary to store the GMM SV.

  At the time of operation, the GMM at the time of operation is generated using the K feature data in the cut section C, and further, the SV of the GMM at the time of operation is generated. Then, the generated SV at the time of operation is in the state where the projection matrix and the SV portion of the entire GMM are fixed values (these are read from the predictive detection model storage means 62 as parameters). To calculate the i vector at the time of operation (actual calculation is performed by EM algorithm or the like).

  As described above, at the time of operation, i-vectors at the time of operation are obtained from K feature data in the cut-out section C, and i-vectors of the sections A1, A2, A3, A4, and AX are obtained in advance. Under the situation stored in the sign detection model storage unit 63, the similarity between i vectors can be calculated. That is, the similarity R1, R2, R3, R4, RX between the i vector at the time of operation and the i vector of each section A1, A2, A3, A4, AX can be calculated. At this time, as the similarity, for example, a cosine similarity can be used.

  Since the calculated similarities R1, R2, R3, R4, and RX are subscores, a final score is calculated and output using these subscores. For example, the conversion process from the subscore to a numerical value indicating the magnitude of the probability is performed. Such conversion processing may be performed by a function or a table. Further, the function of such conversion processing may be provided in the output means 70, and in that case, the similarity that is the sub-score becomes the final output score of the sign detector 60.

  <When forming a segment in each section A1-A4, AX>

  In the example of FIG. 17, segments that are small sections are formed in the sections A1 to A4 and AX. In the predictive time interval A1, segments A1 (S1), A1 (S2), A1 (S3), A1 (S4),... Are formed. Therefore, the time length of these segments is shorter than the time length of the predictive time interval A1. Moreover, the section of each segment may overlap and may be formed without overlapping. The same applies to the other sections A2 to A4 and AX.

  In the example of FIG. 15 described above, the representative value (representative vector) of the distribution of each section A1 to A4, AX obtained by learning, and the representative value of the distribution of the cut section C during operation, which is a shorter period than this. Similarities L1 to L4 and LX between distributions with (representative vectors) were calculated. In the example of FIG. 16 described above, the i vector or super vector (SV) of each of the sections A1 to A4 and AX obtained by learning, and the i vector of the cut section C during operation that is shorter than this. Alternatively, similarities R1 to R4 and RX between vectors with SV are calculated. Therefore, in the example of FIGS. 15 and 16 described above, for example, in the case of the predictive time interval A1, the content indicating the entire interval A1 (content averaged in the interval A1) and the shorter interval are cut. The contents of the outgoing section C are compared. For this reason, even if there is a part of the content similar to the content of the cut section C in the section A1, it may not always be directly reflected in the similarity. Therefore, a segment which is a small section is formed to try to catch a partial similarity. Therefore, although the time length of each segment is arbitrary, it is preferable that it is the same as or comparable to the time length of the cutting section C at the time of operation.

  In the example of FIG. 17, at the time of learning, a super vector (SV) or i vector is created for each segment using the feature data belonging to each segment in the same manner as in FIG. 16, or in the case of FIG. Similarly to the above, a distribution and its representative value (representative vector) are created. Therefore, for the predictive time interval A1, the SV group, i vector group, distribution group, or representative value of each segment A1 (S1), A1 (S2), A1 (S3), A1 (S4),. A representative vector group is created. The same applies to the other sections A2 to A4 and AX.

  Further, the calculation of the SV and i vector of the cut section C during operation is the same as that in FIG. 16, and the distribution of the cut section C during operation or the calculation of the representative value (representative vector) is also illustrated in FIG. It is the same as the case of.

  The calculation of the similarity in the example of FIG. 17 is for the predictive time interval A1 between the vectors of the SV and i vectors in the cut-out interval C during operation and each of the SV groups and i vector groups in the interval A1. Similarity (for example, cosine similarity) is calculated, or the representative value (representative vector) of the distribution of the cut section C during operation and the representative value (representative vector) group of the distribution group of the section A1 Similarity between distributions (for example, KL distance) is calculated. The same applies to the other sections A2 to A4 and AX. Then, according to the K-NN method or the like, the SV or i vector of the cut section C at the time of operation or the representative value (representative vector) of the distribution is assigned to any of the sections A1 to A4, AX, or The degree is calculated. Or you may obtain | require by the process (algorithm) by the static pattern recognizer which does not have an internal memory. For example, all methods used for general identification problems, such as GMM, simple NN, and SVM, can be employed. To put it simply, there are several similar segments in the predictive time interval A1 with respect to the SV and i vector of the cut-out interval C during operation, or the representative value (representative vector) of the distribution, and within the predictive time interval A2. Includes information indicating that there are several similar segments, and that there are several similar segments in the predictive time interval AX, as a similarity (subscore). Then, if necessary, a conversion process from the sub-score is performed and a final score is output, or the obtained similarity (sub-score) is directly output as a final score.

  As described above, the predictive detection system and program according to the present invention include, for example, livestock delivery, suicide of prisoners in a cell, development of depression, occurrence of accidents and dangerous acts, collapse of a tunnel, flooding of a river, eruption of a volcano It is suitable for detecting signs of disasters such as theft, crimes such as theft and terrorist attacks.

DESCRIPTION OF SYMBOLS 10 Predictive detection system 40 Status data acquisition means 50 Predictive feature extractor 60 Predictive detector 70 Output means

Claims (7)

A sign detection system that detects signs that occur before an event occurs,
State data acquisition means for continuously or repeatedly acquiring data including perceptual data recognizable by human perception as state data indicating a state of a sign detection target that may cause the event;
Using the state data acquired by the state data acquisition means, the state of the detection target is determined in advance related to a sign including a specific state that is related to the sign and may appear at all times. Executes pattern recognition processing that outputs data indicating whether or not it corresponds to at least one specific state or the degree of similarity to the specific state, and repeatedly executes processing for extracting feature data obtained by this pattern recognition processing A predictive feature extractor,
A predictive detector for executing a time-series pattern recognition process using the time-series feature data extracted by the predictive feature extractor and outputting a score indicating the presence or absence or presence / absence of the predictor. Predictive detection system. The predictive feature extractor is
It is configured to execute pattern recognition processing by a neural network, and to execute processing for extracting output data of an intermediate layer that reaches at least one output node corresponding to the specific state as the feature data. The sign detection system according to claim 1. The sign detector is
The score corresponding to a plurality of predictive time sections set as sections in which a sign may appear by dividing a time zone before the occurrence of an event is configured to be output. The sign detection system described in 1. Prior information storage means for storing advance information that is predicted or scheduled in advance about the occurrence time of the event,
Combining the probability distribution of event occurrence created using the prior information stored in the prior information storage means and the probability distribution of event occurrence created using the score output from the sign detector By displaying the composite distribution or inputting the prior information stored in the prior information storage means together with the feature data from the predictive feature extractor into the predictive detector from the predictive detector The sign detection system according to claim 1, further comprising: an output unit that executes a process of displaying a composite distribution created using the output score. An output means for executing a process of displaying an event occurrence probability on a calendar using a score output from the predictive detector;
This output means is
When the time length does not match or the section is deviated between at least one of the predictive time intervals and the unit unit of the calendar, the predictive time interval is changed to the calendar. Is divided into corresponding unit intervals, and the probability distribution in the predictive time interval is regarded as a uniform distribution and assigned to the calendar unit interval by dividing the score corresponding to the predictive time interval, or a plurality of The sign detection system according to claim 3, wherein at least a part of each score corresponding to the sign time interval is assigned to the same unit interval of the calendar. An output means for executing a process of displaying a probability distribution of event occurrence or a cumulative distribution thereof using a score output from the sign detector;
This output means is
(1) When a plurality of the predictive time intervals are set without overlapping, the probability distribution of each of the multiple predictive time intervals is simulated using any of a plurality of types of distributions prepared in advance. The local gradient change of the probability distribution curve in a state in which the probability distributions of the plurality of predictive time intervals are combined or connected is minimized, or less than or less than a predetermined threshold. A process of selecting a distribution
(2) When a plurality of the predictive time sections are set to overlap, for a section where a plurality of the predictive time sections having different time lengths overlap, the score corresponding to the shortest predictive time section is used. Adopts a pseudo event distribution or cumulative distribution of event occurrences, and employs a probability distribution or cumulative distribution of event occurrences that are artificially created using scores corresponding to other predictive time intervals. Do not process,
(3) When a plurality of the predictive time sections are set to overlap, for a section in which a plurality of the predictive time sections having different time lengths overlap, the score corresponding to each of the predictive time sections is used. A process of synthesizing a probability distribution of event occurrence created in a pseudo manner or a cumulative distribution thereof at a predetermined ratio;
(4) When a plurality of the predictor time intervals are set in an overlapping manner, the probability distribution of each of the plurality of predictor time intervals or the cumulative distribution thereof is any one of a plurality of types of distributions prepared in advance. A process of selecting a distribution to be used for creating one's own probability distribution or its cumulative distribution using a score corresponding to the predictive time interval having a shorter time length than that of the self when using The sign detection system according to claim 3, wherein at least one of the processes is executed.   A program for causing a computer to function as the sign detection system according to claim 1.

Images