JP6717461B2 - Condition monitoring system - Google Patents

Description

The present invention relates to a state monitoring system for detecting a specific state corresponding to a monitoring purpose of a monitoring target, for example, in order to prevent damage to the crops and environmental destruction by wild birds and beasts, to a farm, ranch, or forest to be protected. When automatically detecting the invasion of wild birds and beasts of animals, when automatically detecting the invasion of wild animals into the feeding grounds and bedding of livestock, when learning and automatically detecting the behavior of livestock and pests, facilities such as homes and hospitals When monitoring the condition of elderly people and patients who need long-term care, monitoring the conditions around buildings for crime prevention, predicting various dangers in places where people gather, landslides caused by earthquakes and heavy rainfall It can be used for forecasting and early detection of various disasters such as avalanche, river flood caused by bank breakage, lightning strike, and fire.

In recent years, with the development of pattern recognition technology represented by image recognition technology, research and development of a state monitoring system having a function of automatically detecting a specific state from a camera image have been advanced.

Generally, in order to introduce a high-performance pattern recognition technology with few detection omissions (that is, high recall) and few false detections (that is, high accuracy) into an automatic monitoring system, the monitoring target is monitored. An enormous amount of data obtained by sensing with a sensor and an appropriate tag indicating its state must be given as a set.

Therefore, if a large amount of tagged data can be obtained in the actual environment where the monitoring sensor is actually used to monitor the status, it becomes possible to construct a high-performance pattern recognition device. It is possible to collect data, improve performance, and continue operation.

However, it is difficult to obtain a large amount of data in the real environment before starting the actual operation.Therefore, before starting the status monitoring service, collect the test data set under a test environment different from the real environment. In many cases, a pattern recognition device using an initial model created by using the data is constructed and service operation is started.

Note that, as a technique related to the state monitoring technique of the present invention, there are the following techniques, for example. First, as a related technique in that a plurality of types of sensors are used, an intelligent monitoring system is known that enables automatic notification by combining an image sensor and a radiation dose detection sensor (see Patent Document 1). .. This system enables automatic state detection by combining a visual sensor and an environmental sensor.

In addition, as a related technology in terms of crowdsourcing, a real-time integrated platform for service and analysis that takes the form of a cloud computing platform and enables quick and easy development, deployment and management of sensor-driven applications is known. (See Patent Document 2). This service and platform provides a mechanism that enables real-time development, testing and deployment of sensor-based applications to facilitate crowdsourcing application development, with user input data from crowdsourcing as data from software sensors. It is possible to process.

JP, 2013-164774, A Special table 2014-534487 gazette

However, as described above, when the pattern recognition device is constructed using the initial model created by the data collected under the test environment different from the real environment and the actual service is deployed, the pattern recognition device is not As a result of erroneous recognition caused by the difference in data from the environment or lack of data, the user (the person who receives the service provided by the condition monitoring system and the person who receives the information output from the condition monitoring system) We cannot guarantee the expected quality.

On the other hand, in order to enable the operation of a high-performance pattern recognition device immediately after the start of service, it is necessary to collect data in a wide variety of real environments, build models, and evaluate recognition results. A huge investment will be required to build a quality initial model.

It is also possible to use a sensor for a specific purpose separately to ensure the performance required by the user, but such a sensor is generally expensive, and in developing the business of status monitoring service, There is a cost problem.

In the first place, it is not easy to construct a highly accurate pattern recognition system in an automatic monitoring system that requires high recall. In a pattern recognition system, there is a trade-off relationship that if the recall is prioritized to be high, the accuracy is sacrificed, and the accuracy is lowered. Therefore, there is a problem that both cannot be pursued at the same time. is there.

For such problems, techniques such as integrating the recognition results of multiple pattern recognition systems with different performance may be used, but in general what kind of combination can solve the problem? There is no effective method. Therefore, in order to achieve high accuracy while maintaining high recall, it has conventionally been necessary for a system developer to select and decide various methods and their integration methods by trial and error.

Furthermore, various problems remain when a high-performance pattern recognition system is successfully developed and put into operation. For example, as the developed pattern recognition system is introduced in various environments, and as the environment of the installation destination changes over time after the installation, the actual environment of the installation destination of the pattern recognition system and its pattern recognition In some cases, the difference from the environment in which the data used to create the system usage model is collected will increase. In such a case, the quality required by the user cannot be guaranteed. This is because even if a high-performance pattern recognition system is developed by the conventional method, which requires time, effort, and cost, the pattern recognition system cannot continue to exhibit high performance in any environment. This is because it is not designed to be able to flexibly follow the introduction to the environment and subtle changes in the environment of the installation destination.

Therefore, in such cases, it is essential to update the model and algorithm at the installation destination, but in the first place, a general method for finding out which installation destination such a case or There is no mechanism. Further, when the performance of the model or algorithm is changed in a bad direction as a result of updating, the problem remains that the user is adversely affected.

An object of the present invention is to provide a condition monitoring system that can exhibit high performance, can be easily and inexpensively constructed, and can flexibly cope with introduction into various environments.

The present invention is a state monitoring system for detecting a specific state corresponding to a monitoring purpose of a monitoring target,
A monitoring sensor that detects the status of the monitoring target,
Provided in order to obtain perceptual data recognizable by human perception of the state of the monitored object, separate from the monitoring sensor, or a sensory data acquisition means also serving as the monitoring sensor,
A pattern recognition means for executing a pattern recognition process for determining whether or not the state of the monitoring target corresponds to a specific state by using the output data of the monitoring sensor, or which of a plurality of specific states ,
When the reliability of the recognition result determined by using the likelihood output from the pattern recognition means is equal to or less than or equal to a predetermined threshold value, the perceptual data acquired by the perceptual data acquisition means or the processed data is used. , The state of the monitoring target that is transmitted to the verification terminal via the network as verification data for verifying the correctness of the judgment by the pattern recognition means, and is judged by the operator of the verification terminal using the verification data. Whether or not corresponds to a specific state, or to which of a plurality of specific states, the data of the result of the manual judgment is received from the verification terminal via the network, and the received manual judgment is made. Verification means for executing verification processing for verifying the correctness of the judgment by the pattern recognition means using the resulting data,
When the verification processing by this verification means is performed, the judgment result by hand is adopted, and when the verification processing is not carried out, the judgment result by the pattern recognition means is adopted, and the data of the adopted judgment result is converted into the network. To the cooperative system via the display, screen display on the display device, print on the printing device, voice output, or output a notification sound or notification light according to the judgment result adopted And output means.

In such a state monitoring system of the present invention, if the reliability of the recognition result determined by using the likelihood output from the pattern recognition means is less than or less than the threshold value, the verification means processes the perceptual data or this. Inquiry processing is performed to transmit the verified data as verification data to the verification terminal, and a manual judgment result is obtained. When this verification process (inquiry process) is performed, the determination result by hand is used as the system output instead of the determination result by the pattern recognition.

Therefore, at the initial stage of introducing the condition monitoring system into a certain environment, it is not clear how well the pattern recognition means by the initial model created using the data collected in the test environment suits the environment of the installation destination, Although the reliability value of the recognition result by the pattern recognition means becomes small, in that case, the inquiry processing to the verification terminal is performed, and the judgment result is corrected to the correct result by the manual judgment result. It is possible to improve the quality of the information (judgment result about the state of the monitoring target) output by the output unit. Therefore, even if the recall is prioritized, the accuracy is not sacrificed, the problem of trade-off is solved, and a high-performance condition monitoring system that can always satisfy the quality required by the system user is provided. Can be realized.

Also, after the introduction, time passes, the cumulative number of pattern recognition processes increases, and the verification means repeats the correctness verification process of the judgment by the pattern recognition and accumulates the verification results. Since the degree of conformity of the pattern recognition method to the actual environment becomes clear, the reliability of the recognition result increases, and as a result, the frequency of inquiring processing to the verification terminal decreases. Come on. Therefore, over time, the number of cases in which system output is performed by the output means will increase without relying on the human judgment results received from the verification terminal.Therefore, reduce the time required from event occurrence to output. Is possible. Therefore, a state monitoring system that can flexibly respond to the environment of the installation destination regardless of what environment it is installed and can finally output an accurate judgment result in a short time is realized. It becomes possible.

Further, since the verification means is provided, the reliability of the recognition result by the pattern recognition means at the beginning of introduction may be low, so an expensive monitoring sensor and pattern recognition means capable of exhibiting high performance from the beginning of introduction can be used. It is not necessary to prepare it, and when creating the initial model, it is not always necessary to collect data in the actual environment or an environment very close to it, so it is possible to reduce the investment in the initial model and operate the condition monitoring service. It is possible to accelerate the start time. For this reason, it becomes possible to easily and inexpensively construct a system, thereby achieving the above-mentioned object.

Although the intelligent monitoring system described in Patent Document 1 described above combines an image sensor and a radiation dose detection sensor, it does not include a device equivalent to the verification means of the present invention, so event detection is performed. We cannot solve the problem of misrecognition of time.

Further, although the service and the real-time integrated platform for analysis described in Patent Document 2 described above perform crowdsourcing, it is provided with a mechanism for verifying the recognition result of the pattern recognition means by crowdsourcing as in the present invention. Do not mean.

<Configuration for updating the model>

Also, in the status monitoring system described above,
Model storing means for storing a model used in the pattern recognition means,
Learning data storage means for storing the output data of the monitoring sensor in association with the data of the judgment result adopted by the output means,
A learning unit that repeatedly creates a model using the output data of the monitoring sensor stored in the learning data storage unit and executes a process of updating the model stored in the model storage unit,
The pattern recognition means
It is desirable that the latest model stored in the model storage means is used to execute the pattern recognition processing.

When the model is updated by using the output data of the monitoring sensor stored in the learning data storage unit as described above, the performance of the pattern recognition unit can be improved. Because the output data of the monitoring sensor stored in the learning data storage means is the data collected in the actual environment while operating the system, and the output data of these monitoring sensors are stored together with the tag information. The tag information is the data of the judgment result adopted by the output means, and the verification result by the verification means is utilized, so the accuracy is guaranteed by the learning means because it is the tag information whose accuracy is guaranteed. This is because proper learning processing using the learned learning data is automatically executed. Therefore, the reliability of the recognition result by the pattern recognition means is further increased, and as a result, the frequency of performing inquiry processing to the verification terminal is further reduced, and the time required from event occurrence to output is reduced. It is possible to make it even shorter.

Further, when the model is updated by making use of the verification result by the verification means in this way, the reliability of the recognition result increases, but there are the following two cases. First, there is a case where the reliability value of the recognition result is expected to increase only after the model is updated by utilizing the verification result. Next, even if the model update is not performed, the reliability value of the recognition result is expected to increase due to the accumulation of the verification result, but the effect may be more remarkable by performing the model update that makes use of the verification result. is there.

The intelligent monitoring system described in Patent Document 1 described above does not have a mechanism for automatically collecting and learning data.

<Structure with multiple monitoring sensors>

Furthermore, in the above-mentioned condition monitoring system,
Equipped with multiple monitoring sensors,
The pattern recognition means
It is configured to include a plurality of pattern recognizers corresponding to a plurality of monitoring sensors,
The model storage means
It is configured to store multiple models corresponding to multiple pattern recognizers,
The learning data storage means is
Each of the output data of the plurality of monitoring sensors is configured to be stored in association with the data of the determination result adopted by the output means,
The learning method is
A configuration for repeatedly creating each of a plurality of models using each of the output data of the plurality of monitoring sensors stored in the learning data storage means and executing a process of updating the plurality of models stored in the model storage means, Is
The verification means is
When the highest reliability among the reliability of the recognition result determined by using the likelihood output from each of the plurality of pattern recognizers is less than or less than the threshold value, the verification data is transmitted to the verification terminal. At the same time, it is configured to execute a verification process of receiving the data of the judgment result by hand from the verification terminal,
The output means is
If the verification process is performed by the verification means, the judgment result by human is adopted, and if the verification process is not performed, the judgment result by the pattern recognizer with the highest reliability value among the plurality of pattern recognizers. It is desirable that the configuration is such that processing that adopts is executed.

Here, "multiple monitoring sensors" refers to multiple types of monitoring sensors, multiple monitoring sensors of the same type that are installed in different locations or in different states, and combinations of these. Is included.

When outputs from a plurality of monitoring sensors are input to one pattern recognizer and one recognition result is obtained, those plurality of monitoring sensors are regarded as one monitoring sensor in the present invention. Further, when one recognition result is obtained by inputting outputs from a plurality of monitoring sensors to a plurality of pattern recognizers and integrating outputs from the plurality of pattern recognizers, a plurality of those recognition results are obtained. The monitoring sensor and the plurality of pattern recognizers are regarded as one monitoring sensor and one pattern recognizer in the present invention. In short, by considering the minimum unit that finally generates the information (judgment result about the state of the monitoring target) output from the output means as a set of one monitoring sensor and one pattern recognizer, the monitoring sensor Is the unit of attachment/detachment, and for the pattern recognizer, it is the unit of selection of use/non-use.

In the case of the configuration including a plurality of monitoring sensors in this way, the reliability of the largest value among the reliability of the recognition result determined by using the likelihood output from each of the plurality of pattern recognizers is the threshold value. When the value is less than or equal to or less than, since the inquiry to the verification terminal is performed, the reliability with the largest value exceeds the threshold value or is greater than or equal to the threshold value, the inquiry to the verification terminal is not performed. Whether or not to make an inquiry to the verification terminal depends on the content of the output of the pattern recognizer having the highest performance among the plurality of pattern recognizers.

Generally, a pattern recognizer (more accurately, a combination of a monitoring sensor and a pattern recognizer, but here, a pattern recognizer will be representatively described) exhibits high performance from the beginning, but , A pattern recognizer that has the characteristic that the performance does not improve much even if learning is repeated, or a pattern recognizer that has the characteristic that the performance rises rapidly at the beginning, but the performance suddenly increases as the learning is repeated. There is a pattern recognizer having various characteristics such as a detector (see FIG. 7 described later). Therefore, depending on the combination of the characteristics of the pattern recognizer, the pattern recognizer that exhibits the highest performance may be replaced over time.Therefore, the pattern that determines whether to perform an inquiry process to the verification terminal The recognizer will be replaced.

Therefore, by combining pattern recognizers with different characteristics, it is possible to continue to exhibit high performance, reduce the frequency of inquiries to the verification terminal, and shorten the time required from event occurrence to output. Becomes

Further, as described above, the pattern recognizer that exhibits the highest performance may be replaced, so after the replacement, the monitoring sensor with lower performance was removed and the output data of that monitoring sensor was processed. It is also possible to stop using the pattern recognizer. Therefore, for example, immediately after the operation of the system is started, an expensive monitoring sensor is made to function to ensure the quality, and then, when the performance of the pattern recognizer by the inexpensive monitoring sensor is improved, the expensive monitoring sensor is removed and another It is possible to supply to the introduction destination of. Further, the pattern recognizer and the monitoring sensor having different characteristics can be added and operated later.

<Structure that supports the decision to remove the monitoring sensor>

Then, as described above, in the case of the configuration including a plurality of monitoring sensors,
Verification result storage means for accumulating and storing the verification results of the plurality of pattern recognizers by the verification means,
Using the verification results accumulated and stored for each pattern recognizer in the verification result storage means, performance calculation means for executing a process of calculating performance for each of the plurality of pattern recognizers,
The performance of the plurality of pattern recognizers calculated by the performance calculation means, or in addition to these performances, an output log indicating which judgment result is adopted by the output means stored in the output log storage means is stored in the system. System management that executes the performance display process that displays on the administrator terminal screen, or executes the recognizer necessity reception process that accepts whether to use each pattern recognizer in addition to this performance display process It is desirable to have a structure including means.

If the system management means is provided when multiple monitoring sensors are installed in this way, the system administrator should refer to the performance and output logs of multiple pattern recognizers on the system administrator terminal. It becomes possible to check the current status (the latest status at each point in time), while performance becomes relatively low, and it is possible to judge whether to remove the unnecessary monitoring sensor or not. It is possible to determine whether or not to stop using the pattern recognizer that has been processing the output data.

<Structure for supporting the decision to remove the sensory data acquisition means>

Also, in the status monitoring system described above,
Verification result storage means for accumulating and storing verification results by the verification means,
A performance calculation unit that executes a process of calculating the performance of the pattern recognition unit using the verification results accumulated and stored in the verification result storage unit;
The performance of the pattern recognition means calculated by this performance calculation means, or in addition to this performance, an output log indicating which judgment result is adopted by the output means stored in the output log storage means is displayed as a system administrator terminal. Or a system management unit that executes the performance display process for displaying on the screen, or in addition to this performance display process, executes the verification necessity reception process that receives the selection of the necessity of the verification process by the verification unit. Is desirable.

When the system management means is provided in this way, the system administrator can refer to the performance of the pattern recognition means and the output log at the system administrator terminal. Whether the perceptual data acquisition means may be removed while checking the latest situation at each point of time, that is, whether the verification means does not have to perform an inquiry process (verification process) to the verification terminal. Can be determined. It should be noted that the determination of the necessity of the inquiry process to the verification terminal by the verification means here is a determination as to whether or not the inquiry process to the verification terminal does not have to be performed until the end. It is not a judgment as to whether or not the inquiry processing is required every time by comparing the reliability of the recognition result determined by using the likelihood output from the means with the threshold value thereof.

As described above, according to the present invention, when the reliability of the recognition result determined by using the likelihood output from the pattern recognition means is less than or less than the threshold value, the verification means processes the perceptual data or this. This data is sent to the verification terminal as verification data, the result of manual judgment is obtained, and this is used as the system output instead of the result of judgment by pattern recognition, so it is possible to demonstrate high performance from the beginning. There is an effect that a monitoring system can be easily and inexpensively constructed, and a state monitoring system capable of flexibly adapting to introduction into various environments can be realized.

The whole block diagram of the state monitoring system of one embodiment of the present invention. The figure which shows an example of the database structure of the state monitoring system of the said embodiment. The figure of the flow chart which shows the flow of the state monitoring processing by the state monitoring system of the above-mentioned embodiment. The figure which shows an example of the verification screen displayed on the verification terminal of the said embodiment. The figure which shows another example of the verification screen displayed on the verification terminal of the said embodiment. The figure which shows an example of the management screen displayed on the system administrator terminal of the said embodiment. Explanatory drawing which shows the change of the performance of each pattern recognizer of the state monitoring system of the said embodiment.

An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 shows the overall configuration of a state monitoring system 10 of this embodiment. FIG. 2 shows the configuration of the database 72 that stores verification results, learning data, and the like. FIG. 3 is a flowchart showing the flow of the state monitoring process by the state monitoring system 10. 4 and 5 show examples of the verification screens 200 and 300 displayed on the verification terminal 100, and FIG. 6 shows the management screen 400 displayed on the system administrator terminal 120. An example is shown. Further, FIG. 7 is an explanatory diagram showing that the performances of the pattern recognizers A and B included in the pattern recognition means 51 change with the passage of time.

<Overall configuration of status monitoring system 10>

In FIG. 1, the state monitoring system 10 includes a sensory data acquisition unit 20 provided to acquire sensory data recognizable by human perception of a state of a monitoring target, and a monitoring sensor 30 (for detecting a state of the monitoring target). In the present embodiment, as an example, a plurality of monitoring sensors 30A, 30B, and 30C), and one or a plurality of computers (servers) that process the outputs of the perceptual data acquisition unit 20 and the monitoring sensor 30. Is configured.

In the present embodiment, as an example, the parts that process the output of the perceptual data acquisition unit 20 and the monitoring sensor 30 are respectively a connection device 40 that is a separate computer, a pattern recognition device 50, a verification device 60, and a data collection device. 70, the output device 80, and the update device 90 are connected by the network 1. However, the state monitoring system 10 does not necessarily have to be configured by a plurality of computers in this way, and the functions realized by the devices 40 to 90 may be configured in a single computer, or The functions may be distributed to any number of computers. For example, the pattern recognition device 50 and the verification device 60 may be configured by one computer, the data collection device 70 and the update device 90 may be configured by one computer, or the output device 80 may be used. The functions to be realized may be distributed to two (plural) computers, and the functions to be realized by the update device 90 may be distributed to two (plural) computers.

Here, the network 1 is mainly an external network such as the Internet, but it may be a combination of this and an internal network such as an intranet or a LAN, and may be wired, wireless, or a mixed wired/wireless type. It doesn't matter.

Further, in the network 1, a plurality (a large number) of verification terminals 100 operated by a crowd and one or a plurality of terminals managed by a collaborator who uses an output from the status monitoring system 10 (judgment result about the status of the monitoring target). The linked system 110 and the one or more system administrator terminals 120 operated by the system administrator who manages the state monitoring system 10 are connected. In FIG. 1, the verification terminal 100, the cooperation system 110, and the system administrator terminal 120 are described as being placed outside the state monitoring system 10, but a general-purpose browser or general-purpose software is used. However, if dedicated software is installed in these and used, these may be included in a part of the configuration of the condition monitoring system 10.

<Monitoring target, monitoring purpose, and “specific state” corresponding to the monitoring purpose>

Examples of the “specific state” corresponding to the monitoring target, the monitoring purpose, and the monitoring purpose by the state monitoring system 10 include the following. First, the damage to crops by wild birds and beasts amounted to tens of billions of yen, and since the damage has not decreased, a solution is needed. In addition, there is a similar damage in forestry, and the area of forest damage caused by wild birds and animals such as deer and bears reaches about 9000 hectares nationwide, of which about 80% is the damage caused by deer foliage damage and peeling. The situation is serious. These damages are not limited to economic damages, and may lead to a decline in willingness to manage farming and forestry, an increase in abandoned cultivated land, and a loss of biodiversity in forests and soil runoff. Therefore, the monitoring target in this case is wild birds and beasts such as deer, boars, and monkeys that invade the area to be protected, and the purpose of monitoring is to prevent or suppress damage by these wild birds and beasts or grasp the behavior of wild birds and beasts for that purpose. The “specific state” corresponding to the monitoring purpose is, for example, a state in which wild birds and beasts are attacking the fence partitioning the protected area with a body attack, a state of jumping over the fence, or the like.

Next, there is food damage (damage of feed etc. given to livestock). For example, there is plagiarism of the compound feed in the barn by raccoon dogs, raccoons, cabbage, badgers and the like. In addition, there are feeding and lodging damages of forage crops such as raccoon dogs, raccoons, palm civets, badgers, and also large bears and wild boars, deer, and dent corn from monkeys. Furthermore, damage to grazing land and grassland includes damage from Hokkaido deer, and in Honshu, damage to grazing land and grassland due to sika deer is increasing. In addition, deer, foxes, and raccoons smash the grass wrap after harvesting grass, degrading grass quality. In addition, there is a problem that the feeding of grass by the boar is not noticed and it is mistaken for the poor growth of grass due to a decrease in soil fertility. In addition, there are hygiene problems. Dispersion of pathogens from the outside due to invasion, stealing, and excretion of wild animals into the livestock shelter will affect the health of livestock. Also, in winter, wild animals invade the composting houses in search of an environment where they can easily live, so that the spread of pathogens from the outside also occurs. If left unattended or stolen by such wild animals, agricultural crops, mixed feeds, forage crops (especially cold-season grasses), etc., will be important nutrients that lead to life when there is no food in the winter. It reduces the mortality rate of wild animals, which in turn leads to an increase in the number of wild animals, which creates a vicious cycle of even more damage. Therefore, the target of monitoring in this case is wild animals such as raccoon dogs, raccoons, palm civets, badgers, bears, boars, deer, monkeys, etc. that invade, steal, excrete, etc. in the livestock shed, and/or originally in the livestock shed. Livestock, or places where these wild animals and livestock appear, such as feeding grounds and bedding in livestock shelters.The purpose of monitoring is to prevent or suppress the invasion, stealing, and excretion of wild animals into livestock sheds. For that purpose, the behavior of wild animals and livestock is grasped, and the "specific state" corresponding to the monitoring purpose is, for example, the state where the wild animals are stealing, the state where the livestock are eating properly, the wild animals that have invaded. Are sleeping, livestock are sleeping properly, etc.

In addition, there is also condition monitoring of humans such as the elderly requiring care, children and babies, or suspicious persons. Whether the elderly requiring nursing care are bathing normally, the child is eating dangerous food, playing dangerously, the baby is not lying on his/her face, whether a suspicious person is prowling around the building, etc. Need to be monitored. In this case, the monitoring target is an elderly person requiring nursing care, a child or baby, a suspicious person, etc., and the monitoring purpose is prediction or prevention of danger or crime, early detection, etc. The state is, for example, a state in which the elderly requiring care fall down in the bathroom, a state in which the child reaches for something that cannot be eaten, a state in which the baby lies on his/her face, a state in which a suspicious person is prowling around the building. Etc.

In addition to the above, for example, prediction and early detection of various disasters such as a landslide, an avalanche, a river inundation due to an embankment failure, a lightning strike, and a fire may be the monitoring purpose, for example, due to an earthquake or heavy rainfall. In this case, the targets to be monitored are places, objects, and spaces that can detect the precursors of disasters such as landslides, avalanches, river floods, lightning strikes, and fires. Is the state where pebbles flow downward, the state where thunderclouds are occurring, the state where smoke is emitted, etc. Therefore, if the state changes and the change has some effect, it can be the target of the state monitoring according to the present invention.

<Sensing and transmission of its output>

The perceptual data acquisition means 20 acquires perceptual data for creating the verification data necessary for the crowd (humans) to judge the state of the monitoring target in the verification terminal 100. Perceptual data is sensing data that can be captured by the human five senses (visual sense, auditory sense, olfactory sense, taste sense, and tactile sense). This is the data used for the judgment work of the crowd (human beings) in the verification terminal 100, so in the currently prevailing situation of the verification terminal 100, the image data of the camera that can be visually captured. (A moving image or a still image may be used.), sound data of a microphone that can be auditorily captured, or a combination thereof is the mainstream. However, the verification terminal 100 that can reproduce the olfaction, taste, and tactile sense is provided. When it has spread, it may be data of olfaction, taste, and touch.

Further, the perceptual data acquisition unit 20 may be a device physically different from the monitoring sensor 30 or may be a combined device. When the perceptual data acquisition means 20 is also used as the monitoring sensor 30, the output data of the monitoring sensor 30 is transmitted to the pattern recognition device 50 and input to the pattern recognition means 51, and the same output data is sent to the perceptual data acquisition means. This is a case where the perceptual data acquired by 20 is also transmitted to the verification device 60 and passed to the verification means 61. That is, the output data of the monitoring sensor 30 is used as the pattern recognition data and also as the verification data.

The monitoring sensor 30 is, for example, a camera that captures an image (whether it is a moving image or a still image), a microphone that picks up sound, an acceleration sensor that measures the acceleration of the installation location (object), and an object that is irradiated with laser light and reflected by the sensor. A range sensor that captures an object, an IoT tag, a positioning sensor such as GPS, a gyro sensor that detects an inclination, a magnetic sensor, a marker for mounting an object that constitutes a motion capture, and a tracker that detects this movement, a temperature sensor, a humidity sensor, and smoke. Examples are sensors and odor sensors.

In the present embodiment, as an example, the case where three monitoring sensors 30A, 30B, and 30C are installed is shown, but the number of monitoring sensors 30 installed is arbitrary and may be one. However, in order to improve the performance of the pattern recognition means 51 in the entire system, it is preferable to install a plurality of monitoring sensors 30, particularly a plurality of types of monitoring sensors 30 having different characteristics. Moreover, the three monitoring sensors 30A, 30B, and 30C are not limited to the case where all three are installed from the beginning, and include, for example, the case where the third monitoring sensor 30C is additionally installed afterwards. Has become. Furthermore, when the third monitoring sensor 30C is additionally installed, this is also an example including a case where the first monitoring sensor 30A has already been removed.

Further, when the output data of the plurality of monitoring sensors is input to one pattern recognizer and one recognition result is obtained, those plurality of monitoring sensors are regarded as one monitoring sensor 30 in the present invention. Also, when a single recognition result is obtained by integrating the outputs of a plurality of pattern recognizers, a group of monitoring sensors that send data to the plurality of pattern recognizers are regarded as one monitoring sensor 30. Therefore, each of the monitoring sensors 30A, 30B, 30C may be composed of a plurality of monitoring sensors.

The connection device 40 receives each output data of the monitoring sensors 30A, 30B, 30C, and each of the received output data is monitoring sensor identification information for identifying which one of the monitoring sensors 30A, 30B, 30C. Pattern recognition via the network 1 together with tag information indicating what state (specific state) is detected by the monitoring sensors 30A, 30B, 30C, and event identification information automatically assigned to identify the event that has occurred The processing for transmitting to the pattern recognition means 51 of the device 50 is executed.

Further, the connection device 40 receives the perceptual data output from the perceptual data acquisition unit 20, and what state (specific state) the received perceptual data is detected by each of the monitoring sensors 30A, 30B, 30C. The tag information indicating what is to be verified with the acquired perceptual data and the event identification information are then transmitted to the verification means 61 of the verification device 60 via the network 1 or the pattern recognition of the pattern recognition device 50. It also executes the process of transmitting to the means 51. When transmitted to the pattern recognition means 51 of the pattern recognition device 50 like the latter, the perceptual data, the tag information, and the event identification information are transmitted to the verification means 61 via the pattern recognition means 51.

Further, the connection device 40 converts each output data of the monitoring sensors 30A, 30B, 30C and the perceptual data output from the perceptual data acquisition means 20 into a data format for transmitting to the network 1 and a communication method. It also plays the role of a gateway.

If there is only one specific state detected by each monitoring sensor 30A, 30B, 30C, the transmission of the tag information from the connection device 40 to the pattern recognition means 51 and the verification means 61 may be omitted.

The data transmission from the monitoring sensors 30A, 30B, 30C and the perceptual data acquisition means 20 to the connection device 40 may be wired transmission or wireless transmission.

Further, the data transmission from the connection device 40 to the pattern recognition means 51 and the verification means 61 (transmission of each output data of the monitoring sensors 30A, 30B, 30C and the perceptual data output from the perceptual data acquisition means 20) causes an event occurrence. This is performed at the time, that is, when the data transmission is triggered based on the outputs of the monitoring sensors 30A, 30B, and 30C (when the change in the state is captured). However, the process of capturing the event occurrence may be performed by each of the monitoring sensors 30A, 30B, 30C, or may be performed by the connection device 40 that continuously receives each output data of the monitoring sensors 30A, 30B, 30C. ..

<Pattern recognition processing>

The pattern recognition device 50 is configured to include a pattern recognition unit 51 that executes an algorithm of pattern recognition processing, and a model storage unit 52 that stores a model (parameter) used by the pattern recognition unit 51.

In the present embodiment, the pattern recognition means 51 is configured to include a plurality of (three) pattern recognizers A, B, and C as an example. These pattern recognizers A, B, and C perform pattern recognition processing by different algorithms. For example, when the types of the monitoring sensors 30A, 30B, and 30C are different, pattern recognition is performed. The same algorithm may be used among the containers A, B, and C. The respective judgment results by the pattern recognizers A, B and C are transmitted to the data collection device 70 via the network 1 by the pattern recognition means 51 and stored in the database 72 (see FIG. 2), or The data is transmitted to the verification device 60 and stored in the database 72 by the verification means 61 together with each verification result and the like.

Here, as the combination of the monitoring sensors 30A, 30B, 30C and the pattern recognizers A, B, C, various combinations can be adopted according to the monitoring target and the monitoring purpose. For example, when the sound data from the monitoring sensor 30 is input, an HMM (Hidden Markov Model) or a linear discriminator can be used as the pattern recognizer. When inputting the image data from the monitoring sensor 30, , Linear discriminator, DNN (deep neural network), etc. can be adopted. When inputting the output data of the acceleration sensor, HMM etc. can be adopted and the output data of the range sensor is input. In that case, a linear discriminator or the like can be adopted, and when inputting image data and sound data from the monitoring sensor 30, DNN or the like can be adopted.

In the adoption, for example, as shown in FIG. 7, the combination of the monitoring sensor 30A and the pattern recognizer A, which is relatively expensive at the beginning but has a rapid rise in performance from the beginning, is effectively operated (that is, The necessity of making an inquiry to the verification terminal 100 is determined mainly based on the reliability of the recognition result of the pattern recognizer A.) After that, the performance is low at the beginning of the introduction, but the performance increases with the passage of time. Switching to a relatively inexpensive combination of the monitoring sensor 30B and the pattern recognizer B, and if necessary, installing a combination of the monitoring sensor 30C and the pattern recognizer C having different characteristics (perceptual data acquisition means). If it is also used as 20, the installation is performed from the beginning of the introduction, but if not, additional installation may be performed later.) A plan such as switching the monitoring sensor 30 to an inexpensive one It is preferable to set a target and adopt it.

More specifically, for example, the monitoring sensor 30A is a range sensor, the pattern recognizer A is a linear discriminator, the monitoring sensor 30B is an acceleration sensor, the pattern recognizer B is an HMM, and the monitoring sensor 30C is image data acquisition. Camera (may also be used as the perceptual data acquisition means 20) and the pattern recognizer C is a DNN.

Further, as a specific example of the monitoring target or the monitoring purpose, when automatically detecting the invasion of wild birds and animals into the farm, ranch, or forest to be protected, the monitoring sensors 30A and 30B are fences that partition the area to be protected. In addition to installing a range sensor that can detect whether or not you have jumped over, install an acceleration sensor on the fence so that you can detect a body contact attack on the fence. At the same time, a camera, a microphone, or the like is installed as the perceptual data acquisition unit 20 or as the monitoring sensor 30C that also serves as the perceptual data acquisition unit 20. Then, from the output data of the acceleration sensor installed on the fence, the type and attack method of the wild birds and beasts that invade the protected area are automatically learned, and the wind and natural vibrations are also learned. This makes it possible to automatically detect an attack or preparation for an attack by a wild bird or beast. Further, in the future, advanced detection can be performed only with an inexpensive camera and an acceleration sensor, or the pattern recognition based on the output data of the acceleration sensor can be performed so that the verification process by the inquiry to the verification terminal 100 becomes unnecessary. If the performance is improved, it is possible to perform advanced detection only with the acceleration sensor.

Further, as another specific example of the monitoring target and the monitoring purpose, when monitoring a feeding place or a bed of livestock, a range sensor for detecting movement of livestock or wildlife is installed as the monitoring sensors 30A and 30B. At the same time, an IoT tag (with an acceleration sensor) is attached to livestock. At the same time, a camera is installed as the perceptual data acquisition unit 20 or as the monitoring sensor 30C that also serves as the perceptual data acquisition unit 20. As a result, it is possible to automatically detect the invasion of wild animals into the feeding area or the bed. Also, the behavior of livestock can be learned and automatically detected. That is, the abnormal state of the livestock can be automatically detected by learning that the livestock is eating properly or by learning that the livestock is sleeping properly. Furthermore, it is possible to distinguish between the state in which the livestock are eating and the state in which the wild pests are eating. By attaching the IoT tag (with microphone or camera, GPS, and acceleration sensor) to the wild pests, the behaviors of the wild pests can be learned and automatically detected. Will be able to predict the attack in advance.

The output data of the physically same monitoring sensor 30 may be input to the pattern recognizers of different algorithms. For example, the image data output from the physically same monitoring sensor 30 may be input to both the linear discriminator and the DNN. As a result, even when the output data from the same monitoring sensor 30 is used physically, the performance of both pattern recognizers will be different if the characteristics of the algorithm of the pattern recognizers are different. Have different values, and substantially, it is possible to switch from the linear discriminator to the DNN or from the DNN to the linear discriminator. That is, with respect to the reliability of the recognition result that determines the necessity of the verification process based on the inquiry to the verification terminal 100, the pattern recognizer that outputs the larger value, between the linear discriminator and the DNN over time. It may be replaced. In such a case, although the monitoring sensor 30 is physically the same, in the present invention, the combination of the monitoring sensor 30 that outputs image data and the linear discriminator, and another monitoring sensor 30 and DNN that outputs image data are used. It is assumed that there is a combination with. However, since the monitoring sensors 30 are physically the same, the monitoring sensor 30 cannot be removed unless there is a combination of another type of monitoring sensor and pattern recognizer.

In the present embodiment, as an example, the model storage means 52 is associated with a plurality (three) of pattern recognizers A, B, and C, and a plurality (three) of models α, β, which are used in these patterns are recognized. Remember γ. Although the algorithms of the pattern recognizers A, B, and C do not change, the models α, β, and γ are updated, so that the performances of the pattern recognizers A, B, and C increase with the passage of time. ..

<Verification process of correctness of judgment by pattern recognition>

The verification device 60 stores the verification means 61 for executing a process of verifying whether the pattern recognizers A, B, and C of the pattern recognition means 51 are correct, the verification data, and the comparison data for comparison. The verification data storage means 62 is provided. In the present embodiment, the verification device 60 functions as a server when the verification terminal 100 operated by the crowd is used as a client.

Here, the verification data stored in the verification data storage unit 62 is the perceptual data acquired by the perceptual data acquisition unit 20 or the data obtained by processing the perceptual data, and the comparison data is the state of the monitoring target. Is a data that shows a case where is really in a specific state, and both the verification data and the comparison data are data that can be recognized by human perception and are materials for microtask automatic generation. The former verification data is data that is temporarily stored so that the operator (crowd) of the verification terminal 100 can download it to his or her terminal, and is stored in association with the event identification information and the tag information. There is. The latter comparison data is data prepared in advance by the system administrator (may be data provided by a collaborator who manages the collaboration system 110), is stored in association with the tag information, and is stored in a plurality of specific data. When there is a state, a plurality of comparison data are prepared. The latter comparison data may be pseudo data such as synthetic image data or picture drawing data.

The verifying means 61 determines whether the highest reliability among the reliability of the recognition result determined by using the likelihoods output from the pattern recognizers A, B, and C is less than or less than or equal to a predetermined threshold. Transmits the verification data (the perceptual data acquired by the perceptual data acquisition unit 20 or the processed data thereof) and the comparison data stored in the verification data storage unit 62 to the verification terminal 100 via the network 1. In addition, whether the state of the monitoring target determined by the operator (crowd) of the verification terminal 100 by comparing the verification data and the comparison data corresponds to a specific state, or whether a plurality of specific states Data of the result of the manual judgment as to which one of them is received from the verification terminal 100 via the network 1, and each pattern recognizer A, B, C is used by using the received data of the manual judgment result. The process of verifying the correctness of the determination by the above, and storing the verification result and the like in the database 72 (see FIG. 2) of the data collection device 70 is executed.

Here, the threshold value of the reliability of the recognition result may be described in the program forming the verification unit 61, or stored in the threshold value storage unit (not shown) installed in the verification device 60 or another device. You may keep it. In the latter case, the system administrator inputs and sets a threshold value on the management screen 400 (see FIG. 6) displayed on the system administrator terminal 120, and the system management means 82 stores it in the threshold storage means. Can be

More specifically, first, the verification unit 61 receives the likelihood output from each of the pattern recognizers A, B, and C from the pattern recognition unit 51 via the network 1, and for each received likelihood. , The verification result of each pattern recognizer A, B, C stored in the verification result storage means 73 (see FIG. 2) of the data collection device 70 (correction information about the recognition result by each pattern recognizer A, B, C). ) Is directly or indirectly used to perform a correction process to calculate the reliability of the recognition result by the pattern recognizers A, B, and C.

At this time, the correction process for calculating the “reliability of the recognition result” from the “likelihood” is performed by, for example, a function of the reliability of the recognition result=f (likelihood, verification result) (however, the verification result is an indirect input. It may be an element). Since this correction processing is performed for each pattern recognizer A, B, C, the reliability of the recognition result for each pattern recognizer A, B, C is calculated. Further, when there are a plurality of specific states judged by one monitoring sensor and one pattern recognizer corresponding thereto (state X, state Y, state Z,... ), a plurality of likelihoods from one pattern recognizer. Since the degree is output, the function f becomes the reliability of the recognition result=f (likelihood for state X, likelihood for state Y, likelihood for state Z,..., Verification result).

The function f is, for example, [1] a function that outputs the highest value of the class posterior probabilities calculated from the likelihoods of a plurality of classes (if this value is high, the reliability is high), [2] a plurality of A function that calculates the reciprocal of entropy or a negative value of the recognition result from the likelihood of the class (if this value is high, the reliability is considered high), [3] Is the recognition result reliable using the likelihood pattern of multiple classes? A pattern recognition device that determines whether or not there is a reliability (the reliability is the reliability. It is learned by using the likelihood patterns of a plurality of classes for the data used for verification and the verification results), [4] the likelihood pattern of a plurality of classes Associative memory device that determines how far the recognition result is from the reliable state using (the reciprocal of the error from the reliable state obtained from the associative memory device or a negative value is the reliability. Data used for verification The associative memory device is constructed by using the likelihood patterns of a plurality of classes with respect to the data that was correct.

Among these, when the functions [1] and [2] are adopted, the models α, β, and γ that utilize the verification result by the verification unit 61 are updated, and the reliability value of the recognition result increases. Expected to go. Further, when the functions [3] and [4] are adopted, the verification result may be utilized and the reliability value of the recognition result may be increased without updating the models α, β, and γ. In addition, it is expected that the effects will become more remarkable by updating the models α, β, and γ using the verification results.

Next, the verification unit 61 performs a process of comparing the reliability of the recognition results of the pattern recognizers A, B, and C with the threshold value. In this case, the reliability of the plurality of recognition results has the highest value. The large reliability of is compared with the threshold. Then, the verification unit 61 causes the verification terminal 100 to compare the verification data with the verification data when the highest reliability among the reliability of the recognition results of the pattern recognizers A, B, and C is less than or equal to the threshold value. A verification process (inquiry process that asks the crowd to make a decision) is performed to transmit the usage data. Therefore, when the reliability of the largest value among the reliability of the plurality of recognition results exceeds the threshold value or is equal to or more than the threshold value, the verification process (query process) is not executed. In addition, a verification process (inquiry process) is performed to determine whether or not the reliability other than the reliability with the largest value among the reliability of the recognition results of the pattern recognizers A, B, and C is less than or equal to the threshold value or less than the threshold value. Does not affect the decision whether or not. However, if the reliability of a plurality of recognition results is the same, the reliability is the largest, and both exceed the threshold value or both are above the threshold value, and if the recognition results are different, which of the Since it is not possible to determine whether to adopt (correct) the recognition result, a verification process (inquiry process) for transmitting the verification data and the comparison data to the verification terminal 100 is executed, and any recognition result Verify that it is correct.

When the verification unit 61 determines to execute the verification process (inquiry process) of transmitting the verification data and the comparison data to the verification terminal 100, the verification unit 61 directly connects the connection device 40 via the network 1. By processing the perceptual data (acquired data obtained by the perceptual data acquisition means 20) transmitted via the pattern recognition means 51 of the pattern recognition device 50 or by the microtask automatic generation material. After performing a microtasking process (specifically, the preparation process) to create certain verification data, the created verification data is associated with the tag information and event identification information received with the sensory data, and the verification data is created. Processing for storing in the storage unit 62 is executed. The perceptual data may be directly used as the verification data without being processed. In this case, the perceptual data is directly stored in the verification data storage means 62 as a microtask automatic generation material used for the microtasking process. Let

In this microtasking process (here, the preparatory process of the material for microtask automatic generation, which is the preparatory step), the sensory data is processed so that it can be easily compared with the comparison data prepared in advance. This is a process of creating various verification data, for example, a process of creating still image data when the perceptual data is moving image data. As a result, in the verification terminal 100, the crowd (human) compares the verification data with the comparison data so that the state indicated by the verification data matches the true specific state indicated by the comparison data. You can easily determine whether or not. Therefore, the crowd (for example, a housewife, a student, etc.) has what kind of data the verification data and the comparison data compared with the verification terminal 100 are (for example, the meaning, purpose, nature, situation of the data, It is possible to carry out judgment work without knowledge or skills, without paying particular attention to (value, etc.).

The download of the verification data and the comparison data from the verification device 60 to the verification terminal 100 is executed by the verification means 61, for example, by the following method. A first method is to use a general-purpose application that is a communication tool such as LINE or a communication means such as electronic mail to request the operator (crowd) of the verification terminal 100 to perform a task (a so-called crowdsourcing microtask). ) Communicate that there is. The operator who has received this notification accesses a predetermined website or a website specified in a transmission text (text or voice) by a communication tool or electronic mail. This website is configured by the verification device 60 in the present embodiment. When the operator (crowd) of the verification terminal 100 browses the electronic bulletin board provided on the website with a general-purpose browser and selects an event he/she is in charge of on the electronic bulletin board (selects any event at will). Alternatively, the verification unit 61 may select an event designated by the system.), and the verification unit 61 stores the verification stored in the verification data storage unit 62 in association with the event identification information about the selected event. Data, comparison data stored in the verification data storage unit 62 in association with the same tag information as the tag information associated with the verification data, and stored in the verification data storage unit 62 in advance. Display data for the verification screens 200 and 300 (see FIGS. 4 and 5) is created using the displayed display form, and the display data (event identification information, verification data, comparison data, micro data) is displayed. A process of transmitting the task work result including the reply destination information of the manual judgment result) to the verification terminal 100 via the network 1 is executed. In addition, in the present embodiment, an example of using crowdsourcing operated in the system is described, but crowdsourcing operated externally may be used, and in the case of using this external service, This will be described in detail in a modified form described later.

As a second method, an electronic bulletin board is displayed on the screen of the verification terminal 100 by a dedicated application installed in the verification terminal 100, and the operator of the verification terminal 100 displays his/her own bulletin board on the electronic bulletin board. The event in charge may be selected, and the other points are the same as in the case of the first method described above.

In either of the first method and the second method, a plurality of people (for example, an odd number of people such as three people and five people) may make a judgment about one event, and one person may make a judgment. May be. In the case of multiple people, when the number of people from the crowd reaches the planned number of people, and in the case of one person, when there is one access from the crowd, the electronic bulletin board The display of the event in will disappear or become invalid. When multiple people make judgments, the judgments of multiple people may be divided. At that time, the judgment of the larger number of people (for example, judgment of two or more out of three people, judgment of five The judgment that 3 or more people agree) is the correct judgment.

Then, the verification unit 61 verifies, together with the event identification information, the data of the result of the manual judgment as to whether the monitored state corresponds to a specific state or which of a plurality of specific states. When there is data of the judgment result by a plurality of people from the user terminal 100 through the network 1, the judgment result of the one with a large number of people is determined, and the received judgment result by the hand and each pattern recognizer A, When the judgment results of the pattern recognitions of B and C match, it is judged that the recognition results of the pattern recognizers A, B and C are correct, and when they do not match, the pattern recognizer A of each pattern recognizer A , B, C are judged to be erroneous, and the verification result (correctness information) is associated with the event identification information to store the verification result of the data collecting device 70 for each pattern recognizer A, B, C. It is stored in the means 73 (see FIG. 2). When the verification process (inquiry process) of transmitting the verification data and the comparison data to the verification terminal 100 is not performed, the verification unit 61 has the largest value among the pattern recognizers A, B, and C. The recognition result of the pattern recognizer that gives the reliability is correct, and whether the judgment result by each pattern recognizer A, B, C is correct or incorrect depending on whether the recognition result of this pattern recognizer is correct or not. The correct/wrong information is associated with the event identification information as the verification result and stored in the verification result storage means 73 (see FIG. 2) of the data collection device 70 for each of the pattern recognizers A, B, and C.

<Data collection and management processing>

The data collection device 70 performs a data collection process and a management process of collected data (including a normal process of a database management system (DBMS)), and a database 72 (FIG. 2)).

As shown in FIG. 2, each table 72A, 72B, 72C, 72D, 72E of the database 72 includes a verification result storage unit 73, a learning data storage unit 74, an output log storage unit 75, and a performance storage unit 76. I am configuring.

The table 72A is a table that stores data related to the combination of the monitoring sensor 30A and the pattern recognizer A, and includes event identification information, output data of the monitoring sensor 30A (may be a path indicating a storage location of the output data), and a pattern. The judgment result (recognition result) by the recognizer A, the likelihood output from the pattern recognizer A, the reliability of the recognition result of the pattern recognizer A, and the verification result which is the correctness information of the judgment by the pattern recognizer A. And are stored in association with each other.

The table 72B is a table that stores data relating to a combination of the monitoring sensor 30B and the pattern recognizer B, and includes event identification information, output data of the monitoring sensor 30B (may be a path indicating a storage location of the output data), and a pattern. The judgment result (recognition result) by the recognizer B, the likelihood output from the pattern recognizer B, the reliability of the recognition result of the pattern recognizer B, and the verification result which is the correctness information of the judgment by the pattern recognizer B. And are stored in association with each other.

Although there is a table that stores data related to the combination of the monitoring sensor 30C and the pattern recognizer C, the description is omitted. Alternatively, since the monitoring sensor 30C has not been installed yet, it may be considered that the table is not generated or the record is not generated in the table.

Further, in the tables 72A and 72B, when there are a plurality of specific states detected by one monitoring sensor 30 (state X, state Y, state Z,... ), one pattern recognition is performed corresponding to those states. Since a plurality of likelihoods are output from the container, a plurality of likelihoods are stored in the likelihood column. Then, in the column of the judgment result by the pattern recognizers A and B in this case, for example, “state X (is)”, “state Y (is)”, “state Z (is)”, “state” "Neither X, Y, Z, ..." is stored. It should be noted that the same applies to the manual judgment result column of the table 72C and the adopted judgment result column of the table 72D.

The table 72C is a table for storing the data obtained by the inquiry processing (so-called crowdsourcing microtask) to the verification terminal 100, and stores the event identification information and the result of manual judgment in association with each other. .. When the verification means 61 receives the data of the judgment results by a plurality of people from the verification terminal 100, the judgment result of the larger number is stored. Further, a record in which the result of manual determination is NULL (or blank) is a record of an event for which the inquiry process to the verification terminal 100 has not been performed.

The table 72D is a table that stores the output log of the output unit 81, and stores the event identification information, the adopted information source, and the adopted judgment result in association with each other. Here, the “adopted information source” means whether the output unit 81 has adopted the manual judgment result (cloud) obtained by the inquiry processing to the verification terminal 100 as the final system output, or the pattern. It indicates whether the judgment result (recognition result) by the recognizer A, the judgment result (recognition result) by the pattern recognizer B, or the judgment result (recognition result) by the pattern recognizer C is adopted. Information. The “adopted judgment result” is a final system output.

The table 72E is a table that stores the performance of each pattern recognizer A, B, and C, and stores the identification information of the pattern recognizer and the performance of the pattern recognizer in association with each other. Although the performance of each pattern recognizer A, B, C is updated, the performance before the update may be saved without overwriting.

The verification result storage unit 73 is configured to include a column of event identification information of the tables 72A and 72B and a column of verification results (correctness information) of the pattern recognizers A and B.

Further, the learning data storage means 74 stores the event identification information column of the tables 72A and 72B, the output data column of the monitoring sensors 30A and 30B, the event identification information column of the table 72D, the adopted determination result (finally The correct judgment result adopted as the system output) column is included.

Further, the output log storage means 75 is composed of a table 72D, and the performance storage means 76 is composed of a table 72E.

<System output processing>

The output device 80 includes an output unit 81 that executes a process of outputting a final judgment result of the system, and a system management unit 82 that executes a management process of the system including various settings necessary for system operation. Is configured.

When the verification process (inquiry process) by the verification unit 61 is performed, the output unit 81 receives the data of the result of the manual judgment transmitted from the verification unit 61 of the verification device 60 via the network 1, If the result of the manual judgment is adopted and the verification process (inquiry process) is not performed, the pattern recognition means 51 of the pattern recognition device 50 directly transmits the verification result or the verification device 60 verifies the verification result. The decision result adopted by the pattern recognition means 51 transmitted through the means 61 (the recognition result having the highest reliability value among the recognition results of the pattern recognizers A, B, and C) is adopted, and the adopted decision result Data of the above is transmitted to the cooperation system 110 via the network 1, displayed on a screen of a display device (not shown), printed by a printing device (not shown), or output as voice by a speaker (not shown). , Or an alarm sound or light according to the adopted judgment result is output by a notifying device (not shown) such as a buzzer or a lamp, and the adopted judgment result data is output to a database 72 (see FIG. 2) output log. The processing for storing in the table 72D that constitutes the storage means 75 is executed.

The system management means 82 transmits the display data of the management screen 400 (see FIG. 6) to the system administrator terminal 120 via the network 1 in response to the request from the system administrator terminal 120 from the system administrator. Various setting information received by the system administrator using the management screen 400 and transmitted from the system administrator terminal 120 via the network 1 is received by the pattern recognition device 50 or the verification device 60. The setting information storage means (not shown) stores the processing.

More specifically, the system management means 82 displays the management screen 400 (see FIG. 6) on the system administrator terminal 120, and on the management screen 400, the pattern recognition means 51 (each pattern recognizer A, B, C). ), a setting input of a threshold value for the reliability of the recognition result is accepted, and the accepted threshold value is stored in a threshold value storage unit (not shown) installed in the verification device 60 or another device. The threshold may be described in the program forming the verification unit 61.

Further, the system management unit 82, on the management screen 400 (see FIG. 6), the performance of each pattern recognizer A, B, C calculated by the performance calculation unit 92, and the output log of the database 72 (see FIG. 2). A performance display process for displaying an output log (information indicating which judgment result is adopted by the output unit 81) stored in the table 72D configuring the storage unit 75 is executed, and the pattern recognizers A, B, and A recognition device necessity reception process of receiving a selection of use/non-use of C and storing the received selection information in a setting information storage unit (not shown) provided in the pattern recognition apparatus 50 and the verification by the verification unit 61. A process (inquiry process) necessity/non-necessity selection (the selection here is a selection as to whether or not the inquiry process is not performed till the end) is accepted, and the accepted selection information is set in the verification device 60 or the like. A verification necessity reception process to be stored in an information storage unit (not shown) is executed. Then, the pattern recognizers A, B, C of the pattern recognition means 51 and the verification means 61 refer to the selection information stored in these setting information storage means (not shown), and whether or not to perform the pattern recognition processing. In other words, it is determined whether or not to perform verification processing (inquiry processing) for transmitting the verification data and the comparison data to the verification terminal 100.

<Update process>

The update device 90 includes a learning unit 91 and a performance calculating unit 92.

The learning means 91 monitors the output data (data stored in the tables 72A, 72B, etc.) of the monitoring sensors 30A, 30B, 30C for each event stored in the learning data storage means 74 of the database 72 (see FIG. 2) and monitors. The model α, β, γ is repeatedly created using the tag information indicating the correct judgment result for the target state (the judgment result adopted by the output means 81 stored in the table 72D), and the created model α, By transmitting β and γ to the pattern recognition device 50 via the network 1, a process of updating the models α, β and γ stored in the model storage means 52 is executed.

The timing of updating the models α, β, γ by the learning unit 91 may be arbitrary and may be every event occurrence, or each time the total number of event occurrences is increased by a predetermined number (for example, 100 times). May be the cycle (monthly, weekly, etc.).

Further, as the learning data used by the learning means 91, all of the output data of the monitoring sensors 30A, 30B, 30C and the tag information (correct judgment result regarding the state indicated by those output data) are stored in the database 72 (see FIG. 2). Although it may be accumulated and stored in the learning data storage means 74, part of the data used for the updating process of the models α, β, γ may be deleted in order to efficiently use the resources. For example, when there is one specific state, the data (data with a small likelihood value) belonging to the boundary part that corresponds to the specific state or does not correspond to the specific state is effective for learning from the next time onward. The other data (data having a large likelihood value) is deleted without being deleted, and so on. In addition, when there are a plurality of specific states (for example, state X and state Y), the data belonging to the boundary portion of these specific states (for example, the likelihood for state X and the likelihood for state Y) , Data with almost no difference in value) are effective for learning from the next time onward, so keep them without deleting them, and delete other data (data with a significant difference in each likelihood value), etc. Is. Furthermore, when deleting data, if the average, variance, and total number of data for the deleted data are saved, they can be utilized in subsequent learning.

The performance calculation means 92 uses the verification results (correctness information) stored and stored for each pattern recognizer A, B, C in the verification result storage means 73 (tables 72A, 72B, etc.) of the database 72 (see FIG. 2). Then, the performance is calculated for each of the plurality of pattern recognizers A, B, C, and the calculated performance of each pattern recognizer A, B, C is transmitted to the data collection device 70 via the network 1. The processing for updating the performance of each pattern recognizer A, B, C stored in the performance storage means 76 (table 72E) of the database 72 (see FIG. 2) is executed. For example, the performance is obtained by dividing the number of times the recognition result is correct by the total number of events (the number of times the recognition result is correct and the number of times the recognition result is incorrect).

The timing of updating the performance by the performance calculating unit 92 is arbitrary, and may be every time an event occurs, or every time the total number of event occurrences increases by a predetermined number (for example, 100 times), a predetermined cycle (every month). , Weekly, etc.) and may be synchronized with the update timing of the models α, β, γ.

At this time, regarding the verification results accumulated and stored in the verification result storage means 73 (tables 72A, 72B, etc.), if the updating cycle of the models α, β, γ by the learning means 91 is short, the pattern recognizer using the same model is used. A large number of verification results regarding the pattern recognizer using the same model are not stored at all when the models α, β and γ are updated every time an event occurs. For example, when the model α is updated in a short cycle with the models α1, α2, α3, α4,..., The verification result of the pattern recognizer A using the same model α is hardly accumulated or is not accumulated at all. Therefore, in such a case, in order to calculate the performance of the pattern recognizer using the latest model, the update stored in the verification result storage means 73 (tables 72A, 72B, etc.) of the database 72 (see FIG. 2). Since it is not appropriate to use the verification result when the previous model is used, the monitoring sensors 30A, 30B, 30C stored in the learning data storage means 74 (tables 72A, 72B, etc.) of the database 72 (see FIG. 2). Using the output data of the above, the pattern recognition processing is redone by the pattern recognizers A, B, and C using the updated model, and each judgment result (each recognition result) obtained by the redone is used as the learning data storage means 74. By evaluating the correct judgment result (the judgment result adopted by the output means 81) stored in the (table 72D), new correct/incorrect information is obtained, and each pattern recognition is performed using the new correct/incorrect information. It suffices to calculate the performance of the vessels A, B, and C. If the model α, β, γ has a relatively long update period, a certain number of verification results obtained when the same model is used are accumulated. Therefore, each pattern recognizer A, The performances of B and C may be calculated.

In the above, the pattern recognition means 51, the verification means 61, the data collection means 71, the output means 81, the system management means 82, the learning means 91, and the performance calculation means 92 are inside each computer main body constituting each of the devices 50 to 80. It is realized by a central processing unit (CPU) provided in, and one or a plurality of programs that define the operating procedure of this CPU.

Further, the model storage means 52, the verification data storage means 62, and the database 72 are preferably realized by, for example, a hard disk, a solid state drive (SSD), etc., but within a range where there is no problem in storage capacity, access speed, etc. If so, it may be configured by another recording medium such as a digital versatile disk (DVD).

<External Configuration of Condition Monitoring System 10>

The verification terminal 100 is a client terminal operated by a crowd (human), is composed of a computer, and allows the operator to process the crowdsourcing microtask provided by the verification means 61 so that the verification data and the comparison data can be processed. It is provided with an output device such as a display device and a speaker for outputting, and input means such as a mouse and a keyboard. Further, the verification terminal 100 may be a mobile device such as a mobile phone or a tablet terminal. A crowd (human) operating the verification terminal 100 can receive a reward such as money or points by processing the crowdsourced microtasks, but may receive no reward. Further, these crowds (humans) may be persons (members or the like) registered in advance in the state monitoring system 10 and may have a mechanism that does not require registration.

The cooperation system 110 is configured by one or a plurality of computers, receives information regarding the state of the monitoring target transmitted from the output device 80 via the network 1 (the determination result adopted by the output unit 81), It is a system for utilizing the received information and is managed by a collaborator (for example, a farm or ranch manager, a forest manager, etc.).

The system administrator terminal 120 is a terminal device operated by the system administrator, is composed of a computer, and is provided with input means such as a mouse and a keyboard, and a display device such as a liquid crystal display. Further, the system administrator terminal 120 may be a mobile device such as a mobile phone or a tablet terminal.

<Flow of status monitoring processing>

In the present embodiment as described above, the state monitoring system 10 performs the state monitoring process for detecting a specific state corresponding to the monitoring purpose of the monitoring target as described below.

In FIG. 3, first, when a change in state is detected by the output data of any of the monitoring sensors 30A, 30B, 30C and a trigger is applied, an event occurs. At this time, the output data received from the monitoring sensors 30A, 30B, and 30C by the connection device 40 is sent to the network together with the monitoring sensor identification information, the tag information indicating the detected specific state, and the automatically assigned event identification information. 1 to the pattern recognition means 51 of the pattern recognition device 50 (step S1).

In parallel with this, the connection device 40 transmits the perceptual data received from the perceptual data acquisition unit 20 to the verification unit 61 of the verification device 60 via the network 1 together with the tag information and the event identification information. Alternatively, it is transmitted to the pattern recognition means 51 of the pattern recognition device 50 (step S1).

Then, in the pattern recognition device 50, the pattern recognition means 51 receives each output data of the monitoring sensors 30A, 30B, 30C and inputs each received output data to the corresponding pattern recognizer A, B, C. Then, the pattern recognition processing by each pattern recognizer A, B, C using the latest models α, β, γ stored in the model storage means 52 is executed (step S2). At this time, each pattern recognizer A, B, C outputs a recognition result (for example, corresponding to a specific state X) and its likelihood. When there are a plurality of specific states X, Y, Z,..., Likelihoods corresponding to the respective states X, Y, Z,... Are output, and the state with the highest likelihood is the pattern. It becomes the judgment result (recognition result) by recognition. Then, the pattern recognition means 51 transmits the recognition results and likelihoods of the pattern recognizers A, B, and C to the verification device 60 via the network 1 and stores them in the database 72 (see FIG. 2). ..

Then, in the verification device 60, the verification means 61 receives the recognition result and likelihood of each pattern recognizer A, B, C transmitted from the pattern recognition device 50 via the network 1, and the reliability of the recognition result. =f (likelihood, verification result), the reliability of the recognition result of each pattern recognizer A, B, C is calculated, and the reliability having the largest value among the calculated reliability of the recognition results is calculated in advance. It is determined whether it is less than or equal to or less than the set threshold value (step S3).

Here, when it is determined that the reliability having the highest value among the calculated reliability of the recognition result is less than or less than the predetermined threshold value, the verification unit 61 causes the connection device 40 or the pattern recognition device 50. Using the perceptual data (perceptual data acquired by the perceptual data acquiring means 20) received from the network 1 as a preparatory step for the process of microtasking, verification data, which is a material for automatically generating microtasks, is created. (Step S4). The sensory data may be directly used for the microtasking process without being processed.

Then, the verification means 61 executes the process of microtasking. That is, after storing the created verification data in the verification data storage unit 62, the created verification data and the corresponding comparison data are displayed on the electronic bulletin board provided on the website configured by the verification device 60. Post crowdsourcing microtasks that are performed using data. The display forms for the verification screens 200 and 300 as shown in FIGS. 4 and 5 are also prepared in advance and stored in the verification data storage means 62.

Further, the verification means 61 notifies the crowd (human beings) operating the verification terminal 100 that there is a requested work (so-called crowdsourcing microtask) by a communication tool such as LINE or an electronic mail. Upon receiving this notification, the operator accesses the website and selects the microtask related to the event he is in charge of on the electronic bulletin board.

Then, the verification unit 61 stores the verification data stored in the verification data storage unit 62 (the perceptual data acquired by the perceptual data acquisition unit 20 or the processed data thereof) in response to a request from the operator of the verification terminal 100. ) And the display data of the verification screens 200 and 300 (see FIGS. 4 and 5) including the comparison data are transmitted to the verification terminal 100 via the network 1 (step S5). Then, the verification screens 200 and 300 as shown in FIGS. 4 and 5 are displayed on the screen of the verification terminal 100.

The verification screen 200 of FIG. 4 is a case where there is one specific state (only the state X), and the verification screen 200 uses the comparison data to provide a truly specific state prepared in advance. An image showing X is displayed, and an image of which the state is desired to be judged is displayed using the verification data. Therefore, the microtask executed by the operator of the verification terminal 100 compares these images, determines whether the image whose state is desired to be determined appears in the specific state X, and selects according to the determination result. All that is required is to check the parts 201 and 202 and push down the "send" button 210.

The verification screen 300 of FIG. 5 is for a case where the number of specific states is plural (two) (state X, state Y), and this verification screen 300 is prepared in advance using comparison data. An image showing a truly specific state X and an image showing a truly specific state Y are displayed, and an image of which the state is desired to be judged using the verification data is also displayed. Therefore, the microtask executed by the operator of the verification terminal 100 compares these images and judges whether the image of which the state is desired looks like the specific state X or the specific state Y. All that is required is to determine whether or not the state is visible, check the selection units 301, 302, 303 according to the determination result, and push down the "send" button 310.

Further, when the specific state is plural (two states) (state X and state Y), first, the comparison is made instead of the microtask for requesting the three-choice decision work as in the verification screen 300 of FIG. Using the data for display, an image showing a truly specific state X prepared in advance is displayed, and the image for which the state is desired to be displayed is displayed using the data for verification, and these images are compared. Determine whether or not the image whose state you want to judge looks like the specific state X, and then use the comparison data to display an image that shows a truly specific state Y prepared in advance, and By using the verification data, you can display the images you want to judge the state of, you can compare these images, and you can judge whether the image you want to judge the state looks like a specific state Y or not. Alternatively, it may be a microtask that causes a simple two-choice decision work to be performed stepwise.

Then, on the verification screens 200 and 300 as shown in FIGS. 4 and 5, the operator of the verification terminal 100 checks the selection units 201, 202, 301 to 303 and clicks the “send” buttons 210 and 310. When the button is pushed down, the data of the result of the manual judgment as to whether or not the state of the monitoring target corresponds to the specific state X, or which of the plurality of specific states X and Y, is output from the verification terminal 100. It is transmitted to the verification device 60 via the network 1. In the verification device 60, when the verification means 61 receives the manual judgment result data, it verifies the correctness of the judgment by each of the pattern recognizers A, B, and C using the received manual judgment result data. The verification result and the like are transmitted to the data collection device 70 via the network 1 and stored in the database 72 (see FIG. 2) (step S6). Further, the verification means 61 transmits the data of the judgment result by hand and the data of the recognition result of each of the pattern recognizers A, B and C to the output device 80 via the network 1.

Further, in step S3 described above, when it is determined that the reliability having the largest value among the calculated reliability of the recognition result exceeds the predetermined threshold value or is equal to or more than the threshold value, the verification terminal 100 Inquiry processing (steps S4 to S6) to the cloud for transmitting the verification data and the comparison data to the cloud is not performed.

Then, in the output device 80, the output means 81 receives the data of the judgment result by hand and the data of the recognition result of each of the pattern recognizers A, B, C, and determines the judgment result to be adopted (step S7). At this time, when the output unit 81 receives the data of the result of the manual judgment, it is the case where the inquiry process to the verification terminal 100 is performed. When the data of the judgment result is not received, it means that the inquiry process to the verification terminal 100 has not been performed. Therefore, the judgment result (recognition result) by each of the pattern recognizers A, B, and C is the highest. The recognition result with a high reliability value is adopted. Then, the output unit 81 executes an output process such as transmitting the adopted determination result data to the cooperation system 110 via the network 1 (step S7).

After that, when the learning timing arrives, the learning means 91 causes the learning data storage means 74 of the database 72 (see FIG. 2) to output the output data of the monitoring sensors 30A, 30B, 30C for each event (table 72A). , 72B and the like) and tag information indicating the correct judgment result regarding the state of the monitoring target (judgment result adopted by the output means 81 stored in the table 72D). γ is created, and the created models α, β, γ are transmitted to the pattern recognition device 50 via the network 1 to update the models α, β, γ stored in the model storage unit 52 (step S8). ..

Further, when the performance calculation timing arrives, the performance calculation unit 92 stores the verification result storage unit 73 (tables 72A, 72B, etc.) of the database 72 (see FIG. 2) for each pattern recognizer A, B, C. Using the stored verification results (correctness information), the performance of each pattern recognizer A, B, C is calculated, and the calculated performance of each pattern recognizer A, B, C is transferred via the network 1. The performance of each pattern recognizer A, B, C stored in the performance storage means 76 (table 72E) of the database 72 (see FIG. 2) is updated by transmitting to the collection device 70 (step S9).

Then, if the state monitoring process of the monitoring target is continued (step S10), the process returns to step S1. On the other hand, if the process is not continued, the state monitoring process by the state monitoring system 10 ends.

<Effect of status monitoring system 10>

According to the present embodiment as described above, there are the following effects. That is, since the state monitoring system 10 includes the verification unit 61, if the reliability of the recognition result determined using the likelihood output from the pattern recognition unit 51 is less than or less than the threshold value, the perceptual data acquisition unit. An inquiry process for transmitting the perceptual data acquired by 20 or the data obtained by processing the perceptual data as verification data to the verification terminal 100 can be performed to obtain a manual judgment result. When this verification process (inquiry process) is performed, the output unit 81 can replace the judgment result by the pattern recognition with the system judgment result.

Therefore, at the initial stage of introducing the state monitoring system 10 into a certain environment, it is not clear that the pattern recognition means 51 based on the initial model created by using the data collected in the test environment is compatible with the environment of the introduction destination. However, the reliability value of the recognition result by the pattern recognition means 51 becomes small, but in that case, the inquiry process to the verification terminal 100 is performed, and the judgment result is corrected to the correct result by the manual judgment result. Therefore, it is possible to improve the quality of the information finally output by the output means 81 (judgment result about the state of the monitoring target). Therefore, even if the recall is prioritized, the accuracy is not sacrificed, the trade-off problem can be solved, and the high-performance state that can always satisfy the quality required by the user of the system. The monitoring system 10 can be realized.

In addition, when time passes after the introduction, the cumulative number of pattern recognition processes increases, and the verification unit 61 repeats the verification process of correctness of the judgment by the pattern recognition and accumulates the verification results. Since the degree of conformity of the used pattern recognition means 51 to the actual environment becomes clear, the reliability of the recognition result increases, and as a result, the inquiry processing to the verification terminal 100 is performed. The frequency is getting lower. Therefore, as time passes, the number of cases in which system output is performed by the output unit 81 increases without relying on the manual judgment result received from the verification terminal 100, and therefore the time required from event occurrence to output is shortened. I can go. Therefore, the state monitoring system 10 can be flexibly adapted to the environment of the installation destination regardless of the environment in which it is installed and can finally output the accurate judgment result in a short time. can do.

Furthermore, since the verification means 61 is provided, the reliability of the recognition result by the pattern recognition means 51 at the beginning of introduction may be low, and therefore the expensive monitoring sensor 30 and pattern that can exhibit high performance from the beginning of introduction. It is not necessary to prepare the recognizing means 51, and when creating the initial model, it is not always necessary to collect data in a real environment or an environment very close to it, so that investment in the initial model can be suppressed and the state can be reduced. It is possible to accelerate the operation start time of the monitoring service. Therefore, the system can be constructed easily and inexpensively.

Further, since the state monitoring system 10 includes the learning data storage means 74 and the learning means 91, the models α, β, γ stored in the model storage means 52 can be updated, so that the latest model α, Pattern recognition processing can be executed by the pattern recognizers A, B, and C using β and γ. Therefore, the performance of the pattern recognition means 51 can be improved. This is because the output data of the monitoring sensor 30 stored in the learning data storage means 74 is the data collected in the actual environment while operating the system 10, and the output data of these monitoring sensors 30 together with the tag information. The stored tag information is the data of the judgment result adopted by the output means 81, and the verification result of the verification means 61 is utilized, so that the tag information is guaranteed to be accurate, and therefore the learning means. This is because 91 makes it possible to perform appropriate machine learning using learning data whose accuracy is guaranteed. Therefore, the reliability of the recognition result by the pattern recognition unit 51 is further increased, and as a result, the frequency of performing the inquiry process to the verification terminal 100 is further reduced, and the time required from the event occurrence to the output is output. Can be further shortened.

Furthermore, since the state monitoring system 10 includes a plurality of monitoring sensors 30A, 30B, 30C and a plurality of pattern recognizers A, B, C corresponding thereto, as shown in FIG. After the start, the level of performance of each pattern recognizer A, B, C may be switched. Therefore, by operating the pattern recognizers having different characteristics in combination, high performance can be continuously exhibited, so that the frequency of inquiry processing to the verification terminal 100 is reduced and the time required from event occurrence to output is reduced. Can be shortened.

Further, as shown by the dotted line in FIG. 7, the pattern recognizer exhibiting the highest performance may be replaced, so after the replacement, the monitoring sensor 30 with lower performance is removed and the output of the monitoring sensor 30 is changed. It is also possible to discontinue use of the pattern recognizer that was processing the data. Therefore, for example, immediately after the operation of the system is started, the expensive monitoring sensor 30A is made to function to ensure the quality, and thereafter, when the performance of the pattern recognizer by the inexpensive monitoring sensor 30B is improved, the expensive monitoring sensor 30A is replaced. It can be removed and supplied to another destination. Furthermore, it is possible to add and operate the pattern recognizer and the monitoring sensor 30 having different characteristics later.

Since the state monitoring system 10 includes the system management unit 82, the verification result storage unit 73, the performance calculation unit 92, and the output log storage unit 75, the management screen 400 displayed on the system administrator terminal 120 (see FIG. 6), the performance display process of displaying the performance of each pattern recognizer A, B, C and the output log on the screen, and the necessity of the recognizer for accepting selection of use/non-use of each pattern recognizer A, B, C It is possible to execute the verification necessity reception process that receives the acceptance process and the selection of the necessity of the verification process (the inquiry process) by the verification unit 61 (hereinafter, the selection of whether or not the inquiry process may be stopped until the end). ..

Therefore, the system administrator can refer to the performances and output logs of the pattern recognizers A, B, and C on the system administrator terminal 120, and confirm the current status (the latest status at each time). However, the performance becomes relatively low, and it is determined whether or not the unnecessary monitoring sensor 30 is removed, and whether or not to stop using the pattern recognizer that processes the output data of the monitoring sensor 30. Alternatively, it is possible to judge whether or not the inquiry process (verification process) to the verification terminal 100 by the verification unit 61 may be stopped.

<Deformation form>

It should be noted that the present invention is not limited to the above embodiment, and modifications and the like within the range in which the object of the present invention can be achieved are included in the present invention.

For example, in the above-described embodiment, the condition monitoring system 10 is configured to include a plurality of monitoring sensors 30A, 30B, 30C and a plurality of pattern recognizers A, B, C corresponding thereto, The condition monitoring system of the present invention may be configured to include only one set of the monitoring sensor 30 and the pattern recognizer. However, it is preferable to operate by combining a plurality of pattern recognizers having different characteristics, and by doing so, the pattern recognition means 51 can continue to exhibit high performance, and therefore the frequency of inquiry processing to the verification terminal 100. The time required from the event occurrence to the output can be shortened, and the unnecessary monitoring sensor 30 can be removed.

Further, in the above-described embodiment, processing of microtasking for creating verification data which is a material for automatic generation of microtasks by processing the perceptual data acquired by the perceptual data acquiring means 20 (specifically, its preparation step). Is performed by the verification means 61 of the verification device 60, but may be performed by the connection device 40, or the data processed by the connection device 40 may be further processed by the verification means 61 of the verification device 60.

Further, in the above-described embodiment, the server that provides the microtasks to the crowd is configured by the verification device 60, but a system different from the condition monitoring system 10 (for example, a provider that provides a service that provides the microtasks to the crowd). System managed by the above). In this case, the verification means 61 of the verification device 60 causes the verification data, which is the material for automatically generating microtasks, and the comparison data corresponding thereto, and the verification screens 200 and 300 (see FIGS. 4 and 5). The display form is sent to another system via the network 1 together with the event identification information, and is stored in the other system at least until the work of the microtask by the crowd is completed. Since the comparison data and the display form for the verification screen do not change every event, it may be sent to another system and registered in advance. The verification means 61 of the device 60 transmits the tag information together with the verification data to each system for each event, so that the other system uses any comparison data or any verification screen. It is possible to determine whether to complete the process of microtasking using the display form of. Further, on the state monitoring system 10 side (the verification means 61 side of the verification device 60), display data (event identification information, verification data, comparison data) such as the verification screens 200 and 300 (see FIGS. 4 and 5). , Including the reply destination information of the human judgment result which is the microtask work result) is completed, and the display data of the verification screen in the completed state is transmitted to another system via the network 1, It may be stored, or another system may perform a process of adding only the reply destination information. In short, the microtasking process may be shared between the state monitoring system 10 and another system, or may be performed only by the state monitoring system 10 side (the verification means 61 side of the verification device 60). Further, the process of transmitting the fact that there is a microtask to the crowd (human) may be performed by another system or by the verification unit 61 of the verification device 60. Then, a crowd (human being) operating the verification terminal 100 accesses the other system, and on the verification screens 200 and 300 (FIGS. 4 and 5) transmitted from the other system to the verification terminal 100. Performs judgment tasks for microtasks. The manual judgment result data returned from the verification terminal 100 may be received by the verification means 61 of the verification device 60 via another system, or directly by the verification means 61 of the verification device 60. You may. Therefore, the transmission/reception processing performed between the verification means and the verification terminal in the present invention (the invention according to the claims) includes the case where the transmission/reception processing is performed via another system as described above.

When the crowdsourcing operated externally in this way is used, the microtask generated inside the system of the present invention (finally after receiving the material for microtask automatic generation from the system of the present invention, Externally completed microtasks) are presented to the crowd (cloud workers) alongside other systems of microtasks generated by other systems. The cloud worker freely selects a task from among various tasks, but if it happens to select the microtask generated by the system of the present invention, the cloud worker becomes the operator (crowd) of the verification terminal 100. .. The verification terminal 100 in this case is usually a terminal equipped with general software such as a web browser of a smartphone. However, in order to perform microtask determination work by communicating with another system from another system or from the system of an application distributor (distribution system of various applications) requested by a company that operates another system. When a dedicated application is distributed, for example, the verification terminal 100 equipped with the dedicated application may be used.

As described above, the condition monitoring system of the present invention, for example, in order to prevent damage to the crops and environmental destruction by wild birds and beasts, when automatically detecting the invasion of wild birds and beasts into the farm or ranch, forest to be protected, livestock, In order to automatically detect the invasion of wild animals into the feeding grounds and bedding, learn the behaviors of livestock and pests, and automatically detect the conditions of the elderly and patients requiring care at home or in a facility such as a hospital. When monitoring, for crime prevention, when monitoring the surroundings of buildings, when predicting various dangers in places where people gather, landslides due to earthquakes and torrential rain, avalanches, river floods due to bank breaks, lightning strikes, It is suitable for use in predicting and early detection of various disasters such as fire.

1 Network 10 State Monitoring System 20 Perceptual Data Acquisition Means 30, 30A, 30B, 30C Monitoring Sensor 51 Pattern Recognition Means 52 Model Storage Means 61 Verification Means 73 Verification Result Storage Means 74 Learning Data Storage Means 75 Output Log Storage Means 81 Output Means 82 System management means 91 Learning means 92 Performance calculation means 100 Verification terminal 110 Cooperative system 120 System administrator terminal A, B, C Pattern recognizer α, β, γ model

Claims (4)

A state monitoring system for detecting a specific state corresponding to a monitoring purpose of a monitoring target,
A monitoring sensor for detecting the state of the monitoring target,
Provided in order to obtain perceptual data that can be perceived by human perception of the state of the monitoring target, different from the monitoring sensor, or perceptual data acquisition means also used as the monitoring sensor,
A pattern recognition process is executed using the output data of the monitoring sensor to determine whether or not the state of the monitoring target corresponds to the specific state, or which of a plurality of specific states the monitoring target state corresponds to. Pattern recognition means,
If the reliability of the recognition result determined using the likelihood output from the pattern recognition means is less than or less than a predetermined threshold value, the perceptual data acquired by the perceptual data acquisition means or the processed data thereof. Is transmitted to the verification terminal via the network as verification data for verifying the correctness of the judgment by the pattern recognition means, and the operator of the verification terminal judges using the verification data. Whether or not the state of the monitoring target corresponds to the specific state, or data of a result of manual determination of which of a plurality of the specific states, the data of the determination result from the verification terminal via the network. Verification means for receiving and executing verification processing for verifying the correctness of the judgment by the pattern recognition means using the received data of the judgment result by the human,
When the verification processing by the verification means is performed, the judgment result by the human is adopted, and when the verification processing is not carried out, the judgment result by the pattern recognition means is adopted, and the data of the adopted judgment result. Is transmitted to the cooperation system via the network, is displayed on the screen of the display device, is printed by the printing device, is output by voice, or outputs a notification sound or notification light according to the judgment result adopted. and output means for performing,
Model storing means for storing a model used in the pattern recognition means,
Learning data storage means for storing the output data of the monitoring sensor in association with the determination result data adopted by the output means,
Using the output data of the monitoring sensor stored in the learning data storage means, repeatedly creating the model, and a learning means for performing a process of updating the model stored in the model storage means ,
The pattern recognition means,
A state monitoring system characterized by being configured to execute the pattern recognition processing using the latest model stored in the model storage means . A plurality of the monitoring sensors,
The pattern recognition means,
It is configured to include a plurality of pattern recognizers corresponding to the plurality of monitoring sensors,
The model storage means,
It is configured to store a plurality of models corresponding to the plurality of pattern recognizers,
The learning data storage means,
Each of the output data of the plurality of monitoring sensors is configured to be stored in association with the data of the determination result adopted by the output means,
The learning means is
A process of repeatedly creating each of the plurality of models using each of the output data of the plurality of monitoring sensors stored in the learning data storage unit and updating the plurality of models stored in the model storage unit Is configured to run
The verification means is
Among the reliability of the recognition result determined by using the likelihood output from each of the plurality of pattern recognizers, the largest reliability is the verification data to the verification terminal when the reliability is the threshold value or less or less. Is transmitted, and is configured to perform a verification process of receiving the data of the judgment result by the human from the verification terminal,
The output means is
When the verification processing by the verification means is performed, the result of the manual judgment is adopted, and when the verification processing is not performed, the pattern having the highest reliability value among the plurality of pattern recognizers is adopted. The condition monitoring system according to claim 1 , wherein the condition monitoring system is configured to execute a process that employs a determination result by the recognizer. Verification result storage means for accumulating and storing verification results of the plurality of pattern recognizers by the verification means, for each pattern recognizer;
Using the verification results accumulated and stored for each pattern recognizer in the verification result storage means, performance calculation means for executing a process of calculating performance for each of the plurality of pattern recognizers,
The performance of the plurality of pattern recognizers calculated by this performance calculation means, or in addition to these performances, an output log indicating which judgment result is adopted by the output means stored in the output log storage means is displayed. , Perform performance display processing to be displayed on the screen of the system administrator terminal, or in addition to this performance display processing, perform recognizer necessity reception processing that accepts selection of use/non-use of each of the plurality of pattern recognizers The system monitoring means according to claim 2 , further comprising: A state monitoring system for detecting a specific state corresponding to a monitoring purpose of a monitoring target,
A monitoring sensor for detecting the state of the monitoring target,
Provided in order to obtain perceptual data that can be perceived by human perception of the state of the monitoring target, different from the monitoring sensor, or a sensory data acquisition unit also used as the monitoring sensor,
A pattern recognition process is executed using the output data of the monitoring sensor to determine whether the state of the monitoring target corresponds to the specific state or which of a plurality of specific states the monitoring target state corresponds to. Pattern recognition means,
If the reliability of the recognition result determined using the likelihood output from the pattern recognition means is less than or less than a predetermined threshold value, the perceptual data acquired by the perceptual data acquisition means or the processed data thereof. Is transmitted to a verification terminal via a network as verification data for verifying the correctness of the judgment by the pattern recognition means, and the operator of the verification terminal judges using the verification data. Whether or not the state of the monitoring target corresponds to the specific state, or the data of the result of the manual determination of which of the plurality of specific states, from the verification terminal via the network Verification means for receiving and executing verification processing for verifying the correctness of the judgment by the pattern recognition means using the received data of the judgment result by the human being;
When the verification processing by the verification means is performed, the judgment result by the human is adopted, and when the verification processing is not performed, the judgment result by the pattern recognition means is adopted, and the data of the adopted judgment result is adopted. Is transmitted to the cooperation system via the network, is displayed on the screen of the display device, is printed by the printing device, is output by voice, or outputs a notification sound or notification light according to the judgment result adopted. and output means for performing,
Verification result storage means for accumulating and storing verification results by the verification means;
A performance calculation means for executing a process of calculating the performance of the pattern recognition means using the verification results accumulated and stored in the verification result storage means;
The performance of the pattern recognition means calculated by the performance calculation means, or, in addition to this performance, an output log indicating which judgment result is adopted by the output means stored in the output log storage means is managed by the system management. A system management unit that executes a performance display process for displaying on a person terminal screen, or in addition to this performance display process, executes a verification necessity reception process that receives a selection of necessity of the verification process by the verification unit . A condition monitoring system characterized in that

Images