JP2023112782A - State detection device, state detection method, and state detection program - Google Patents

Abstract

To simply suppress the frequency at which a state of an observation object is erroneously recognized due to difference between a learning environment and an operation environment.SOLUTION: A storage part stores observation data inputted to an input part and a recognition result of a state of the observation object recognized, on the basis of a feature amount of the observation data, by a recognition model generated by machine learning in association with each other. When the recognition result is designated to be erroneous recognition, an extraction part extracts, on the basis of a feature amount of observation data having acquired the designated recognition result, other observation data having a possibility that the recognition result is erroneous recognition. When receiving an update permission of the recognition model, an update part updates a parameter to be used to determine a state of the observation object by the recognition model.SELECTED DRAWING: Figure 4

Description

The present invention relates to technology for recognizing the state of an observation target based on observation data of the observation target.

Conventionally, there has been a device for recognizing the identity of an object to be detected and a known object to be detected (see Patent Document 1).

The device of Patent Document 1 stores information (spatial information) that constitutes a reference space (Mahalanobis space) generated from the feature amount of a known detected object for each type of known detected object by machine learning in a storage unit. are doing. This device obtains the input feature amount of the object to be detected. This device calculates the distance (Mahalanobis distance) between the reference space of the type and the obtained feature amount of the detected object for each known type of detected object. This device recognizes which of the known detection objects the object to be detected is based on the distance calculated for each type of known detection object.

In addition, the apparatus of Patent Document 1 accumulatively stores a feature amount acquired from a detected object that has not been recognized as a known detected object. In addition, this device performs re-learning at an appropriate timing using the feature amount of a detected object that has not been recognized as to which of the known detected objects is stored, thereby improving the recognition accuracy of the detected object. I am planning.

JP 2005-214682 A

However, the device of Patent Document 1, due to the difference between the environment (learning environment) in which learning data used for machine learning is acquired and the environment (operating environment) in which observation data of the object to be recognized is acquired, It is not assumed that misrecognition of the body will occur. That is, the apparatus of Patent Document 1 cannot suppress erroneous recognition of the detected object caused by the difference between the learning environment and the operating environment.

In addition, the operating environment may change with the passage of time, and even if sufficient recognition accuracy is obtained at the start of operation, the recognition accuracy will decline with the passage of time (increase in the frequency of misrecognition). .

SUMMARY OF THE INVENTION An object of the present invention is to provide a technique that can easily suppress the frequency of erroneous recognition of the state of an observation target due to the difference between the learning environment and the operating environment.

In order to achieve the above objects, the state detection device of the present invention is constructed as follows.

Observation data of an observation target is input to the input unit. An observation target is a target whose state is to be recognized, and is, for example, space, a person, an object, or a substance. Observation data is sensing data obtained by sensing an observation target with a sensor. The observation data may be, for example, a frame image of the observation target captured by a camera, sound data around the observation target collected by a microphone, or the temperature of the observation target measured by a temperature sensor. may Also, the number of observation data may be plural instead of one.

The storage unit stores the observation data input to the input unit in association with the recognition result of the state of the observation target recognized by the recognition model generated by machine learning based on the feature amount of the observation data. The recognition model may have, for example, a Variational Autoencoder (VAE) trained using learning data including normal observation data of the observation target, or other types may be a recognition model of The recognition model may recognize, for example, whether the state of the observation target is a normal state or an abnormal state, or recognize which of three or more predetermined states it is. may be

If the recognition result is specified as an erroneous recognition, the extracting unit extracts other observation data that may have been an erroneous recognition based on the feature amount of the observation data that obtained the specified recognition result. Extract from memory. The extraction unit extracts, for example, observation data whose feature amount is similar to the feature amount of the observation data from which the recognition result pointed out as erroneous recognition was obtained.

The update unit updates the parameters used by the recognition model to determine the state of the observation target when permission to update the recognition model is received after the observation data extracted by the extraction unit is output. For example, the observation data extracted by the extraction unit may be displayed on a display so that the user can visually recognize it.

With this configuration, it is possible to allow the user to check other observation data that may have been misrecognition as a result of recognition. In addition, the user checks the extracted observation data for the possibility that the recognition result was an erroneous recognition, and if the recognition for all the output observation data is erroneous recognition, the user is allowed to update the recognition model. Such an input operation is performed. As a result, the update unit updates the parameters that the recognition model uses to determine the state of the observation target.

Therefore, the user can suppress the occurrence frequency of erroneous recognition of the observation target caused by the difference between the learning environment and the operating environment, etc., by a simple operation of selecting and designating the erroneously recognized recognition result.

Further, for example, when the observation data extracted by the extraction unit is output, and the specification of the observation data whose recognition result is not to be corrected is received, the extraction unit receives the observation data previously extracted based on the feature amount of the specified observation data. A configuration may be provided in which an instruction unit is provided to instruct narrowing down of other observation data for which the recognition result may have been an erroneous recognition.

With this configuration, if the observation data extracted by the extracting unit includes observation data in which the recognition result is not erroneous, the recognition result is excluded and the recognition model determines the state of the observation target. You can update the parameters used for Therefore, it is possible to prevent the parameter used by the recognition model to determine the state of the observation target from being updated to an inappropriate value.

Further, when the recognition model is, for example, a VAE, the VAE acquires the distribution of the latent variables of the observation data input to the input unit as the distribution for judgment as the feature amount, and the distribution for judgment and the reference distribution are obtained. may be configured to recognize the state of the observation target based on the distance of .

For the distance between the judgment distribution and the reference distribution, for example, the Mahalanobis distance may be used.

The observation data is the difference between the first average image of the n frame images captured temporally continuously and the second average image of the m frames captured temporally continuously. It can be an image. However, in this case, n is larger than m, and m is 1 or more. Also, the frame image captured at the latest time used to generate the first average image and the frame image captured at the latest time used to generate the second average image are the same frame image. .

With this configuration, an image (difference image) of an object appearing in the imaging area is captured immediately before the timing of capturing the latest frame image used to generate the first average image and the second average image. , can be obtained as observation data. Therefore, in the case of a recognition model that recognizes the state of an observation target based on the types, sizes, positions, etc. of objects located within the observation target area, it is possible to use observation data suitable for recognizing the state of the observation target. can.

According to this invention, it is possible to easily suppress the frequency of erroneous recognition of the state of the observation target due to the difference between the learning environment and the operating environment.

It is a figure which shows the abnormality recognition system to which the state detection apparatus of this example is applied. It is an example of the frame image which the video camera imaged the station premises which are observation targets. It is an example of the frame image which the video camera imaged the road which is an observation object. It is a figure which shows the structure of the principal part of the state detection apparatus of this example. It is a block diagram which shows the functional structure of the recognition part of the state detection apparatus of this example. It is a flowchart which shows the state recognition processing of the state detection apparatus of this example. It is a flowchart which shows the update process of the state detection apparatus of this example.

Embodiments of the present invention will be described below.

<1. Application example>
FIG. 1 is a diagram showing an abnormality recognition system to which the state detection device of this example is applied. An abnormality recognition system 100 of this example includes a state detection device 1 , a video camera 2 , a display 3 and an input device 4 .

The state detection device 1 of this example recognizes whether the state of the observation target is normal or abnormal, and outputs the recognition result. The observation target may be a railway station premises or a road on which a vehicle travels. The state detection device 1 of this example recognizes an abnormal state if there is a suspicious object 105 left unattended when the observation target is a railway station premises. Further, when the observation target is a road on which a vehicle travels, the state detection device 1 of this example recognizes an abnormal state when there is an obstacle on the road that hinders the travel of the vehicle 110 . The obstacle is, for example, a fallen object (baggage) that has fallen onto the road from the carrier of the vehicle 110 or the like, or the vehicle 110 that has stopped on the road due to a failure such as engine trouble.

The video camera 2 captures an image of an observation target and outputs the captured frame image to the state detection device 1 . The video camera 2 is attached at an angle that allows the observation target to fit within the imaging area. In other words, all or part of the imaging area of the video camera 2 is the observation target. 2 and 3 are schematic diagrams showing frame images captured by a video camera. FIG. 2 is an example of a frame image captured by the video camera 2 inside a station, and FIG. 3 is an example of a frame image captured by the video camera 2 of a road. The frame rate of the video camera 2 is, for example, several tens of frames/sec (eg, 10 to 30 frames/sec). Observation data referred to in the present invention is obtained from frame images of an observation target captured by the video camera 2 .

The video camera 2 may be a digital still camera that captures a still image in response to a release signal input from an external device.

The display 3 displays the recognition result of the state of the observation target in the state detection device 1 and the like. The display 3 displays a screen based on screen display data input from the state detection device 1 . That is, the state detection device 1 controls the screen displayed on the display 3 . The display 3 also displays screens related to a GUI (Graphical User Interface).

The input device 4 is a keyboard, a mouse, or the like with which the user performs an input operation on the state detection device 1 .

The state detection device 1 recognizes whether or not the state of the observation target is normal using a recognition model generated by machine learning. This recognition model recognizes whether or not the state of the observation target is normal based on the feature amount of the observation data obtained from the frame images of the observation target captured by the video camera 2 . The state detection device 1 stores observation data and recognition results in association with each other.

Further, when receiving an input specification indicating that the recognition result of the recognition model is an erroneous recognition, the state detection device 1 determines that the recognition result is an erroneous recognition based on the feature amount of the observation data for obtaining the specified recognition result. Extract and output other observation data that may have been. For example, the state detection device 1 causes the screen of the display 3 to display part or all of the observation data extracted as a possibility that the recognition result was an erroneous recognition.

When the user confirms that all of the observation data extracted as the result of recognition may have been misrecognition is misrecognition, the user operates the input device 4 to permit the update of the recognition model. Enter 1. When receiving an update permission input, the state detection device 1 corrects the values of the parameters used to determine the state of the observation target for all the observation data extracted for the possibility that the recognition result was an erroneous recognition. update to a value that gives the correct recognition result.

In addition, when the user confirms observation data with a correct recognition result (not misrecognition) among the observation data extracted with the possibility that the recognition result was misrecognition, the user operates the input device 4. Then, the non-permission of the update of the recognition model is input to the state detection device 1 . At this time, the user performs an input operation to designate one or more pieces of observation data for which the recognition result is correct, among the observation data extracted as being possibly misrecognised. When the state detection device 1 receives an input indicating that the update is not permitted, based on the feature amount of the observation data for which the recognition result specified this time is correct, the observation extracted based on the possibility that the previous recognition result was an erroneous recognition. Narrowing is performed by excluding observation data whose recognition result is estimated to be correct from the data.

The state detection device 1 displays part or all of the narrowed down observation data on the screen of the display device 3, and waits for the user's input to permit or deny update.

Therefore, the user can update the value of the parameter used to determine the state of the observation target by performing an input operation to specify the recognition result that has been confirmed to be misrecognition. Therefore, the state detection device 1 of this example can easily suppress the occurrence frequency of erroneous recognition of the observation target caused by the difference between the learning environment and the operating environment.

<2. Configuration example>
FIG. 4 is a diagram showing the configuration of the main part of the state detection device of this example. The state detection device 1 of this example includes a control unit 11 , an input section 12 , an output section 13 , a storage section 14 , a display output section 15 and an input reception section 16 .

The control unit 11 controls the operation of each part of the state detection device 1 main body. The control unit 11 also has a recognition section 11a, an extraction section 11b, an update section 11c, an instruction section 11d, and a display control section 11e. The recognition unit 11a, the extraction unit 11b, the update unit 11c, the instruction unit 11d, and the display control unit 11e included in the control unit 11 will be described later.

The input unit 12 is connected to a video camera 2 attached at an angle for capturing an image of an observation target. The input unit 12 receives a frame image of an observation target captured by the video camera 2 .

The output unit 13 outputs the recognition result of recognizing the state of the observation target to the host device.

The storage unit 14 associates and stores the observation data of the observation target and the recognition result of the state of the observation target with respect to the observation data. The storage unit 14 is a storage medium such as a HDD (Hard Disk Drive), an SSD (Solid State Drive), an SD memory card, or the like.

The display device 3 is connected to the display output unit 15 . The display output unit 15 outputs screen display data to the display device 3 . The display device 3 displays a screen based on this screen display data. The display output unit 15 is an interface circuit with the display device 3 .

An input device 4 is connected to the input reception unit 16 . Input operation data corresponding to the input operation of the input device 4 by the user is input to the input reception unit 16 . The input reception unit 16 is an interface circuit with the input device 4 .

Next, the recognition section 11a, the extraction section 11b, the update section 11c, the instruction section 11d, and the display control section 11e of the control unit 11 will be described.

The recognition unit 11a recognizes the state of the observation target (normal or abnormal in this example). FIG. 5 is a block diagram showing the functional configuration of the recognition section of the state detection device of this example. In this example, the recognition unit 11a includes a variational autoencoder 20 (hereinafter referred to as a VAE 20 (Variational Autoencoder)), a feature value calculation unit 23, a distance calculation unit 24, a distribution storage unit 25, and a determination a portion 26;

The VAE 20 is a recognition model generated by deep learning using learning data including normal observation data of an observation target. VAE 20 has encoder 21 and decoder 22 . Encoder 21 and decoder 22 are neural networks. The encoder 21 extracts the latent variable z from the input observation data. The decoder 22 receives the latent variable z and reconstructs observation data. Here, the observation data restored from the latent variable z by the decoder 22 is called restored data.

The feature quantity calculator 23 calculates the feature quantity of the latent variable z. The distribution storage unit 25 stores the distribution of the feature amount of the latent variable z obtained for observation data when the state of the observation target is normal (hereinafter referred to as normal distribution). The distribution storage unit 25 stores the distribution of the feature amount of the latent variable z in the reference space. The reference space is explained as Mahalanobis space in this example, but it may be a space generated by another statistical method.

Note that the distribution storage unit 25 may store the distribution of the feature amount of the latent variable z obtained for the observation data when the state of the observation target is abnormal (hereinafter referred to as an abnormal distribution), Both normal and abnormal distributions may be stored.

The distance calculation unit 24 applies the feature amount of the latent variable z calculated for the observation data by the feature amount calculation unit 23 to the Mahalanobis space, which is the reference space, to calculate the distance (Mahalanobis distance) from the normal distribution.

The determination unit 26 compares the Mahalanobis distance calculated by the distance calculation unit 24 with a predetermined determination distance, and determines whether the state of the observation target is normal or abnormal. For example, if the Mahalanobis distance to the normal distribution calculated by the distance calculation unit 24 is equal to or less than the judgment distance, the judgment unit 26 judges that the state of the observation target is normal. If the Mahalanobis distance to the distribution is not equal to or less than the judgment distance, it is judged that the state of the observation target is abnormal.

When the distribution storage unit 25 stores the abnormal distribution of the feature amount of the latent variable z, the distance calculation unit 24 stores the feature amount of the latent variable z calculated for the observation data by the feature amount calculation unit 23 in the reference space. Calculate the Mahalanobis distance to the abnormal-normal distribution when applied to the Mahalanobis space. Further, if the Mahalanobis distance to the abnormal distribution calculated by the distance calculation unit 24 is equal to or greater than the judgment distance, the judgment unit 26 judges that the state of the observation target is normal, and determines that the state of the observation target is normal. If the Mahalanobis distance to the distribution is not equal to or greater than the judgment distance, it may be judged that the state of the observation target is abnormal.

Furthermore, when the distribution storage unit 25 stores the normal distribution and the abnormal distribution of the feature amount of the latent variable z, the distance calculation unit 24 stores the feature amount of the latent variable z calculated for the observation data by the feature amount calculation unit 23. The Mahalanobis distance to the normal distribution and the Mahalanobis distance to the abnormal normal distribution when the quantity is applied to the Mahalanobis space, which is the reference space, may be calculated. In this case, the determination unit 26 compares the Mahalanobis distance (first distance) for the normal distribution calculated by the distance calculation unit 24 and the Mahalanobis distance (second distance) for the abnormal distribution calculated by the distance calculation unit 24. may be used to determine the state of the observation target. Specifically, the determination unit 26 determines that the first distance calculated by the distance calculation unit 24 is normal if it is shorter than the second distance, and determines that it is abnormal if the first distance is longer than the second distance.

When it is specified that the recognition result of the state of the observation target in the recognition unit 11a is an erroneous recognition, the extraction unit 11b extracts the observation target in the recognition unit 11a based on the feature amount of the observation data for obtaining the specified recognition result. Other observation data for which there is a possibility that the recognition result of the state was an erroneous recognition is extracted from the storage unit 14 . The number of other observation data extracted by the extraction unit 11b may be one or more.

The update unit 11c recognizes that other observation data extracted by the extraction unit 11b (other observation data for which the recognition result of the state of the observation target in the recognition unit 11a may have been an erroneous recognition) is also erroneously recognized, When the update of the parameters used for state recognition is instructed, the parameters used for state recognition are updated. In this example, the determination unit 26 updates the determination distance used to determine the state of the observation target.

Note that the update unit 11c may update the distribution of the feature amount of the latent variable z in the reference space, which is stored in the distribution storage unit 25. FIG.

The instructing unit 11d causes erroneous recognition of any of the other observation data extracted by the extracting unit 11b (other observation data for which the recognition result of the state of the observation target in the recognizing unit 11a may have been an erroneous recognition). If we accept the designation that it is not (the recognition result is correct), we exclude the observation data designated as not being misrecognition, and other observation data that may have been the recognition result of the state of the observation target being misrecognition. The extraction unit 11b is instructed to narrow down the The extraction unit 11b, according to the instruction of the instruction unit 11d, narrows down other observation data for which there is a possibility that the recognition result of the state of the observation target was an erroneous recognition. Other observation data that may have resulted in misrecognition).

The display control unit 11 e generates screen display data to be displayed on the screen of the display device 3 .

When the hardware CPU constituting the control unit 11 of the state detection device 1 executes the state detection program according to the present invention, the recognition unit 11a, the extraction unit 11b, the update unit 11c, the instruction unit 11d, and the display control unit 11e works as The memory also has an area for developing the state detection program according to the present invention and an area for temporarily storing data generated when the state detection program is executed. The control unit 11 may be an LSI that integrates a hardware CPU, memory, and the like. Also, the hardware CPU is a computer that executes the state detection method according to the present invention.

<3. Operation example>
State recognition processing for recognizing the state of an observation target by the state detection device 1 of this example will be described. FIG. 6 is a flow chart showing the state recognition processing of the state detection device of this example.

The state detection device 1 has an observation data storage unit (not shown) that temporarily stores observation data of an observation target input to the input unit 12 . In this example, the observation data are frame images captured by the video camera 2 of the observation target. The state detection device 1 selects observation data to be processed from the observation data temporarily stored in the observation data storage unit (s1). For example, frame images of a moving image captured by the video camera 2 may be sequentially selected as the observation data to be processed, or the m-th frame image from the frame image selected as the observation data to be processed last time. may be selected as the observation data to be processed.

Note that the method of selecting the observation data to be processed is an example, and other methods may be used for selection.

The state detection device 1 encodes the observation data to be processed selected in s1 with the encoder 21 of the VAE 20, and extracts the latent variable z of this observation data (s2). At this time, the VAE 20 may or may not restore observation data from the latent variable z extracted by the decoder 22 .

The feature quantity calculator 23 calculates the feature quantity of the latent variable z extracted from the observation data (s3). The feature value of the latent variable z calculated in s3 may be calculated for each pixel of the observation data (frame image) to be processed, or may be calculated for each divided area obtained by dividing the frame image into a plurality of predetermined areas. may be The frame image may be divided according to the surrounding environment of the object to be observed. For example, the frame image may be divided into n×m rectangular regions divided vertically by n and horizontally by m. However, it may be divided into a plurality of regions surrounded by curved lines, or may be divided by other methods.

In s3, the feature amount of the latent variable z calculated is a feature amount of a type whose normal distribution is stored in the distribution storage unit 25. FIG.

The distance calculator 24 applies the feature amount calculated in s3 to the Mahalanobis space, which is the reference space, and calculates the distance (Mahalanobis distance) to the normal distribution (s4). In this example, the Mahalanobis distance to the normal distribution is calculated, but if the distribution storage unit 25 stores the abnormal distribution of the feature quantity of the latent variable z, the Mahalanobis distance to the abnormal distribution is calculated. When the distribution storage unit 25 stores the normal distribution and the abnormal distribution of the feature amount of the latent variable z, the Mahalanobis distance to the normal distribution and the Mahalanobis distance to the abnormal distribution are calculated.

Based on the Mahalanobis distance calculated in s4, the determination unit 26 performs determination processing for determining whether the state of the observation target is normal or abnormal (s5). The determination result of the determination unit 26 is the recognition result of the recognition unit 11a.

The state detection device 1 outputs the determination result by the determination unit 26, and stores the current observation data, the feature amount calculated in s3, and the recognition result in association with each other in the storage unit 14 (s6, s7), and s1 back to In s6, for example, only when the recognition result determines that the state of the object to be observed is abnormal, it may be output to the host device. Further, in s6, the display control unit 11e may generate screen display data for causing the display 3 to display a screen showing the current observation data and the recognition result, and output the screen display data to the display 3. FIG. By looking at the screen of the display device 3, the user can confirm the recognition result together with the frame image of the object to be observed.

Next, update processing for updating the parameters used when the state detection device 1 of this example recognizes the state of the observation target will be described. FIG. 7 is a flowchart showing update processing of the state detection device of this example. The state detection device 1 executes this update process when the user performs an operation on the input device 4 indicating that the recognition result of the state of the observation target is an erroneous recognition.

The state detection device 1 waits until the input receiving unit 16 receives an input indicating that the recognition result of the state of the observation target is erroneous recognition (s11).

When the input receiving unit 16 receives an input indicating that the recognition result of the state of the observation target is an erroneous recognition, the state detecting device 1 stores the observation data specified as the current erroneous recognition by the extracting unit 11b. Among the recognition results stored in the unit 14, other observation data that may have been erroneous recognition are extracted (s12). In s12, for example, based on the feature amount of the latent variable z extracted from the observation data designated to be erroneous recognition this time, the recognition result is determined to be erroneous recognition among the recognition results stored in the storage unit 14. Extract other observations that may have been. Specifically, the similarity of the feature amount of the latent variable z is similar to the observation data designated as misrecognition this time (the similarity exceeds a predetermined threshold). The recognition result is extracted as other observation data that may have been an erroneous recognition. Also, for example, there is a possibility that the observation data whose difference from the Mahalanobis distance with respect to the normal distribution calculated for the observation data designated as being erroneously recognized this time is within a predetermined range, was erroneously recognized as the recognition result. Extract as other observation data.

The display control unit 11e generates screen display data for displaying on the screen of the display device 3 other observation data extracted in s12 that may have been misrecognition as the recognition result, and outputs the data to the display device 3. (s13). As a result, the display device 3 displays on the screen other observation data for which the recognition result extracted this time may have been an erroneous recognition.

The user confirms the screen of the display device 3 and confirms whether or not all of the other observation data that may have been erroneous recognition of the recognition result extracted this time are erroneous recognitions. In other words, the user confirms the screen of the display device 3, and the observation data in which the recognition result is not an erroneous recognition is included in the other observation data extracted this time that may have been an erroneous recognition. Check if it is

If all of the other observation data for which the recognition result extracted this time may have been erroneous recognition is erroneous recognition, the user performs an input operation for permitting update on the input device 4 . In addition, if the observation data extracted this time that may have been misrecognition includes observation data that does not result in misrecognition, the user may Observation data for which the result was not an erroneous recognition) is specified, and an input operation is performed to instruct disapproval of update.

The state detection device 1 waits for an input regarding update permission or update non-permission (s14, s15). In the state detection device 1, when the user performs an input operation for permitting updating on the input device 4, the updating unit 11c instructs the recognizing unit 11a to update the parameters used to determine the state of the observation target. According to this instruction, the recognition unit 11a updates the parameters used for determining the state of the observation target (s16), and returns to s11. In s16, for example, the judgment distance may be updated, or the normal distribution, the abnormal distribution, or both the normal distribution and the abnormal distribution of the feature amount of the latent variable z stored in the distribution storage unit 25 may be updated. good. In s16, the updating unit 11c updates the parameters so that the recognition result of the observation data for which the user has confirmed that the recognition is erroneous changes.

In addition, when the user performs an input operation to disallow update on the input device 4, the state detection device 1 detects the observation data extracted in s12 (other observation data whose recognition result may have been an erroneous recognition). ) is narrowed down (s17). In s17, the instructing unit 11d selects an erroneous recognition group in which the observation data extracted in s12 is determined to be an erroneous recognition based on the observation data specified together with the input of the update disapproval, and the recognition result is not an erroneous recognition. and an appropriate recognition group in which the recognition result is determined to be appropriate. For example, based on the feature amount of the latent variable z of the observation data designated as misrecognition this time and the feature amount of the latent variable z of the observation data designated as not being misrecognition this time, extraction is performed in s12. We divide the observed data into two groups. For example, for each observation data extracted in s12 (excluding observation data specified that the current recognition result is not an erroneous recognition), the extraction unit 11b extracts the current recognition result A distance (first feature value distance) from the feature value of the latent variable z of observation data designated as erroneous recognition is calculated. In addition, for each observation data extracted in s12 (excluding the observation data specified that the current recognition result is not an erroneous recognition), the extraction unit 11b extracts the current recognition result is not an erroneous recognition, and the distance (second feature distance) between the observation data and the feature of the latent variable z is calculated. The extraction unit 11b determines whether the observation data belongs to the incorrect recognition group or the proper recognition group, depending on the magnitude of the first feature amount distance and the second feature amount distance. Specifically, the extraction unit 11b determines that the observation data whose first feature distance is shorter than the second feature distance is an erroneously recognized group. Conversely, the extraction unit 11b determines that the observation data whose first feature distance is longer than the second feature distance belongs to the proper recognition group.

The display control unit 11e generates screen display data for displaying the observation data on the screen of the display device 3 for each group divided in s17 as a narrowing result, and outputs the data to the display device 3 (s18). As a result, the display unit 3 displays a screen that distinguishes between the observation data extracted in step s12 for which the recognition result is determined to be erroneous recognition and the observation data for which the recognition result is determined to be proper recognition. do. Therefore, when the recognition result of the observation data designated this time is set to be correct for the user, the state detection device 1 determines which observation data recognition result is erroneous recognition, and which observation data recognition result is correct. It can be made to confirm whether it is recognized or not.

After completing the processing of s18, the state detection device 1 returns to s14 and repeats the above-described processing.

In this way, the user updates the parameters used by the recognition unit 11a (recognition model) to recognize the state of the observation target by a simple operation of specifying the observation data for which the recognition result is determined to be an erroneous recognition. can. Therefore, the state detection device 1 of this example can easily suppress the occurrence frequency of erroneous recognition of the state of the observation target caused by the difference between the learning environment and the operating environment.

It should be noted that the parameters referred to here are not limited to the judgment distance used for judgment of normality or abnormality, as described above, but the normal distribution of the feature amount of the observation data when the state of the observation target is normal, It may be an abnormal distribution or the like of the feature amount of observation data when it is abnormal.

<4. Variation>
In the above example, the recognition unit 11a uses the frame images captured by the video camera 2 as the observation data of the observation target. A difference image is generated between the first average image of the images and the second average image of the temporally continuous m frames (n>m) captured by the video camera 2, and the difference image is used as observation data. It may be a configuration used as.

In this case, n is 2 or more and m is 1 or more. Also, the frame images of n frames used to generate the first average image may or may not include the frame images of m frames used to generate the second average image.

For example, a first average image is generated using n frames of frame images captured by the video camera 2 that are temporally continuous, and among the frame images of the n frames, the frame images of m frames captured later are selected. may be used to generate a second average image. In this case, the m frame images used to generate the second average image are included in the n frame images used to generate the first average image.

Further, for example, among the frame images of p frames (p<n+m) that are temporally continuous captured by the video camera 2, the frame images of the n frames that are captured earlier are used to generate the first average image, Among the frame images of the p frames, the frame images of the m frames captured later may be used to generate the second average image. In this case, some of the frame images used to generate the second average image are included in the n frame images used to generate the first average image.

Further, for example, among the temporally continuous n+m frame images captured by the video camera 2, the frame images of the n frames captured earlier are used to generate the first average image, and the frames of the n+m frames are generated. The second average image may be generated by using the m frame images captured later in the images. In this case, the second average image is generated by the image of m frames temporally consecutive to the frame images of n frames whose imaging times are used to generate the first average image.

Further, for example, generating a first average image using the frame image of the n frame whose imaging time is earlier among the frame images of q frames (q>n+m) that are temporally continuous captured by the video camera 2, Among the q frame images, the second average image may be generated using the m frame images captured later. In this case, the q frames include frame images that are not used to generate the first average image and the second average image. In this case, among the q frames, the number of frame images that are not used for generating the first average image and the second average image should be about several frames to several tens of frames.

With this configuration, the recognition unit 11a recognizes an object (for example, a suspicious object 105 or a falling object that has fallen from the vehicle 110) that appeared in the imaging area (inside the observation target area) of the video camera 2 immediately before the second time. Since the image is used as observation data (the difference image between the first average image and the second average image), observation data suitable for recognizing the state of the observation target can be obtained.

In the above example, the recognition unit 11a is not limited to the VAE 20 described above, and may be another type of recognition model as long as it has a recognition model generated by machine learning.

In the above example, the frame image or differential image of the observation target is used. Alternatively, it may be the temperature around the observation target, or sensing data obtained by measuring the observation target or the periphery of the observation target with a sensor other than a temperature sensor. Also, the number of observation data may be not one but may be plural.

In the above example, the recognizing unit 11a is configured to recognize whether the state of the observation target is normal or abnormal. It may recognize which of the 1 state, the 2nd state, and the intermediate state (the state between the 1st state and the 2nd state).

It should be noted that the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying constituent elements without departing from the scope of the present invention at the implementation stage. Also, various inventions can be formed by appropriate combinations of the plurality of constituent elements disclosed in the above embodiments. For example, some components may be omitted from all components shown in the embodiments. Furthermore, constituent elements of different embodiments may be combined as appropriate. Moreover, the order of the processes shown in FIGS. 6 and 7 is not limited to the illustrated order, and may be changed as appropriate.

Furthermore, the correspondence relationship between the configuration according to the present invention and the configuration according to the above-described embodiment can be described as the following additional remarks.
<Appendix>
an input unit (12) into which observation data of an observation target is input;
The observed data input to the input unit (12) are associated with the recognition result of the state of the observation target recognized by the recognition model (20) generated by machine learning based on the feature amount of the observed data. a storage unit (14) for storing
If the recognition result is specified as an erroneous recognition, the storage unit stores other observation data that may have been an erroneous recognition based on the feature amount of the observation data that obtained the specified recognition result. an extraction unit (11b) for extracting from (14);
an update unit (11c) for updating a parameter used by the recognition model to determine the state of the observation target when permission to update the recognition model is received after the observation data extracted by the extraction unit (11b) is output; and a state detection device (1).

Reference Signs List 1 State detection device 2 Video camera 3 Display device 4 Input device 11 Control unit 11a Recognition unit 11b Extraction unit 11c Update unit 11d Instruction unit 11e Display control unit 12 Input unit 13 Output unit 14 storage unit 15 display output unit 16 input reception unit 20 variational autoencoder (VAE)
21 Encoder 22 Decoder
23... Feature amount calculation unit 24... Distance calculation unit 25... Distribution storage unit 26... Judgment unit

Claims (10)

an input unit into which observation data of an observation target is input;
a storage unit that associates and stores observation data input to the input unit and recognition results of the state of the observation target recognized by a recognition model generated by machine learning based on the feature amount of the observation data; ,
If the recognition result is specified as an erroneous recognition, the storage unit stores other observation data that may have been an erroneous recognition based on the feature amount of the observation data that obtained the specified recognition result. an extractor that extracts from
an update unit that updates parameters used by the recognition model to determine the state of the observation target when permission to update the recognition model is received after the observation data extracted by the extraction unit is output. detection device. After outputting the observation data extracted by the extraction unit, when the specification of the observation data whose recognition result is not to be corrected is received, the extraction unit receives the previously extracted recognition based on the feature amount of the specified observation data. 2. The state detection device according to claim 1, further comprising an instruction unit for instructing narrowing down of other observation data whose result may have been erroneous recognition. The recognition model is
Machine learning using learning data including observation data of the observation target under normal conditions, including a variational autoencoder that acquires a distribution of latent variables extracted from observation data under normal conditions as a reference distribution,
the variational autoencoder obtains, as the feature quantity, a distribution of latent variables of the observation data input to the input unit as a distribution for judgment;
Recognizing the state of the observation target based on the distance between the determination distribution and the reference distribution;
The state detection device according to claim 1 or 2. 4. The state detection device according to claim 3, wherein the distance between said distribution for judgment and said reference distribution is a Mahalanobis distance. The state detection device according to any one of claims 1 to 4, wherein said observation data is a frame image of an object to be observed. The observation data is a difference image between a first average image of n frame images captured temporally continuously and a second average image of m frame images captured temporally continuously. and
n is greater than m,
The state detection device according to any one of claims 1 to 4, wherein m is 1 or more. The state detection device according to any one of claims 1 to 6, further comprising a display control section for displaying the observation data extracted by the extraction section on a display. The state detection device according to any one of claims 1 to 7, wherein the recognition model recognizes whether the state of the observation target is normal or abnormal. The observation data of the observation target input to the input unit and the recognition result of the state of the observation target recognized by the recognition model generated by machine learning based on the feature amount of the observation data are associated and stored in the storage unit. a recognition result storing step for storing;
If the recognition result is specified as an erroneous recognition, the storage unit stores other observation data that may have been an erroneous recognition based on the feature amount of the observation data that obtained the specified recognition result. an extraction step of extracting from
an update step of updating a parameter used by the recognition model to determine the state of the observation target when permission to update the recognition model is received after outputting the observation data extracted in the extraction step; State detection method. The observation data of the observation target input to the input unit and the recognition result of the state of the observation target recognized by the recognition model generated by machine learning based on the feature amount of the observation data are associated and stored in the storage unit. a recognition result storing step for storing;
If the recognition result is specified as an erroneous recognition, the storage unit stores other observation data that may have been an erroneous recognition based on the feature amount of the observation data that obtained the specified recognition result. an extraction step of extracting from
causing a computer to execute an update step of updating a parameter used by the recognition model to determine the state of the observation target when permission to update the recognition model is received after outputting the observation data extracted in the extraction step; State detection program.

Images